Update - Summary: nas-12-2 could be online Friday at the earliest, but more likely early next week.
In consultation with Adam Getchell, the decision has been made to do low-level disk copy from the old, failing drives, to new drives. This will minimize the potential for data loss.
This process is expected to finish Thursday at the earliest. Subsequently, the new disks will be added back to the ZFS pool, and we will trigger a full ZFS data scrub. When that finishes, we will know exactly how much, if any, data loss there is and which files are impacted. That data scrub will take a minimum of 24 hours, so the earliest nas-12-2 could be back in service is late Friday. It is more likely the scrub will run through the weekend, so a more realistic return-to-service is early next week.
Apr 02, 2025 - 13:59 PDT
In consultation with Adam Getchell, the decision has been made to do low-level disk copy from the old, failing drives, to new drives. This will minimize the potential for data loss.
This process is expected to finish Thursday at the earliest. Subsequently, the new disks will be added back to the ZFS pool, and we will trigger a full ZFS data scrub. When that finishes, we will know exactly how much, if any, data loss there is and which files are impacted. That data scrub will take a minimum of 24 hours, so the earliest nas-12-2 could be back in service is late Friday. It is more likely the scrub will run through the weekend, so a more realistic return-to-service is early next week.
Apr 02, 2025 - 13:59 PDT
Identified - nas-12-2 has suffered from multiple disk failures. Admins are investigating the best path forward.
The following group directories are currently unavailable:
awhitehegrp
millermrgrp
millsgrp
runciegrp
weimergrp
yujingrp
The following home directories are unavailable:
aavalos7
awhitehe
barao
bcbaikie
bcweimer
berdeja
crice
crios
cschles
dglemay
djprince
dkblaufu
drbandoy
eabernat
ecgranad
edkoch
emmaluu
eoziolor
fengq
hahudson
hemstrow
hxhu
jagill
jajpark
jamcgirr
jassim
jcariute
jdowen
jenwash
jmiller1
jroach
jrwashab
jxnliu
katng23
ljcohen
madarm11
mam12n
mary363
millermr
mlyjones
mmosmond
motch
mtreiber
namcnabb
nmariano
nreid
pjseba
profeta
prvasque
psbapat
rsbrenna
sakre
saumyaw
scsastry
seboles
sejoslin
smhigdon
spatel23
tmbolt
vfbetsis
vpdunne
wolfie12
xmixu
yoxue
ytakim
ywdong
Mar 31, 2025 - 18:09 PDT
The following group directories are currently unavailable:
awhitehegrp
millermrgrp
millsgrp
runciegrp
weimergrp
yujingrp
The following home directories are unavailable:
aavalos7
awhitehe
barao
bcbaikie
bcweimer
berdeja
crice
crios
cschles
dglemay
djprince
dkblaufu
drbandoy
eabernat
ecgranad
edkoch
emmaluu
eoziolor
fengq
hahudson
hemstrow
hxhu
jagill
jajpark
jamcgirr
jassim
jcariute
jdowen
jenwash
jmiller1
jroach
jrwashab
jxnliu
katng23
ljcohen
madarm11
mam12n
mary363
millermr
mlyjones
mmosmond
motch
mtreiber
namcnabb
nmariano
nreid
pjseba
profeta
prvasque
psbapat
rsbrenna
sakre
saumyaw
scsastry
seboles
sejoslin
smhigdon
spatel23
tmbolt
vfbetsis
vpdunne
wolfie12
xmixu
yoxue
ytakim
ywdong
Mar 31, 2025 - 18:09 PDT
Login
?
Partial Outage
Storage
?
Partial Outage
File transfer node
?
Operational
high2,med2,low2
?
Operational
high,med,low
?
Operational
bmh,bmm
?
Operational
bigmemh,bigmemm
?
Operational
bgpu
?
Operational
gpuh,gpum
?
Operational
Email
?
Operational
Virtualization
Operational
Proxmox Virtualization Nodes
Operational
Ganetti cluster
?
Operational
Slurm
?
Operational
Software
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.