PBS ZFS Drive Replacement

This procedure documents how to replace a failed disk in the four-drive ZFS pool used by pbs.lab. A matching spare drive is kept offline as a cold spare and should only be installed when one of the active pool members fails.

Do not use zpool add when replacing a failed drive. zpool add adds another vdev to the pool. A failed member should be replaced with zpool replace.

1. Identify the Failed Drive

#

Check the current pool state.

zpool status PBS

Record the exact failed device shown by ZFS. It may appear as FAULTED, UNAVAIL, or OFFLINE.

Use stable disk identifiers when possible rather than /dev/sdX, since Linux device letters can change between boots.

ls -l /dev/disk/by-id/
lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE

If the failed disk is still visible and the system is stable enough to do so, it can be taken offline before replacement.

zpool offline PBS /dev/disk/by-id/<FAILED_DRIVE_ID>

2. Replace the Physical Drive

#

Power down pbs.lab before opening the Wyse NAS enclosure.

systemctl poweroff

Remove the failed drive, install the cold spare in the same bay, and power the system back on.

After boot, identify the replacement disk by model and serial number.

lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE
ls -l /dev/disk/by-id/

3. Prepare the Replacement Disk

#

If the replacement drive contains old partition, filesystem, RAID, or ZFS signatures, clear them before using it.

wipefs -a /dev/disk/by-id/<NEW_DRIVE_ID>
sgdisk --zap-all /dev/disk/by-id/<NEW_DRIVE_ID>

Confirm that the command is targeting the replacement disk before running either command.

4. Replace the Failed ZFS Member

#

Tell ZFS to replace the failed member with the newly installed drive.

zpool replace PBS \
  /dev/disk/by-id/<FAILED_DRIVE_ID> \
  /dev/disk/by-id/<NEW_DRIVE_ID>

If zpool status PBS represents the missing disk using a different identifier, use the exact failed-device name shown there as the old-device argument.

5. Monitor the Resilver

#

ZFS will resilver the replacement disk automatically after the replacement begins.

watch -n 10 zpool status PBS

Do not remove another drive or make unnecessary storage changes while the resilver is running.

When complete, verify that the pool has returned to ONLINE and reports no known data errors.

zpool status PBS

Expected final state:

state: ONLINE
errors: No known data errors

6. Verify the Pool

#

After the resilver has completed, start a scrub to verify the repaired pool.

zpool scrub PBS

Monitor it with:

zpool status PBS

Once the scrub finishes, confirm that the pool remains ONLINE and no new read, write, checksum, or data errors are reported.

Quick Reference

#

Check pool          -> zpool status PBS
Identify disks      -> ls -l /dev/disk/by-id/
Optional offline    -> zpool offline PBS <failed-disk>
Power down          -> systemctl poweroff
Install cold spare
Clear old signatures
Replace member      -> zpool replace PBS <failed-disk> <replacement-disk>
Monitor resilver    -> watch -n 10 zpool status PBS
Verify              -> zpool status PBS
Scrub               -> zpool scrub PBS
Final verification  -> zpool status PBS