Однажды утром понедельника обнаружил страшное:
mik17@sol1:~$ zpool status
pool: rpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
c4t0d0s0 ONLINE 0 0 0
errors: No known data errors
pool: zfs_pool_zones
state: FAULTED
status: The pool metadata is corrupted and the pool cannot be opened.
action: Recovery is possible, but will result in some data loss.
Returning the pool to its state as of October 13, 2016 03:10:03 AM MSK
should correct the problem. Approximately 20 minutes of data
must be discarded, irreversibly. After rewind, several
persistent user-data errors will remain. Recovery can be attempted
by executing 'zpool clear -F zfs_pool_zones'. A scrub of the pool
is strongly recommended after recovery.
see: http://illumos.org/msg/ZFS-8000-72
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zfs_pool_zones FAULTED 0 0 1 corrupted data
raidz1-0 DEGRADED 0 0 8
c4t1d0 DEGRADED 0 0 0 too many errors
spare-1 DEGRADED 0 0 0
c4t2d0 DEGRADED 0 0 0 too many errors
c4t6d0 ONLINE 0 0 0
c4t3d0 DEGRADED 0 0 0 too many errors
c4t4d0 DEGRADED 0 0 1 too many errors
c4t5d0 DEGRADED 0 0 0 too many errors
зоны не стартуют, все пропало )))
для начала попробовал "заклиарить" пулл, так как в дисках сомнений нет (это LUN с СХД)
mik17@sol1:~$ sudo zpool clear -F zfs_pool_zones
cannot clear errors for zfs_pool_zones: I/O error
теперь попробуем пулл "импортнуть":
mik17@sol1:~$ sudo zpool import -F zfs_pool_zones
cannot import 'zfs_pool_zones': a pool with that name is already created/imported,
and no additional pools with that name were found
ну хоть метаданные по нему есть, так что сначала навесим на пулл флаг экспорта:
mik17@sol1:~$ sudo zpool export zfs_pool_zones
скушал
теперь опять импорт
mik17@sol1:~$ sudo zpool import -F zfs_pool_zones
mik17@sol1:~$ sudo zpool status zfs_pool_zones
pool: zfs_pool_zones
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Mon Oct 17 11:15:29 2016
306M scanned out of 4.94G at 43.8M/s, 0h1m to go
60.8M resilvered, 6.06% done
config:
NAME STATE READ WRITE CKSUM
zfs_pool_zones ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
c4t1d0 ONLINE 0 0 0
spare-1 ONLINE 0 0 0
c4t2d0 ONLINE 0 0 0
c4t6d0 ONLINE 0 0 0 (resilvering)
c4t3d0 ONLINE 0 0 0
c4t4d0 ONLINE 0 0 0
c4t5d0 ONLINE 0 0 0
errors: 1 data errors, use '-v' for a list
восстановился с ошибками на файловой системе, нужно их scrub'нуть
mik17@sol1:~$ sudo zpool scrub zfs_pool_zones
mik17@sol1:~$ sudo zpool status -v zfs_pool_zones
pool: zfs_pool_zones
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: scrub in progress since Mon Oct 17 11:38:01 2016
225M scanned out of 4.94G at 37.5M/s, 0h2m to go
0 repaired, 4.45% done
config:
NAME STATE READ WRITE CKSUM
zfs_pool_zones ONLINE 0 0 4
raidz1-0 ONLINE 0 0 8
c4t1d0 ONLINE 0 0 0
c4t2d0 ONLINE 0 0 0
c4t3d0 ONLINE 0 0 0
c4t4d0 ONLINE 0 0 0
c4t5d0 ONLINE 0 0 0
spares
c4t6d0 AVAIL
errors: Permanent errors have been detected in the following files:
/zfs_pool_zones/zoneSokolov/root/var/cron/log
/zfs_pool_zones/zonehdscci1/root/var/cron/log
/zfs_pool_zones/zonepostgres1/root/var/svc/log/application-database-postgresql_84:default_64bit.log
/zfs_pool_zones/zonepostgres1/root/var/cron/log
после окончания очистки:
mik17@sol1:~$ sudo zpool status -v zfs_pool_zones
pool: zfs_pool_zones
state: ONLINE
scan: resilvered 57.2M in 0h0m with 0 errors on Mon Oct 17 12:08:47 2016
config:
NAME STATE READ WRITE CKSUM
zfs_pool_zones ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
c4t1d0 ONLINE 0 0 0
c4t2d0 ONLINE 0 0 0
c4t3d0 ONLINE 0 0 0
c4t4d0 ONLINE 0 0 0
c4t5d0 ONLINE 0 0 0
spares
c4t6d0 AVAIL
errors: No known data errors
На другой машинке все сложнее - там и rpool тоже в DEGRADED стейте. но это уже другая история.
