Sequence of Events

Distro upgrade to 16.04.1, apparently successful
reboot and end up in Emergency Mode as described e.g. here
use of systemctl etc. was unsuccessful; a suggestion was using Upstart via GRUB, which booted to desktop
the next boot (into Emergency Mode) did not succeed; now the /home partition was unavailable
boot via USBDrive; /home still unavailable, listed as "unknown partition" by gparted
lengthy investigation via fdisk, testdisk etc. wiki, a howto showed that all superblocks were corrupted, no backups available. I'm not a recovery expert but this seems unusual.
Make a testdisk image into "virtualbox" partition
Follow process described here without success, including last-ditch use of mke2fs -S.
testdisk deeper search finds partition, but not the superblocks; alogn with messages like No ext2, JFS, Reiser, cramfs or XFS marker
Tools like photorec can retrieve some files, but they are garbled, miss filename / structure, and many are encrypted due to it being a /home partition
Decide that as I have a backup of the partition, and the original data, delete and replace the /home partition, which succeeds. Boot into "home-new" and set it up...
and on the next boot, imagine my surprise to find both "home-new" and the "virtualbox" partition are unavailable. Same superblock issue. Well, great.
Using USBDrive, re-re-create home partition in a different disk location; this appears stable. Delete "home-new".
Realise that the backup of /home, and some critical data, are on that now-unreadable partition; obtain a USB disk and make an image on there

Current Situation

My partitions are:

DellUtility (OK)
Windows 7 (OK)
Ubuntu 16.04 (OK)
/home-new (bad, deleted, in same disk location as original /home) - extended
/home (seems OK, at different disk location) - extended
virtualbox (bad) - extended
swap (OK) - extended

The "virtualbox" partition is ~550GB. It is unreadable and has no valid superblocks.

Important data on it: * a testdisk image of my original /home partiton , ~50GB, itself unreadable with no valid superblocks * some original data from /home that wasn't handled by Crashplan. Not enough to be a critical problem, but enough to be annoying.

Note that first boots were fine - it was on the next boot the issues occurred. The "virtualbox" partition has been backed up as a Testdisk image on a USB drive.

fdisk:

sudo fdisk -l /dev/sda
Disk /dev/sda: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x15642c74
Device Boot Start End Sectors Size Id Type
/dev/sda1 * 63 80324 80262 39.2M 6 FAT16
/dev/sda2 18395136 427995135 409600000 195.3G 7 HPFS/NTFS/exFAT
/dev/sda3 427995136 1920120831 1492125696 711.5G f W95 Ext'd (LBA)
/dev/sda4 1920122880 1953523711 33400832 15.9G 82 Linux swap / Solaris
/dev/sda5 438233088 489433087 51200000 24.4G 83 Linux
/dev/sda6 773240832 1920120831 1146880000 546.9G 83 Linux
/dev/sda7 633921536 736321535 102400000 48.8G 83 Linux
Partition 1 does not start on physical sector boundary.
Partition table entries are not in disk order.

dumpe2fs:

sudo dumpe2fs /dev/sda6
dumpe2fs 1.42.13 (17-May-2015)
dumpe2fs: Bad magic number in super-block while trying to open /dev/sda6
Couldn't find valid filesystem superblock.

parted:

sudo parted -l
Model: ATA ST1000DM003-1CH1 (scsi)
Disk /dev/sda: 1000GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags 1 32.3kB 41.1MB 41.1MB primary fat16 boot 2 9418MB 219GB 210GB primary ntfs 3 219GB 983GB 764GB extended lba 5 224GB 251GB 26.2GB logical ext4 7 325GB 377GB 52.4GB logical ext4 6 396GB 983GB 587GB logical 4 983GB 1000GB 17.1GB primary linux-swap(v1)

Hypothesis

As you can imagine, this is very frustrating. Note the "Partition 1 does not start on physical sector boundary." warning in fdisk, which did not appear before.

Going by Partition does not start on physical sector boundary? , I suspect there is some alignment issue which confuses low-level disk utilities, introduced by the 16.04 upgrade, but that's just a suspicion.

gdisk:

sudo gdisk -l /dev/sda
GPT fdisk (gdisk) version 1.0.1
Partition table scan: MBR: MBR only BSD: not present APM: not present GPT: not present
***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory.
***************************************************************
Disk /dev/sda: 1953525168 sectors, 931.5 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 8C15FD44-F839-4637-853E-C092F0959C48
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 1953525134
Partitions will be aligned on 8-sector boundaries
Total free space is 209964007 sectors (100.1 GiB)
Number Start (sector) End (sector) Size Code Name 1 63 80324 39.2 MiB 0700 Microsoft basic data 2 18395136 427995135 195.3 GiB 0700 Microsoft basic data 4 1920122880 1953523711 15.9 GiB 8200 Linux swap 5 438233088 489433087 24.4 GiB 8300 Linux filesystem 6 773240832 1920120831 546.9 GiB 8300 Linux filesystem 7 633921536 736321535 48.8 GiB 8300 Linux filesystem

Questions

How can I identify the cause? My system now seems stable, but am wary of further low-level disk writes.
Is it possible to recover the superblock-less "virtualbox" partition - and then hopefully the superblock-less "home" partition image within?

Perhaps the superblocks are intact, but via an offset that the system isn't aware of.

1 Answer

Have made some progress (and a partial answer):

Noticed that in some applications (e.g. System Monitor), at some point the swap became reported as 546.9GB. The swap should be 15.9GB, and that's a number suspiciously near the broken "virtualbox" partition.

lsblk showed that /dev/sda6 - the partition - was also mapped via cryptswap1 to swap.

/etc/crypttab had:

cryptswap1 /dev/sda6 /dev/urandom swap,cipher=aes-cbc-essiv:sha256

The smoking gun! So hypothesis now is the 16.04 upgrade failed to re-configure swap correctly, and on later boots the swap startup broke the partition (which would explain why first boots were successful).

Disable swap everywhere (used technique in What to do about "the disk drive for /dev/mapper/cryptswap1 is not ready yet or not present"? )
Reboot and confirm swap-less state
Use testdisk to investigate. It attempts to mark active partitions as deleted and can't find the partition, so quit.
Confirm state unchanged with sudo dumpe2fs /dev/sda6
dumpe2fs: Bad magic number in super-block while trying to open /dev/sda6 Couldn't find valid filesystem superblock.
So use the last ditch method described above:
sudo /sbin/mkfs.ext4 -S -v /dev/sda6
sudo mount /dev/sda6 /media/USER/virtualbox-image/
... and ls lists some of the files / directories!

Well, hooray!!

I couldn't access the files, so ran fsck. There were so many errors I gave up with preen and interactive modes and just used -y. Here are some fsck messages for reference:

Group descriptor 4349 checksum is 0xf6d0, should be 0x2ed1. FIXED.
/dev/sda6: Inode 13434881 is in use, but has dtime set. FIXED.
/dev/sda6: Inode 13434881 has an extra size (336) which is invalid FIXED.
/dev/sda6: Inode 13434881 has INDEX_FL flag set but is not a directory. HTREE INDEX CLEARED.
/dev/sda6: Inode 13434881, i_blocks is 137157068659908, should be 0. FIXED.
Inodes that were part of a corrupted orphan linked list found. Fix? yes
Inode 13434886 was part of the orphaned inode list. FIXED.
Inode 13434886 has imagic flag set. Clear? yes
Inode 13434886 has an extra size (62340) which is invalid Fix? yes
Inode 13434886 has compression flag set on filesystem without compression support. Clear? yes
Inode 13434886 has INDEX_FL flag set but is not a directory. Clear HTree index? yes
Inode 13434886, i_size is 18440780219561279704, should be 0. Fix? yes
Inode 13434886, i_blocks is 219803506189340, should be 0. Fix? yes
Inode 13495674 has a bad extended attribute block 21496064. Clear? yes
Inode 13495674 has illegal block(s). Clear? yes
Illegal block #0 (1376321536) in inode 13495674. CLEARED.
File /image_new_superblock.dd (inode #45605, mod time Wed Oct 28 12:58:24 2015) has 1 multiply-claimed block(s), shared with 1 file(s): ... (inode #13455772, mod time Thu Jul 4 03:48:32 1996) Clone multiply-claimed blocks? yes

Many screens of the above, timestamps all over the place. fsck actually aborted several times due to memory allocation, LOL. Eventually it ran clean, with messages:

Running additional passes to resolve blocks claimed by more than one inode...
Pass 1B: Rescanning for multiply-claimed blocks
Pass 1C: Scanning directories for inodes with multiply-claimed blocks
Pass 1D: Reconciling multiply-claimed blocks

Can now mount the partition and copy data. A lot of files still seem intact.

But there's more to do; this partition has been affected for 3 weeks. Presumably if I repeat this on the image made initially, I can recover more or nearly all the data. And still need to investigate "Partition 1 does not start on physical sector boundary." But, looking good!

Recovering a partition with no valid superblocks

Sequence of Events

Current Situation

Hypothesis

Questions

1 Answer

Your Answer

Sign up or log in

Post as a guest

You Might Also Like

Where can I farm Boar Prime in warframe?

Can Vette be corrupted?

What are the advantages/disadvantages of each race?