It happens within the blink of an eye.
A delete command was executed and half a second after you hit the enter button you knew it. That was a mistake.
This is the scenario which leads to this blog entry in where I show you how you can get your files back if you are lucky…
Short summary for the desperate
If you land here you are probably in the same situation I was so here is a short summary
Extundelete did not work for me but ext4magic did and I had to compile it from the sources
- Remount the filesystem read-only or umount it as soon as possible after the incident
- Backup your inode table you will need it for the restore
debugfs -R "dump /tmp/VMSSD01.journal" /dev/mapper/VMSSD01-VMSSD01
- Check at which time your files were still there
ext4magic /dev/mapper/VMSSD01-VMSSD01 -H -a $(date -d "-3hours" +%s)
- List the files within this timepoint
ext4magic /dev/mapper/VMSSD01-VMSSD01 -a 1542796423 -f / -l
- Restore the file to a different disc/mountpoint
ext4magic /dev/mapper/VMSSD01-VMSSD01 -a 1542796423 -f / -j /tmp/VMSSD01.journal -r -d /tmp/recover
- Be happy and promise never doing it again
And now the whole story
So it happened that I deleted two VM images by accident. I was cleaning up my environment and there were two files centos75_base_clone-1.qcow2 and centos75_base_clone-2.qcow2: As you can see I was using a clean and good naming convention which points directly, that these are the OS image files for my “nomachine” and my “networkmaster” machine… Especially the second one with my dhcp, dns, nfs and iscsi configuration would take some time to configure again.
In the first place nothing seemed to be wrong, all VMs were running normally until I tried to restart one of them and I went from 😎 to 😯 and at the end to 😳
I could remember, that it was very important to unmount the filesystem as quickly as possible and stop changing anything on this filesystem
So a solution had to be found. A short Google search brought me to a tool with the promising name “extundelete” which can be found in the CentOS repository in the actual version 0.2.4 from 2012….
yum install -y extundelete and a
man extundelete later I tried the command
extundelete --restore-all --after $(date -d "-2 hours" +%s) /dev/mapper/VMSSD01-VMSSD01
And…. It does not work.
A cryptical core dump and no solution on google so I went from 😯 TO 😥 .
But it was not the time to give up. With the courage of the despaired, I searched around and found the tool ext4magic. Magic never sounded better than in this right moment. The tool was newer then extundelete even when it builds on extundelete. So I downloaded and compiled the newest Version 0.3.2 (from 2014). Before you can compile the source you need some dependencies:
yum install -y libblkid \
zerofree e2fsp* \
and to add some more “Magic” you need also
yum install -y perl-File-LibMagic
./configure && make later I got a binary and to tell it with Star Was: “A New Hope” started growing in me.
I listed out the content on the different timestamps and found at least one of my files. The timestamp 1542797503 showed some files so I tried to list all files from an earlier timestamp and one of my missing image files showed up.
./src/ext4magic /dev/mapper/VMSSD01-VMSSD01 -a 1542796423 -f / -l
My mood started getting better and better and switched from 😥 to :???:.
I tried to restore my file
./src/ext4magic /dev/mapper/VMSSD01-VMSSD01 -a 1542796423 -f / -j /tmp/VMSSD01.journal -r -d /VMSSD02/recovery/
My first file is back 😀 . But the tool did not stop, it recovers more and more files and my hope was growing, to get both files back. The first file was back with the original name. For the second one, it was not that clear what happened. The tool was still running and recovers file after file after file and put all in the subdirectories MAGIC-2.
I tried to cancel the recovery job and give it a shot with the recovered files.
After renaming the *.unknown files I tried to boot up the VM. To my surprise the first try was successful and all my VMs were back online.
- Do not delete your files (obviously).
- Use a clear naming convention for all your files.
- A lsof before deleting a supposed unused file is always a good idea.
ext4magic worked for me and did as promised. My files are back, the VMs are up and running again. I am happy and 😎 .