This afternoon I began to receive "Error establishing a database connection" error from my site. And when trying to login via ssh, the server hangs after the password is entered.As I tried a few more times to log in, it became clear that something was wrong, as I was getting "ssh_exchange_identification: Connection closed by remote host" error instead. After I got home, I tried to reboot the server. Then came the dreadful clicking sound, I realized that the hard disk had failed. Luckily, I have a Virtual Machine on standby, and it took less than an hour to get the website back and running again.
Even though I have up-to-date backup of my site and I restored it without too much a fuss, I was still interested in recovering the data from the failed drive.
I mounted the drive on another computer and used ddrescure to dump the primary Linux partition into a file. It seems that the partition is pretty much intact as I did not get any error while retrieving the image from the failed disk. Note that if you want to mount the dumped image, you would need to dump only the primary partition and not the whole disk (in my case it was /dev/sdd1 instead of /dev/sdd). After successfully dumped the disk image, I was able to see everything on the failed drive by mounting the image to a directory:
sudo mount -o loop ./prod.img /mnt -t ext3
Over the next couple of days I will rebuild my production box and transfer the site back from the VM.
