Recovering files deleted by mistake on Linux/Ubuntu

This article is not about logging, but about a terrible mistake I just did. And hopefully useful if you ever make the same mistake as me.

While testing and re-learning how to use mingw to build a Windows version of a logging agent I am working on, I executed this command on the terminal:

$ rm * .exe

I was trying to delete the .exe files I had just compiled, but mistakenly deleted all the files "*" in the local directory - that had a dozen .c files that I had been working for many days. And since I was troubleshooting and testing - and wasn't finished, I didn't had any backups.

Ok, a few days of lost work is not as terrible as some of the other things happening in the world right now. But it sucked. And I could not let a week of work go to waste.

So the first thing I did was to google for how to recover lost files on Linux - how to recover lost files on Ubuntu. I found many articles with different recommendations and tools:

  • extundelete --restore-all
  • ext4magic /dev/vda1 -f root/log-export
  • testdisk

Unfortunately... None. Of. them. did. anything. for me.

They did not find any files at all. Maybe I used them incorrectly, but I was following many guides online step by step - and did not work.

As a last attempt, I found a recommendation of a tool called PhotoRec. I installed and ran it to see if it would do anything for me:

$ sudo apt install photorec
# photorec

And I was prompted with this screen for me to select the partition.

PhotoRec 7.1, Data Recovery Utility, July 2019
Christophe GRENIER

PhotoRec is free software, and

Select a media (use Arrow keys, then press Enter):
>Disk /dev/vda - 274 GB / 256 GiB (RO)

You choose a partition, accept all the defaults and thats it.

I accepted the defaults and PhotoRec started its work to scan all the unused sectors on my disk looking for the headers of certain file types to recover.

PhotoRec 7.1, Data Recovery Utility, July 2019
Christophe GRENIER

Disk /dev/vda - 274 GB / 256 GiB (RO)
Partition Start End Size in sectors
1 * Linux 2 0 33 532609 15 62 536868831

Destination /root/recover/recup_dir

Pass 1 - Reading sector 445073836/536868831, 2246814 files found
Elapsed time 0h11m32s - Estimated time to completion 0h02m22

It found a lot of files and took about 15 minutes to complete. When done, it had almost 100GB of files inside the /root/recover/recup_dir/.* directory that I had chosen as a restoration directory:

# ls -la /root/recover/

That was a great sign. But how to find the files that I was looking for? Since I was migrating my Linux code to work on Windows, all my files had a few "#ifdef WIN32" on them, so tried that to find the files:

# grep -r "#ifndef WIN32" .
./recup_dir.1075/f135568704.c- #ifndef WIN32
./recup_dir.1075/f135568704.c- #ifndef WIN32
./recup_dir.1075/f135568704.c- #ifdef WIN32
./recup_dir.2550/f230485152.c- #ifndef WIN32
./recup_dir.2550/f230485152.c- #ifndef WIN32
./recup_dir.1173/f140713568.c- #ifndef WIN32
./recup_dir.1173/f140713568.c- #ifndef WIN32
./recup_dir.2294/f213052800.c- #ifndef WIN32
./recup_dir.2294/f213052800.c- #ifndef WIN32
./recup_dir.2148/f204236320.c- #ifndef WIN32
./recup_dir.2148/f204236320.c- #ifndef WIN32

The result returned many many files - often duplicated files of the same one that got deleted as I worked on it. To solve that problem, I went to each one and found a unique function or keyword that would allow me to restore one file at a time. For example, one file had a debug statement: "DEBUG: f should never reach here", so I searched for that:

# grep -r "DEBUG: f should never reach here" .
./recup_dir.1869/f367295352.c- dolog("DEBUG: f should never reach here");
./recup_dir.1707/f331066232.c- dolog("DEBUG: f should never reach here");
./recup_dir.131/f42931080.c- dolog("DEBUG: f should never reach here");

And after that I just had to decide which one was the latest. On my case, I run "ls -la" and picked the largest file as the final one to use:

# ls -la ./recup_dir.1869/f367295352.c ./recup_dir.1707/f331066232.c ./recup_dir.131/f42931080.c ./recup_dir.1531/f295281352.c

And after about 1 hour of grepping and trying to recover all the files, I was done. If you ever make the same mistake I did, try the open source PhotoRec. Or, don't do the same mistake and be careful before you rm anything. And thanks Christophe GRENIER for PhotoRec. Saved my day.

Posted in   troubleshooting-guides     by Daniel Cid (@dcid)

Simple, affordable, log management and analysis.