This happened: I erased some file by accident. I wrote a bit of data later after that by mistake.
You can use klennet ZFS recovery program [1] to scan the ZFS pool of disks and find out if your information is still there. Unfortunately, klennet's program is kind of expensive $400. There must be another way.
Another person [2] mentions a method using a modified zdb and mdb programs to do this. The problem is that mdb is not available for linux, and there are no clear instructions of how to modify it.
The last method is by importing your pool using an older transaction number [3]. zpool manpage mentions that using -T option is “extremely hazardous”. On the article they recommend using the readonly option to avoid changes to disk. It was not clear, if you could go back to an older transaction state and then return back to the last transaction again.
What I want:
Recover as much as possible of the deleted files. Find out if the recovered files are corrupted or not (to re-download them from somewhere else).
Return to the last zfs pool transaction state (where the files are missing) in order to have a consistent zfs filesystem to work with.
This is how I did it:
sudo zpool import -o readonly=on -l tardis
sudo zdb tardis | grep "txg = "
Go to windows. Install Klennet ZFS Recovery program. (you can do this with virtualbox, you will have to create virtual “raw disks” that point to your real disks).
Create virtual raw disks in linux [5]:
For your windows usb to go key:
VBoxManage internalcommands createrawvmdk -filename usbkey.vmdk -rawdisk /dev/disk/by-id/<usb-key-device>
VBoxManage internalcommands createrawvmdk -filename <disk_x>.vmdk -rawdisk /dev/disk/by-id/<disk-x-id>
Configure virtualbox to recognize this disks. Configure your windows 10 vmachine to use this disks.
Configure your windows vmachine to use at least 16GB of RAM. If not, it will crash.
Start Klennet ZFS recovery software and during scan set: LZ4 compression, Fletcher4 checksum, disable Zvol, disable the rest of the compresssions and checksums and disable all raidz options (unless you need it).
Proceed to scan. It will take a lot of time (days even)
At the end you will be able to explore the recognized files. Find out the transaction number that contains your files. Write the number down.
Save the program state (in case you need to run the program again)
Save all the file name list.
Program a script that calculate the files that are missing from your current zfs linux state against the Klennet ZFS cvs file in the transaction of interest. [4] This will tell you exactly what to try to copy.
Now to the good/dangerous stuff
import your zpool with the transaction number from windows.
sudo zpool import -T <klennet old txg number> -o readonly=on -l tardis
Check that you have more files than before in the expected directories.
Copy those files. While copying be sure to redirect the stdout and stderr to a file to have a report of the files that give “read errors”. This files are corrupted. Use this to find them on Internet or somewhere else. Or at least, you know what to cry about now.
export the zpool
Import zpool again but without the -T option still readonly. Check that the filecount on the directories of interest is the correct one.
Export and reimport but without readonly option. All fine now!!!
Optional: run a md5sum on all your zfs files to be sure everything is back in order. If a file is corrupted you will get a read error.
Copy back your recovered files to your zfs pool!!!
All done!!!
And now: Really make a real backup. And remember don't disable your backup during a computer/hard_disk update: this is the moment where a harddrive fails more often, or during which you will make for mistakes and loose your data.
[1] https://www.klennet.com/zfs-recovery/default.aspx
[2] https://www.joyent.com/blog/zfs-forensics-recovering-files-from-a-destroyed-zpool
https://github.com/max123/mdbzfs
http://mbruning.blogspot.com/2008/08/recovering-removed-file-on-zfs-disk.html
https://www.delphix.com/blog/openzfs-pool-import-recovery
[3] https://web.archive.org/web/20200124210957/http://www.c0t0d0s0.org/archives/7621-Back-in-time-or-zpool-import-T.html
https://github.com/openzfs/zfs/issues/9313
[4] Script for finding out the missing files
[5] https://www.howtoforge.com/using-raw-devices-in-virtualbox-vms
http://cuddletech.com/?p=407
https://github.com/Stefan311/ZfsSpy
https://www.eall.com.br/blog/?p=312
https://zfsonlinux.org/msg/ZFS-8000-72/
https://zfsonlinux.org/manpages/0.8.0/man5/zfs-module-parameters.5.html
https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSZdbForFileAnalysishttps://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSZdbForFileAnalysis
https://forums.freebsd.org/threads/cannot-import-one-or-more-devices-is-currently-unavailable-but-all-devices-present-and-online.59660/