We have the following hardware:

• 1x 512GB nvme
• 4x 3TB sata
• 2x 5TB sata

And we want the following:

• Root on 200GB of nvme in mirror with another sata
• Bios doesn't boot from nvme, then boot must be copied on all satas
• UEFI partition also in all satas
• A fast only nvme partition of 200GB
• Be able to do backups of important directories (/home, /etc, and others) on external HD, maybe using snapshots
• Use many of the satas together for a big long storage “data” space with the rest of nvme as cache

How can ZFS help here:

• ZFS can't be used as /boot directory. Use ext4
• Use ZIL and SLOG as write cache for nvme for data volume
• Use L2ARC as read cache for nvme for data volume

We will do this:

• A btrfs for root, swap and “fast” associated with the nvme alone (no redundancy, but /etc will be gitted)
• /boot and UEFI partitions in several disks for booting
• A pool for the rest of the disks using some level of RAIDz1, it will have a small (32GB) SLOG and a bigger(128GB) L2ARC from the nvme disk
• A pool in the backup disks for backuping using zfs znapshot/replication/clone

• Wipe nvme disk (use gdisk /dev/nvme0n1 (o command)
• Create SLOG 32GB partition with bf00 type
• Create L2ARC 128GB partition with bf00 type
• Create Btrfs partition on (rest of 512GB disk, leave a bit for ssd discarding) type 8300
• Create Btrfs filesystem on Btrfs partition:

sudo mkfs.btrfs -L tardis /dev/nvme0n1p3

• Create /, home, var, fast and “swap” subvolumes (swap is really just a special file)

sudo mkdir /mnt/btrfsbase
sudo mount -t btrfs -o defaults,noatime /dev/nvme0n1p3 /mnt/btrfsbase
sudo btrfs subvolume create /mnt/btrfsbase/activeroot
sudo btrfs subvolume create /mnt/btrfsbase/home
sudo btrfs subvolume create /mnt/btrfsbase/var
sudo btrfs subvolume create /mnt/btrfsbase/fast
sudo btrfs subvolume create /mnt/btrfsbase/swap
sudo mkdir /mnt/newroot/
sudo mount -t btrfs -o defaults,noatime,subvol=activeroot /dev/nvme0n1p3 /mnt/newroot/

• Install the minimal system:

sudo debootstrap bullseye /mnt/newroot/

• Configure etc/hostname and etc/hosts correctly
• Configure etc/apt/sources.list and etc/network/interfaces correctly
• Bind mount virtual filesystems:

sudo mount --rbind /dev/ /mnt/newroot/dev/
sudo mount --rbind /proc/ /mnt/newroot/proc/
sudo mount --rbind /sys /mnt/newroot/sys/

• Chroot to the new /

sudo chroot /mnt/newroot

• Mount the rest of the btrfs filesystem volumes

mount -t btrfs -o defaults,noatime,subvol=home /dev/nvme0n1p3 /home/
mount -t btrfs -o defaults,noatime,subvol=var /dev/nvme0n1p3 /mnt/
cd /var
mv * /mnt/
umount /mnt
mount -t btrfs -o defaults,noatime,subvol=var /dev/nvme0n1p3 /var
mkdir swap
mount -t btrfs -o defaults,noatime,subvol=swap /dev/nvme0n1p3 /swap/

• Some extra basic configurations:

ln -s /proc/self/mounts /etc/mtab
apt update
apt upgrade
apt install --yes locales console-setup
dpkg-reconfigure locales
dpkg-reconfigure tzdata
apt install --yes dpkg-dev linux-headers-amd64 linux-image-amd64
apt install --yes zfs-initramfs btrfs-progs dosfstools
echo REMAKE_INITRD=yes > /etc/dkms/zfs.conf

• Prepare /boot and efi/biosgpt partitions from normal sata HD:

mkfs.vfat /dev/sdd2 # efi
mkfs.ext4 /dev/sdd3 # /boot
mount /dev/sdd3 /mnt
cd /boot
mv * /mnt
umount /mnt
mkdir -p /boot/efi #efi
mount /dev/sdd3 /boot
mount /dev/sdd2 /boot/efi # efi
apt install --yes grub-pc # non-efi
apt install --yes grub-efi-amd64 shim-signed # efi
dpkg --purge os-prober
apt install emacs
cp /usr/share/systemd/tmp.mount /etc/systemd/system/
systemctl enable tmp.mount
grub-probe /boot
update-initramfs -c -k all
update-grub
grub-install /dev/sdd # non-efi
grub-install --target=x86_64-efi --efi-directory=/boot/efi \
  --bootloader-id=debian --recheck --no-floppy # efi
apt install --yes openssh-server

• Edit /etc/fstab:

LABEL=tardis					/mnt/btrfsbase btrfs defaults,noatime			0 0
LABEL=tardis   					/	       btrfs defaults,noatime,subvol=activeroot	0 0
LABEL=tardis   					/home	      btrfs defaults,noatime,subvol=home	0 0
LABEL=tardis   					/var	      btrfs defaults,noatime,subvol=var		0 0
LABEL=tardis   		      	    		/swap	      btrfs defaults,noatime,subvol=swap	0 0
UUID="11d65735-75d0-44ce-a158-abd01d98b318"	/boot	      ext4  defaults,noatime			0 0

• Reboot to new system

• option 1: Stripping

sudo zpool create tardis  /dev/sdb4 /dev/sde4 /dev/sdf4

• option 2: raidz1 (with 3 new disks in raidz1 configuration of 2xdisk space)

sudo zpool create tardis raidz1 /dev/sdb4 /dev/sde4 /dev/sdf4

* option 3: raidz1 but degraded. (without one disk at the beginning) Solution: Use a sparse file as one of the starting disks [10].

truncate -s 8T tardisfake.img (Use the same exact size of the real disk partition) 
sudo zpool create tardis raidz1 /dev/sde4 /dev/sdf4 /home/memeruiz/tardisfake.img
sudo zpool offline tardisnew /home/memeruiz/tardisfake.img

• Add log and cache devices:

sudo zpool add tardis log /dev/nvme0n1p1
sudo zpool add tardis cache /dev/nvme0n1p2

• Create encrypted volume

sudo zfs create -o encryption=on -o keyformat=passphrase tardis/data

• Create compressed subvolume

sudo zfs create tardis/data/compressed
sudo zfs set compression=on tardis/data/compressed

• Avoid data to be mounted automatically with zfs:

sudo zfs set mountpoint=legacy tardis/data

• If pool was already activated, deactivate it!

sudo zpool export tardis

• Reactivate pool with -l option to ask for decryption passphrase:

sudo zpool import tardis -l

• Mount fs:

sudo mount -t zfs tardis/data /srv

• Extending a pool:

zpool add tardis /dev/<diskfile>

zpool offline poolname device
zpool offline -t poolname device (until next reboot)

zpool online poolname device

zpool add poolname cache device #cache
zpool add poolname log device  #log

zpool remove poolname device

errors: Permanent errors have been detected in the following files: 

You can fix that be doing a short scrubing and then stopping:

sudo zpool scrub tardis
# wait a bit
sudo zpool scrub -s tardis

Then error messages are gone

For example: if this disk is getting damaged: you may want to replace it with a new one, some files may be broken, but using:

sudo zpool replace tardis old_dev new_dev

This will copy everthing from the zfs from that disk to the new one, attach the new one and detach the old one. In this way, it is not necessary to copy the whole disk with dd.

mkfile 100m file1
zpool create geekpool /file1

• Create a sparse file with the same size of your harddrives. Then remove it before putting files to it.

https://www.reddit.com/r/zfs/comments/8enkt8/can_i_create_a_new_raidz_in_degraded_mode/

https://linuxhint.com/configuring-zfs-cache/

https://arstechnica.com/information-technology/2020/05/zfs-101-understanding-zfs-storage-and-performance/

https://www.howtoforge.com/tutorial/how-to-use-snapshots-clones-and-replication-in-zfs-on-linux/

https://docs.joyent.com/private-cloud/troubleshooting/disk-replacementhttps://docs.joyent.com/private-cloud/troubleshooting/disk-replacement

https://wiki.archlinux.org/index.php/btrfs#Subvolumes

https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Subvolumes

https://wiki.gentoo.org/wiki/Btrfs/Native_System_Root_Guide

https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Buster%20Root%20on%20ZFS.html

https://mohankumar-k.blogspot.com/2018/05/how-to-resolve-zpool-permanent-errors.html

https://docs.oracle.com/cd/E36784_01/html/E36835/gbbvf.html

https://www.thegeekdiary.com/zfs-tutorials-creating-zfs-pools-and-file-systems/

https://www.delphix.com/blog/delphix-engineering/openzfs-code-walk-metaslabs-and-space-maps

[10] https://www.mail-archive.com/zfs-discuss@opensolaris.org/msg22993.html

https://unix.stackexchange.com/questions/322352/create-raid-z2-in-degraded-state-possible