We have the following hardware:

• 1x 512GB nvme
• 4x 3TB sata
• 2x 5TB sata

And we want the following:

• Root on 200GB of nvme in mirror with another sata
• Bios doesn't boot from nvme, then boot must be copied on all satas
• UEFI partition also in all satas
• A fast only nvme partition of 200GB
• Be able to do backups of important directories (/home, /etc, and others) on external HD, maybe using snapshots
• Use many of the satas together for a big long storage “data” space with the rest of nvme as cache

How can ZFS help here:

• ZFS can't be used as /boot directory. Use ext4
• Use ZIL and SLOG as write cache for nvme for data volume
• Use L2ARC as read cache for nvme for data volume

We will do this:

• A btrfs for root, swap and “fast” associated with the nvme alone (no redundancy, but /etc will be gitted)
• /boot and UEFI partitions in several disks for booting
• A pool for the rest of the disks using some level of RAIDz1, it will have a small (32GB) SLOG and a bigger(128GB) L2ARC from the nvme disk
• A pool in the backup disks for backuping using zfs znapshot/replication/clone

• Wipe nvme disk (use gdisk /dev/nvme0n1 (o command)
• Create SLOG 32GB partition with bf00 type
• Create L2ARC 128GB partition with bf00 type
• Create Btrfs partition on (rest of 512GB disk, leave a bit for ssd discarding) type 8300
• Create Btrfs filesystem on Btrfs partition:

sudo mkfs.btrfs -L tardis /dev/nvme0n1p3

• Create /, home, var, fast and “swap” subvolumes (swap is really just a special file)

sudo mkdir /mnt/btrfsbase
sudo mount -t btrfs -o defaults,noatime /dev/nvme0n1p3 /mnt/btrfsbase
sudo btrfs subvolume create /mnt/btrfsbase/activeroot
sudo btrfs subvolume create /mnt/btrfsbase/home
sudo btrfs subvolume create /mnt/btrfsbase/var
sudo btrfs subvolume create /mnt/btrfsbase/fast
sudo btrfs subvolume create /mnt/btrfsbase/swap
sudo mkdir /mnt/newroot/
sudo mount -t btrfs -o defaults,noatime,subvol=activeroot /dev/nvme0n1p3 /mnt/newroot/

• Install the minimal system:

sudo debootstrap bullseye /mnt/newroot/

• Configure etc/hostname and etc/hosts correctly
• Configure etc/apt/sources.list and etc/network/interfaces correctly
• Bind mount virtual filesystems:

sudo mount --rbind /dev/ /mnt/newroot/dev/
sudo mount --rbind /proc/ /mnt/newroot/proc/
sudo mount --rbind /sys /mnt/newroot/sys/

https://linuxhint.com/configuring-zfs-cache/

https://arstechnica.com/information-technology/2020/05/zfs-101-understanding-zfs-storage-and-performance/

https://www.howtoforge.com/tutorial/how-to-use-snapshots-clones-and-replication-in-zfs-on-linux/

https://docs.joyent.com/private-cloud/troubleshooting/disk-replacementhttps://docs.joyent.com/private-cloud/troubleshooting/disk-replacement

https://wiki.archlinux.org/index.php/btrfs#Subvolumes

https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Subvolumes

https://wiki.gentoo.org/wiki/Btrfs/Native_System_Root_Guide

https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Buster%20Root%20on%20ZFS.html