We have the following hardware:

• 1x 512GB nvme
• 4x 3TB sata
• 2x 5TB sata

And we want the following:

• Root on 200GB of nvme in mirror with another sata
• Bios doesn't boot from nvme, then boot must be copied on all satas
• UEFI partition also in all satas
• A fast only nvme partition of 200GB
• Be able to do backups of important directories (/home, /etc, and others) on external HD, maybe using snapshots
• Use many of the satas together for a big long storage “data” space with the rest of nvme as cache

How can ZFS help here:

• ZFS can't be used as /boot directory. Use ext4
• Use ZIL and SLOG as write cache for nvme for data volume
• Use L2ARC as read cache for nvme for data volume

We will do this:

• A btrfs for root, swap and “fast” associated with the nvme alone (no redundancy, but /etc will be gitted)
• /boot and UEFI partitions in several disks for booting
• A pool for the rest of the disks using some level of RAIDz1, it will have a small (32GB) SLOG and a bigger(128GB) L2ARC from the nvme disk
• A pool in the backup disks for backuping using zfs znapshot/replication/clone

• Wipe nvme disk (use gdisk /dev/nvme0n1 (o command)
• Create SLOG 32GB partition with bf00 type
• Create L2ARC 128GB partition with bf00 type
• Create Btrfs partition on (rest of 512GB disk, leave a bit for ssd discarding) type 8300
• Create Btrfs filesystem on Btrfs partition:

sudo mkfs.btrfs -L tardis /dev/nvme0n1p3

• Create /, home, var, fast and “swap” subvolumes (swap is really just a special file)

sudo mkdir /mnt/btrfsbase
sudo mount -t btrfs -o defaults,noatime /dev/nvme0n1p3 /mnt/btrfsbase
sudo btrfs subvolume create /mnt/btrfsbase/activeroot
sudo btrfs subvolume create /mnt/btrfsbase/home
sudo btrfs subvolume create /mnt/btrfsbase/var
sudo btrfs subvolume create /mnt/btrfsbase/fast
sudo btrfs subvolume create /mnt/btrfsbase/swap
sudo mkdir /mnt/newroot/
sudo mount -t btrfs -o defaults,noatime,subvol=activeroot /dev/nvme0n1p3 /mnt/newroot/

• Install the minimal system:

sudo debootstrap bullseye /mnt/newroot/

• Configure etc/hostname and etc/hosts correctly
• Configure etc/apt/sources.list and etc/network/interfaces correctly
• Bind mount virtual filesystems:

sudo mount --rbind /dev/ /mnt/newroot/dev/
sudo mount --rbind /proc/ /mnt/newroot/proc/
sudo mount --rbind /sys /mnt/newroot/sys/

• Chroot to the new /

sudo chroot /mnt/newroot

• Mount the rest of the btrfs filesystem volumes

mount -t btrfs -o defaults,noatime,subvol=home /dev/nvme0n1p3 /home/
mount -t btrfs -o defaults,noatime,subvol=var /dev/nvme0n1p3 /mnt/
cd /var
mv * /mnt/
umount /mnt
mount -t btrfs -o defaults,noatime,subvol=var /dev/nvme0n1p3 /var
mkdir swap
mount -t btrfs -o defaults,noatime,subvol=swap /dev/nvme0n1p3 /swap/

• Some extra basic configurations:

ln -s /proc/self/mounts /etc/mtab
apt update
apt upgrade
apt install --yes locales console-setup
dpkg-reconfigure locales
dpkg-reconfigure tzdata
apt install --yes dpkg-dev linux-headers-amd64 linux-image-amd64
apt install --yes zfs-initramfs btrfs-progs dosfstools
echo REMAKE_INITRD=yes > /etc/dkms/zfs.conf

• Prepare /boot and efi/biosgpt partitions from normal sata HD:

mkfs.vfat /dev/sdd2 # efi
mkfs.ext4 /dev/sdd3 # /boot
mount /dev/sdd3 /mnt
cd /boot
mv * /mnt
umount /mnt
mkdir -p /boot/efi #efi
mount /dev/sdd3 /boot
mount /dev/sdd2 /boot/efi # efi
apt install --yes grub-pc # non-efi
apt install --yes grub-efi-amd64 shim-signed # efi
dpkg --purge os-prober
apt install emacs
cp /usr/share/systemd/tmp.mount /etc/systemd/system/
systemctl enable tmp.mount
grub-probe /boot
update-initramfs -c -k all
update-grub
grub-install /dev/sdd # non-efi
grub-install --target=x86_64-efi --efi-directory=/boot/efi \
  --bootloader-id=debian --recheck --no-floppy # efi
apt install --yes openssh-server

• Edit /etc/fstab:

LABEL=tardis					/mnt/btrfsbase btrfs defaults,noatime			0 0
LABEL=tardis   					/	       btrfs defaults,noatime,subvol=activeroot	0 0
LABEL=tardis   					/home	      btrfs defaults,noatime,subvol=home	0 0
LABEL=tardis   					/var	      btrfs defaults,noatime,subvol=var		0 0
LABEL=tardis   		      	    		/swap	      btrfs defaults,noatime,subvol=swap	0 0
UUID="11d65735-75d0-44ce-a158-abd01d98b318"	/boot	      ext4  defaults,noatime			0 0

zpool offline poolname device
zpool offline -t poolname device (until next reboot)

zpool online poolname device

zpool add poolname cache device #cache
zpool add poolname log device  #log

zpool remove poolname device

https://linuxhint.com/configuring-zfs-cache/

https://arstechnica.com/information-technology/2020/05/zfs-101-understanding-zfs-storage-and-performance/

https://www.howtoforge.com/tutorial/how-to-use-snapshots-clones-and-replication-in-zfs-on-linux/

https://docs.joyent.com/private-cloud/troubleshooting/disk-replacementhttps://docs.joyent.com/private-cloud/troubleshooting/disk-replacement

https://wiki.archlinux.org/index.php/btrfs#Subvolumes

https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Subvolumes

https://wiki.gentoo.org/wiki/Btrfs/Native_System_Root_Guide

https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Buster%20Root%20on%20ZFS.html