zfs_root_cache
This is an old revision of the document!
Table of Contents
Btrfs for root and ZFS for data with ssd cache
We have the following hardware:
- 1x 512GB nvme
- 4x 3TB sata
- 2x 5TB sata
And we want the following:
- Root on 200GB of nvme in mirror with another sata
- Bios doesn't boot from nvme, then boot must be copied on all satas
- UEFI partition also in all satas
- A fast only nvme partition of 200GB
- Be able to do backups of important directories (/home, /etc, and others) on external HD, maybe using snapshots
- Use many of the satas together for a big long storage “data” space with the rest of nvme as cache
How can ZFS help here:
- ZFS can't be used as /boot directory. Use ext4
- Use ZIL and SLOG as write cache for nvme for data volume
- Use L2ARC as read cache for nvme for data volume
We will do this:
- A btrfs for root, swap and “fast” associated with the nvme alone (no redundancy, but /etc will be gitted)
- /boot and UEFI partitions in several disks for booting
- A pool for the rest of the disks using some level of RAIDz1, it will have a small (32GB) SLOG and a bigger(128GB) L2ARC from the nvme disk
- A pool in the backup disks for backuping using zfs znapshot/replication/clone
Steps followed
- Wipe nvme disk (use gdisk /dev/nvme0n1 (o command)
- Create SLOG 32GB partition with bf00 type
- Create L2ARC 128GB partition with bf00 type
- Create Btrfs partition on (rest of 512GB disk, leave a bit for ssd discarding) type 8300
- Create Btrfs filesystem on Btrfs partition:
sudo mkfs.btrfs -L tardis /dev/nvme0n1p3
- Create /, home, var, fast and “swap” subvolumes (swap is really just a special file)
sudo mkdir /mnt/btrfsbase sudo mount -t btrfs -o defaults,noatime /dev/nvme0n1p3 /mnt/btrfsbase sudo btrfs subvolume create /mnt/btrfsbase/activeroot sudo btrfs subvolume create /mnt/btrfsbase/home sudo btrfs subvolume create /mnt/btrfsbase/var sudo btrfs subvolume create /mnt/btrfsbase/fast sudo btrfs subvolume create /mnt/btrfsbase/swap sudo mkdir /mnt/newroot/ sudo mount -t btrfs -o defaults,noatime,subvol=activeroot /dev/nvme0n1p3 /mnt/newroot/
- Install the minimal system:
sudo debootstrap bullseye /mnt/newroot/
- Configure etc/hostname and etc/hosts correctly
- Configure etc/apt/sources.list and etc/network/interfaces correctly
- Bind mount virtual filesystems:
sudo mount --rbind /dev/ /mnt/newroot/dev/ sudo mount --rbind /proc/ /mnt/newroot/proc/ sudo mount --rbind /sys /mnt/newroot/sys/
- Chroot to the new /
sudo chroot /mnt/newroot
- Mount the rest of the btrfs filesystem volumes
mount -t btrfs -o defaults,noatime,subvol=home /dev/nvme0n1p3 /home/ mount -t btrfs -o defaults,noatime,subvol=var /dev/nvme0n1p3 /mnt/ cd /var mv * /mnt/ umount /mnt mount -t btrfs -o defaults,noatime,subvol=var /dev/nvme0n1p3 /var mkdir swap mount -t btrfs -o defaults,noatime,subvol=swap /dev/nvme0n1p3 /swap/
- Some extra basic configurations:
ln -s /proc/self/mounts /etc/mtab apt update apt upgrade apt install --yes locales dpkg-reconfigure locales dpkg-reconfigure tzdata apt install --yes dpkg-dev linux-headers-amd64 linux-image-amd64 apt install --yes zfs-initramfs btrfs-progs dosfstools echo REMAKE_INITRD=yes > /etc/dkms/zfs.conf
- Prepare /boot and efi/biosgpt partitions from normal sata HD:
mkfs.vfat /dev/sdd2 # efi mkfs.ext4 /dev/sdd3 # /boot mount /dev/sdd3 /mnt cd /boot mv * /mnt umount /mnt mkdir -p /boot/efi #efi mount /dev/sdd3 /boot mount /dev/sdd2 /boot/efi # efi apt install --yes grub-pc # non-efi apt install --yes grub-efi-amd64 shim-signed # efi dpkg --purge os-prober apt install emacs cp /usr/share/systemd/tmp.mount /etc/systemd/system/ systemctl enable tmp.mount grub-probe /boot update-initramfs -c -k all update-grub grub-install /dev/sdd # non-efi grub-install --target=x86_64-efi --efi-directory=/boot/efi \ --bootloader-id=debian --recheck --no-floppy # efi apt install --yes openssh-server
References
https://linuxhint.com/configuring-zfs-cache/
https://www.howtoforge.com/tutorial/how-to-use-snapshots-clones-and-replication-in-zfs-on-linux/
https://wiki.archlinux.org/index.php/btrfs#Subvolumes
https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Subvolumes
zfs_root_cache.1606129001.txt.gz · Last modified: 2021/02/01 05:55 (external edit)