Versioned backups with btrfs and btrbk
In my a previous post, I showed how to install Ubuntu 18.04 with btrfs filesystem. Now let's use the superpowers of btrfs and create awesome backups.
My setup: A server running Ubuntu 18.04 on a two ssd-drives in a btrfs raid1 configuration. In addition, there are two old-fashioned cheap hard drives installed. They are twice the size of the root partition, so I have plenty of space for backups.
Initially I wanted to use the commands btrfs subvolume snapshot and the btrfs send and btrfs receive them to the hard drives. In fact I did just that for fun and it worked well. The btrfs-wiki-Article on incremental backups is a good starting point for getting familiar with this.
btrfs send in incremental mode can do so much more than rsync. For example if you rename a directory. Good old rsync will think it's deleted and a new one is created. That results in lots of data to be transferred and stored. To btrfs send the renamed directory is only a few bytes of meta-data.
Then I came across btrbk which seems to do just what I was trying to do with a bunch of scripts.
First, I need to format my hard drives with btrfs:
$ mkfs.btrfs -f -d raid1 -m raid1 /dev/sdc /dev/sdd -L backuphd
btrfs-progs v4.15.1
See http://btrfs.wiki.kernel.org for more information.
Label: backuphd
UUID: bee8e186-3047-46e8-bc70-6faf89c0c3e5
Node size: 16384
Sector size: 4096
Filesystem size: 3.64TiB
Block group profiles:
Data: RAID1 1.00GiB
Metadata: RAID1 1.00GiB
System: RAID1 8.00MiB
SSD detected: no
Incompat features: extref, skinny-metadata
Number of devices: 2
Devices:
ID SIZE PATH
1 1.82TiB /dev/sdc
2 1.82TiB /dev/sdd
root@linuxserver2:/mnt# btrfs fi sh
Label: none uuid: 8c33495b-f558-11e8-8c48-ac1f6b156e58
Total devices 2 FS bytes used 3.06GiB
devid 1 size 893.75GiB used 5.03GiB path /dev/sda2
devid 2 size 893.75GiB used 5.03GiB path /dev/sdb2
Label: 'backuphd' uuid: bee8e186-3047-46e8-bc70-6faf89c0c3e5
Total devices 2 FS bytes used 112.00KiB
devid 1 size 1.82TiB used 2.01GiB path /dev/sdc
devid 2 size 1.82TiB used 2.01GiB path /dev/sdd
Notice the UUID in the above output. We need that for fstab later.
Now I add this to my /etc/fstab
UUID=86b86299-ba2d-40d5-9af3-69a5ee981994 /mnt/backuphd btrfs compress=lzo,noatime,nodiratime,space_cache,discard 0 2
Now let's create the mount point and mout the drive
$ cd /mnt/
$ mkdir backuphd
$ mount backuphd
Now let's install btrbk
I did not have make, ruby or asciidoctor on my system, so I needed to install that. You can use the tarball of the release, but I prefer the github version.
$ sudo -i
$ git clone https://github.com/digint/btrbk.git
$ cd btrbk
$ apt install make
$ apt install ruby
$ gem install asciidoctor
$ make install
$ cd /etc/btrbk/
$ cp btrbk.conf.example btrbk.conf
This is my running btrbk.conf. Notice the # before stream_buffer to comment it out. See the instructions https://digint.ch/btrbk/doc/btrbk.conf.5.html for details
transaction_log /var/log/btrbk.log
#stream_buffer 512m
snapshot_dir _btrbk_snap
# Keep all snapshots for 2 days on ssd
snapshot_preserve_min 2d
# Keep 14 daily snapshots on ssd
snapshot_preserve 14d
target_preserve_min no
# Keep 20 daily, 10 weekly and all monthly backups on /mnt/backuphd/
target_preserve 20d 10w *m
# Useful for creating extra archive copies (clones) from your backup disks.
# I don't use this, yet
archive_preserve_min latest
archive_preserve 12m 10y
volume /mnt/btrfs_ssd
subvolume @*
target send-receive /mnt/backuphd/_btrbk
Btrbk does not create the directories. Let's take care of that.
$ mkdir /mnt/btrfs_ssd/_btrbk_snap
$ mkdir /mnt/backuphd/_btrbk
And now we can run btrbk for the first time.
$ btrbk run
Now I have subvolumes containing all my data on both drives.
More commands are here https://digint.ch/btrbk/doc/btrbk.1.html
Now let's run this command every hour by adding this line to /etc/crontab.
50 * * * * root btrbk run -q
This should take care of versioning the files and retaining copies according to /etc/btrbk/btrbk.conf