Versioned backups with btrfs and btrbk
In my a previous post, I showed how to install Ubuntu 18.04 with btrfs filesystem. Now let's use the superpowers of btrfs and create awesome backups.
My setup: A server running Ubuntu 18.04 on a two ssd-drives in a btrfs raid1 configuration. In addition, there are two old-fashioned cheap hard drives installed. They are twice the size of the root partition, so I have plenty of space for backups.
Initially I wanted to use the commands
btrfs subvolume snapshot and the
btrfs send and
btrfs receive them to the hard drives. In fact I did just that for fun and it worked well. The btrfs-wiki-Article on incremental backups is a good starting point for getting familiar with this.
btrfs send in incremental mode can do so much more than
rsync. For example if you rename a directory. Good old rsync will think it's deleted and a new one is created. That results in lots of data to be transferred and stored. To
btrfs send the renamed directory is only a few bytes of meta-data.
Then I came across btrbk which seems to do just what I was trying to do with a bunch of scripts.
First, I need to format my hard drives with btrfs:
$ mkfs.btrfs -f -d raid1 -m raid1 /dev/sdc /dev/sdd -L backuphd btrfs-progs v4.15.1 See http://btrfs.wiki.kernel.org for more information. Label: backuphd UUID: bee8e186-3047-46e8-bc70-6faf89c0c3e5 Node size: 16384 Sector size: 4096 Filesystem size: 3.64TiB Block group profiles: Data: RAID1 1.00GiB Metadata: RAID1 1.00GiB System: RAID1 8.00MiB SSD detected: no Incompat features: extref, skinny-metadata Number of devices: 2 Devices: ID SIZE PATH 1 1.82TiB /dev/sdc 2 1.82TiB /dev/sdd root@linuxserver2:/mnt# btrfs fi sh Label: none uuid: 8c33495b-f558-11e8-8c48-ac1f6b156e58 Total devices 2 FS bytes used 3.06GiB devid 1 size 893.75GiB used 5.03GiB path /dev/sda2 devid 2 size 893.75GiB used 5.03GiB path /dev/sdb2 Label: 'backuphd' uuid: bee8e186-3047-46e8-bc70-6faf89c0c3e5 Total devices 2 FS bytes used 112.00KiB devid 1 size 1.82TiB used 2.01GiB path /dev/sdc devid 2 size 1.82TiB used 2.01GiB path /dev/sdd
Notice the UUID in the above output. We need that for fstab later.
Now I add this to my
UUID=86b86299-ba2d-40d5-9af3-69a5ee981994 /mnt/backuphd btrfs compress=lzo,noatime,nodiratime,space_cache,discard 0 2
Now let's create the mount point and mout the drive
$ cd /mnt/ $ mkdir backuphd $ mount backuphd
Now let's install btrbk
I did not have make, ruby or asciidoctor on my system, so I needed to install that. You can use the tarball of the release, but I prefer the github version.
$ sudo -i $ git clone https://github.com/digint/btrbk.git $ cd btrbk $ apt install make $ apt install ruby $ gem install asciidoctor $ make install $ cd /etc/btrbk/ $ cp btrbk.conf.example btrbk.conf
This is my running
btrbk.conf. Notice the
# before stream_buffer to comment it out. See the instructions https://digint.ch/btrbk/doc/btrbk.conf.5.html for details
transaction_log /var/log/btrbk.log #stream_buffer 512m snapshot_dir _btrbk_snap # Keep all snapshots for 2 days on ssd snapshot_preserve_min 2d # Keep 14 daily snapshots on ssd snapshot_preserve 14d target_preserve_min no # Keep 20 daily, 10 weekly and all monthly backups on /mnt/backuphd/ target_preserve 20d 10w *m # Useful for creating extra archive copies (clones) from your backup disks. # I don't use this, yet archive_preserve_min latest archive_preserve 12m 10y volume /mnt/btrfs_ssd subvolume @* target send-receive /mnt/backuphd/_btrbk
Btrbk does not create the directories. Let's take care of that.
$ mkdir /mnt/btrfs_ssd/_btrbk_snap $ mkdir /mnt/backuphd/_btrbk
And now we can run btrbk for the first time.
$ btrbk run
Now I have subvolumes containing all my data on both drives.
More commands are here https://digint.ch/btrbk/doc/btrbk.1.html
Now let's run this command every hour by adding this line to
50 * * * * root btrbk run -q
This should take care of versioning the files and retaining copies according to