Offsite backup using ssh btrfs and btrbk
In my a previous post, I showed how to create very efficient snapshots and versioned backups to hard drives installed in or attached to your server.
Now we have a time machine and some redundancy. If I delete a file by accident, I can recover it from an older snapshot and if a disk fails, we have several copies. The system even notices bit rot by checksumming.
But we still keep all our eggs in one basket. If lightning strikes, my entire server is gone and all the wonderful redundancy with it. We need offsite backup.
This can be accomplished in two ways. Either by physically attaching and detaching hard disks and carrying them to a different location or by sending the data to a computer located far away.
Btrbk can either push the data from the server to the backup-client or have the backup-client pull the data. Both variants have its upsides and downsides.
If I enable a backup-client to ssh into my server, an attacker could break something on my server. However that can be mitigated by only enabling read access over btrbk.
If I enable my server to ssh into my client, I would have to grant root-access with write privileges to my backup client. In case of malware infection, my offsite backup is my last line of defense. If the infected server has the keys, it could actively spread and damage even my offsite backup. That's why I decided to setup the backup-client as a visitor who comes by and reads data. The visitor can not be called from the server.
Btrbk is very well documented. I highly recommend reading the official docs. This is just my personal setup.
I have a RapsberryPi2 with two external hard drives attached at the USB Ports. That investment is relatively low and it runs pretty efficiently. One thing I had to learn was to get USB-Y cables so I can attach them to an extra USB-power source.
On the software side I'm running raspbian wheezy. There are plenty of tutorials how to set it up. No overclocking was necessary.
The external drives are mounted on
Now let's install btrbk:
$ mkdir /mnt/raid_backup/_btrbk $ sudo -i $ cd ~ $ wget https://digint.ch/download/btrbk/releases/btrbk-0.27.0.tar.xz $ tar xf btrbk-0.27.0.tar.xz $ cd btrbk-0.27.0/ $ make install $ cd /etc/btrbk/ $ cp btrbk.conf.example btrbk.conf
btrbk.conf so the raspberryPi does not create nor delete snapshots on my server.
transaction_log /var/log/btrbk.log snapshot_dir _btrbk_snap snapshot_create no ssh_identity /etc/btrbk/ssh/id_ed25519 ssh_user root ssh_port 1023 lockfile /var/lock/btrbk.lock snapshot_preserve_min all target_preserve_min 7d target_preserve 7d 4w *m archive_preserve_min latest archive_preserve 12m 10y volume ssh://server/mnt/btrfs_ssd subvolume @* target send-receive /mnt/raid_backup/_btrbk
Make sure your server is available on the internet. I'm behind a firewall and had to setup some port forwarding on the router. The above mentioned port 1023 will be forwarded to port 22 of my server.
Warning: We are permitting root login to your server. An attacker could cause serious damage. So make sure you use state of the art encryption. Have secure keys and keep them private.
Now we need to generate the ssh keys. Both my rasperryPi and my Ubuntu 18.04 support the modern ed25519 keys.
raspberrypi $ mkdir ssh raspberrypi $ ssh-keygen -t ed25519 -f /etc/btrbk/ssh/id_ed25519 raspberrypi $ cat /etc/btrbk/ssh/id_ed25519.pub
I don't put a password on the key because I want to do unattended backups that don't require me to enter a password ever time.
cat command shows the public key. We need to transfer that to our server. Some people use the tool ssh-copy-id, but I simply ssh into my server and copy paste it into
/root/authorized_keys on your
Let's try if the ssh connection works
raspberrypi $ ssh -i /etc/btrbk/ssh/id_ed25519 -p 1023 root@server
If it doesn't, take a look at the server's
/etc/sshd_config - maybe root login is prohibited, etc. I only support key-based authentication so I set the
# /etc/sshd_config on the server PasswordAuthentication no
This way nobody can try to guess the root-password. Also you might want to setup
fail2ban and simply ignore computers who tried doing this 3 times.
Don't forget to reload sshd_config by calling
service ssh restart
We can log into the server as root, let's make sure all we can do is execute btrbk commands. Btrbk even thought of this and provided us with this tool.
I copy the file
ssh_filter_btrbk.sh into /root/.ssh/ and modify
authorized_keys like this:
command="/root/.ssh/ssh_filter_btrbk.sh --send --info --restrict-path /mnt/btrfs_ssd" ssh-ed25519 5...PASTE_YOUR_PUBLIC_KEY_HERE...r72g root@raspberrypi
Now, even if a hacker gained access to our key, all he or she could do was copy data from the server. That would be terrible enough. But no deleting or malware installation is possible.
At first I do a dry run on the RaspberryPi
raspberrypi $ btrbk dryrun -------------------------------------------------------------------------------- Backup Summary (btrbk command line client, version 0.27.0) Date: Wed Dec 5 21:47:24 2018 Config: /etc/btrbk/btrbk.conf Dryrun: YES Legend: === up-to-date subvolume (source snapshot) +++ created subvolume (source snapshot) --- deleted subvolume *** received subvolume (non-incremental) >>> received subvolume (incremental) -------------------------------------------------------------------------------- server:/mnt/btrfs_ssd/@ *** /mnt/raid_backup/_btrbk/@.20181205 >>> /mnt/raid_backup/_btrbk/@.20181205_1 >>> /mnt/raid_backup/_btrbk/@.20181205_2 >>> /mnt/raid_backup/_btrbk/@.20181205_3 >>> /mnt/raid_backup/_btrbk/@.20181205_4 >>> /mnt/raid_backup/_btrbk/@.20181205_5 >>> /mnt/raid_backup/_btrbk/@.20181205_6 >>> /mnt/raid_backup/_btrbk/@.20181205_7 >>> /mnt/raid_backup/_btrbk/@.20181205_8 server:/mnt/btrfs_ssd/@home *** /email@example.com >>> /firstname.lastname@example.org_1 >>> /email@example.com_2 >>> /firstname.lastname@example.org_3 >>> /email@example.com_4 >>> /firstname.lastname@example.org_5 >>> /email@example.com_6 >>> /firstname.lastname@example.org_7 >>> /email@example.com_8 NOTE: Dryrun was active, none of the operations above were actually executed!
To automate this, I can either enable the systemd service or add a line to crontab on my raspberrypi and start the pull-process 10 minutes before midnight.
50 23 * * * root btrbk run -q