work-work.work-logo: A man with brains and a shovel

work-work.work

A blog about goals and obstacles, motivation and procrastination, life's random events and getting things done.

Offsite backup using ssh btrfs and btrbk

In my a previous post, I showed how to create very efficient snapshots and versioned backups to hard drives installed in or attached to your server.

Now we have a time machine and some redundancy. If I delete a file by accident, I can recover it from an older snapshot and if a disk fails, we have several copies. The system even notices bit rot by checksumming.

But we still keep all our eggs in one basket. If lightning strikes, my entire server is gone and all the wonderful redundancy with it. We need offsite backup.

This can be accomplished in two ways. Either by physically attaching and detaching hard disks and carrying them to a different location or by sending the data to a computer located far away.

Btrbk can either push the data from the server to the backup-client or have the backup-client pull the data. Both variants have its upsides and downsides.

If I enable a backup-client to ssh into my server, an attacker could break something on my server. However that can be mitigated by only enabling read access over btrbk.

If I enable my server to ssh into my client, I would have to grant root-access with write privileges to my backup client. In case of malware infection, my offsite backup is my last line of defense. If the infected server has the keys, it could actively spread and damage even my offsite backup. That's why I decided to setup the backup-client as a visitor who comes by and reads data. The visitor can not be called from the server.

Btrbk is very well documented. I highly recommend reading the official docs. This is just my personal setup.

I have a RapsberryPi2 with two external hard drives attached at the USB Ports. That investment is relatively low and it runs pretty efficiently. One thing I had to learn was to get USB-Y cables so I can attach them to an extra USB-power source.

On the software side I'm running raspbian wheezy. There are plenty of tutorials how to set it up. No overclocking was necessary.

The external drives are mounted on /mnt/raid_backup

Now let's install btrbk:

$ mkdir /mnt/raid_backup/_btrbk
$ sudo -i
$ cd ~
$ wget https://digint.ch/download/btrbk/releases/btrbk-0.27.0.tar.xz
$ tar xf btrbk-0.27.0.tar.xz
$ cd btrbk-0.27.0/
$ make install
$ cd /etc/btrbk/
$ cp btrbk.conf.example btrbk.conf

Now edit btrbk.conf so the raspberryPi does not create nor delete snapshots on my server.

transaction_log            /var/log/btrbk.log

snapshot_dir               _btrbk_snap
snapshot_create            no

ssh_identity               /etc/btrbk/ssh/id_ed25519
ssh_user                   root
ssh_port                   1023

lockfile                   /var/lock/btrbk.lock

snapshot_preserve_min   all
target_preserve_min     7d
target_preserve         7d 4w *m
archive_preserve_min    latest
archive_preserve        12m 10y

volume ssh://server/mnt/btrfs_ssd
  subvolume  @*
    target send-receive      /mnt/raid_backup/_btrbk

Make sure your server is available on the internet. I'm behind a firewall and had to setup some port forwarding on the router. The above mentioned port 1023 will be forwarded to port 22 of my server.

Warning: We are permitting root login to your server. An attacker could cause serious damage. So make sure you use state of the art encryption. Have secure keys and keep them private.

Now we need to generate the ssh keys. Both my rasperryPi and my Ubuntu 18.04 support the modern ed25519 keys.

raspberrypi $ mkdir ssh
raspberrypi $ ssh-keygen -t ed25519 -f /etc/btrbk/ssh/id_ed25519
raspberrypi $ cat /etc/btrbk/ssh/id_ed25519.pub

I don't put a password on the key because I want to do unattended backups that don't require me to enter a password ever time.

The cat command shows the public key. We need to transfer that to our server. Some people use the tool ssh-copy-id, but I simply ssh into my server and copy paste it into /root/authorized_keys on your server.

Let's try if the ssh connection works

raspberrypi $ ssh -i /etc/btrbk/ssh/id_ed25519 -p 1023 root@server

If it doesn't, take a look at the server's /etc/sshd_config - maybe root login is prohibited, etc. I only support key-based authentication so I set the

# /etc/sshd_config on the server
PasswordAuthentication no

This way nobody can try to guess the root-password. Also you might want to setup fail2ban and simply ignore computers who tried doing this 3 times.

Don't forget to reload sshd_config by calling service ssh restart

We can log into the server as root, let's make sure all we can do is execute btrbk commands. Btrbk even thought of this and provided us with this tool.

I copy the file ssh_filter_btrbk.sh into /root/.ssh/ and modify authorized_keys like this:

command="/root/.ssh/ssh_filter_btrbk.sh --send --info --restrict-path /mnt/btrfs_ssd" ssh-ed25519 5...PASTE_YOUR_PUBLIC_KEY_HERE...r72g root@raspberrypi

Now, even if a hacker gained access to our key, all he or she could do was copy data from the server. That would be terrible enough. But no deleting or malware installation is possible.

At first I do a dry run on the RaspberryPi

raspberrypi $ btrbk dryrun
--------------------------------------------------------------------------------
Backup Summary (btrbk command line client, version 0.27.0)

    Date:   Wed Dec  5 21:47:24 2018
    Config: /etc/btrbk/btrbk.conf
    Dryrun: YES

Legend:
    ===  up-to-date subvolume (source snapshot)
    +++  created subvolume (source snapshot)
    ---  deleted subvolume
    ***  received subvolume (non-incremental)
    >>>  received subvolume (incremental)
--------------------------------------------------------------------------------
server:/mnt/btrfs_ssd/@
*** /mnt/raid_backup/_btrbk/@.20181205
>>> /mnt/raid_backup/_btrbk/@.20181205_1
>>> /mnt/raid_backup/_btrbk/@.20181205_2
>>> /mnt/raid_backup/_btrbk/@.20181205_3
>>> /mnt/raid_backup/_btrbk/@.20181205_4
>>> /mnt/raid_backup/_btrbk/@.20181205_5
>>> /mnt/raid_backup/_btrbk/@.20181205_6
>>> /mnt/raid_backup/_btrbk/@.20181205_7
>>> /mnt/raid_backup/_btrbk/@.20181205_8

server:/mnt/btrfs_ssd/@home
*** /mnt/raid_backup/_btrbk/@home.20181205
>>> /mnt/raid_backup/_btrbk/@home.20181205_1
>>> /mnt/raid_backup/_btrbk/@home.20181205_2
>>> /mnt/raid_backup/_btrbk/@home.20181205_3
>>> /mnt/raid_backup/_btrbk/@home.20181205_4
>>> /mnt/raid_backup/_btrbk/@home.20181205_5
>>> /mnt/raid_backup/_btrbk/@home.20181205_6
>>> /mnt/raid_backup/_btrbk/@home.20181205_7
>>> /mnt/raid_backup/_btrbk/@home.20181205_8

NOTE: Dryrun was active, none of the operations above were actually executed!

To automate this, I can either enable the systemd service or add a line to crontab on my raspberrypi and start the pull-process 10 minutes before midnight.

50 23    * * *   root    btrbk run -q