bella.network Blog

Backups using Borgbackup

Create deduplicated, incremental, compressed and encrypted backups using borg. Easy manageable with fast restore process for directories or entire servers. Resistance is futile.
HDD, a usual place where your data is stored and is mostly cheaper today then SSDs at the same size.
HDD, a usual place where your data is stored and is mostly cheaper today then SSDs at the same size. (Photo by Azamat Esenaliev from Pexels)

After installing a server and securing the operating system, I like to set up backup as early as possible. This prevents me from forgetting to create a backup at a later point in time or from losing previous configurations while setting things up.

For the sake of simplicity, I will refer to a system that creates backups and transfers them to another system as a “client” in this article. The central system on which the backups are stored is the “server”.

Because I use systems from different providers, I need a uniform backup system. This backup system must meet the following criteria:

  • Secure data transfer - Data is transferred in untrustworthy environments/networks
  • Push instead of pull - backups are actively transferred from the client to a central server, so the client can be behind a NAT
  • Resistance to external access - Backups stored on the server can no longer be modified by the client
  • Encryption - Backups are encrypted to enable safe storage
  • Compression - Backups created are transmitted and stored in compressed form in order to save disk space
  • Incremental - Only modified data should be backed up to reduce traffic and disk usage
  • Deduplication - Backups and data in the backup are deduplicated to further save disk space
  • Rotation - Old backups should be deleted on a specific and configurable schedule, for example annual, monthly and weekly backups are retained
  • Restore individual files - Individual files can be restored from the backup
  • Compatible with arm64 - In addition to amd64, arm64 systems such as Raspberry Pi should also be able to be backed up
  • Automatic health checks - Health check of backups, whether they can be restored and whether there is any damage or repair is needed

After further research, I found and now stick to Borgbackup because it can meet all of my requirements. After getting in touch with it I had to admit, resistance is futile.

Borgbackup is an advanced backup tool which allows you to create secure, highly efficient backups of specific folders or your entire system.

BorgBackup (short: Borg) is a deduplicating backup program. Optionally, it supports compression and authenticated encryption.
The main goal of Borg is to provide an efficient and secure way to backup data. The data deduplication technique used makes Borg suitable for daily backups since only changes are stored. The authenticated encryption technique makes it suitable for backups to not fully trusted targets.
– Borg Documentation

A major limitation of Borg is the lack of support for Windows-based systems. The Linux Subsystem of Windows 10 can be used experimentally here, but I would not use it in productive environments.

Borg offers a lot of commands and functions that I won’t go into in this article. I’m focusing on the implementation in my environment and before you use Borg in your productive environment, you should read Quck Start and Frequently asked questions from the official Borg documentation.

In the following article I use some placeholders, which should be changed accordingly to your needs:

  • <client-identifier> - Unique client identification, preferably FQDN such as mywebserver1.bella.pm
  • <target-server> - target server, e.g. borgserver.bella.pm or 10.47.84.72
  • /mnt/backup/ - destination folder for backups
  • <password for borg> - Repository password for encryption

Installation

At https://borgbackup.readthedocs.io/en/stable/installation.html there are several options for installing Borg.
In general, the version of Borg should be the same on every system used to avoid compatibility problems. Systems like Ubuntu & Debian have packages for Borg in the official repository, which I will also use here:

1

root@Client ~ $ apt install borgbackup

Borgbackup must be installed on both the client and the server.

Client configuration

Backups are created on the client as the root user so that all data on the system can be read. If this is not desired, the backup task can also be executed as a different user. Be aware that only files can be backed up which can be read by the selected user.
For the automated login to the server, a new Ed25519 key is created using the following command:

1

root@Client ~ $ ssh-keygen -a 100 -t ed25519 -f ~/.ssh/id_ed25519_remotebackup

When entering a password, leave the input field empty and confirm with Enter.

To establish a connection to the server, the following content is added under ~/.ssh/config:

1
2
3
4
5
6

Host remotebackupserver
	Hostname <target-server>
	User remotebackup
	IdentityFile ~/.ssh/id_ed25519_remotebackup
	ServerAliveInterval 10
	ServerAliveCountMax 30

Replace <target-server> with the IP address or host name of the server on which the backups are to be stored.
With the ServerAlive* parameters, unresponsive connections are closed after 10*30 seconds. This can occur with software problems or unstable connections.

Server configuration

Clients connect to this server to store backups, so this server is an interesting target for attackers and should be secured like any other server or even better.
As previously set up, clients establish an SSH connection for data transfer. Here we can use a simple user without extended rights.
The following command creates a new user named remotebackup with a specific home /mnt/backup/ in which backups are saved:

1

root@Server ~ $ useradd --create-home --home-dir /mnt/backup remotebackup

With the previous command, backups are stored at /mnt/backup/. The folder can be adjusted accordingly and the destination should have enough space for all future backups. For the next steps we will switch the user context to the newly created backup user:

1

root@Server ~ $ sudo -i -u remotebackup

In order to allow access via SSH and to enable access for clients, the folder ~/.ssh is created and the following content is added to the file ~/.ssh/authorized_keys (all in a single line!):

1

command="borg serve --restrict-to-repository /mnt/backup/<client-identifier> --append-only",restrict ssh-ed25519 AAAA[...]

At the inserted line, replace /mnt/backup/<client-identifier> with the desired target path where the backup should be stored and replace ssh-ed25519 AAAA [...] with the content of ~/.ssh/id_ed25519_remotebackup.pub, which you generated on the client.

The inserted line contains several restrictions to the connecting client:
A client with the same SSH key only has append-only access to the repository stored at /mnt/backup/<client-identifier>. All other SSH functions such as X11, port and agent forwarding are deactivated. Existing backups can neither be read, edited nor deleted.

Next, the Borg repository is created on the server as remotebackup user:

1
2
3
4
5
6
7
8
9

remotebackup@Server ~ $ mkdir -p /mnt/backup/<client-identifier>
remotebackup@Server ~ $ borg init --encryption=repokey /mnt/backup/<client-identifier>
Enter new passphrase:
Enter same passphrase again:
Do you want your passphrase to be displayed for verification? [yN]:

IMPORTANT: you will need both KEY AND PASSPHRASE to access this repo!
Use "borg key export" to export the key, optionally in printable format.
Write down the passphrase. Store both at safe place(s).

There are several options for --encryption= attribute:

  • none - No encryption, not recommended
  • repokey - key is stored on the server in the repo
  • keyfile - the key is saved in the $HOME folder on the client (must also be backed up, without the key on the server, backup is worthless!)
  • authenticated (Borg 1.1+ only) - No encryption, but authentication with HMAC-SHA256 as with repokey & keyfile
  • authenticated-blake2, repokey-blake2, keyfile-blake2 (Borg 1.1+ only) - Same as the previous options. Uses BLAKE2b-256 instead of SHA-256 as hash. Details

Backup script

The following script is called once a day using crontab and can be stored under /etc/cron.daily/borg-backup. The script is adapted for Debian / Ubuntu based systems, but can be adapted accordingly your system and specific requirements.
The script generates output to stdout and therefore generates an email every time it is executed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

#!/bin/bash
# Script from https://blog.bella.network/backups-using-borgbackup/
# Version 2021-07-20 - Initial release

# Random delay to reduce target server load if multiple clients
# are configured to backup using cron.daily
# Sleep (1s - 1h)
sleep $[ ( $RANDOM % 3600 )  + 1 ]s

# Define password used for encryption
export BORG_PASSPHRASE='<password for borg>'
# Target server and repository path
REPOSITORY='ssh://remotebackupserver/~/<client-identifier>'

# MySQL database backup, user needs global read access
MYSQL_USER=mysqlbackupuser
MYSQL_PASS=mysqlbackuppassword

# Generate list of locally installed packages
mkdir -p /var/metadata || true
chmod 0600 /var/metadata
dpkg --get-selections > /var/metadata/software.list

# MySQL dump, non-locking and without further compression of files
# skip comments to reduce file changes between backups
# please be aware that this can use a big portion of disk space
mkdir -p /var/mysqldump || true
chmod 0600 /var/mysqldump
mysql -u$MYSQL_USER -p$MYSQL_PASS -s -r -N -e 'show databases' | while read dbname; do
	mysqldump -u$MYSQL_USER -p$MYSQL_PASS --compact --skip-comments --complete-insert --single-transaction "$dbname" > /var/mysqldump/"$dbname".sql
	chmod 0600 /var/mysqldump/"$dbname".sql
done

# Compression: 'none', 'lzma' or 'auto,lzma,X' to compress only files with lzma level X (0..9) which reduce in size using lz4
# Available: 'none', 'lz4', 'zstd,X' (1..3..22), 'zlib,X' (0..6..9), 'lzma,X' (0..6..9)
# On Borg 1.1.4+ prefer zstd as compression method
# More here:  https://borgbackup.readthedocs.io/en/stable/quickstart.html#backup-compression & https://manpages.debian.org/testing/borgbackup/borg-compression.1.en.html
borg create --verbose --filter AME --list --stats --show-rc --compression auto,zstd,6 --exclude-caches \
	--exclude '/home/*/.cache/*'     \
                                         \
	$REPOSITORY::'{now:%Y%m%d-%H}'   \
	/home                            \
	/etc                             \
	/var/mail                        \
	/var/www                         \
	/var/spool                       \
	/var/mysqldump                   \
	/var/metadata

rm /var/mysqldump/*

Backups are created in the name format YYYYMMDD-HH such as 20210720-06.
Warning: Only the directories specified under borg create will be backed up. This list should be maintained accordingly so that all application data is backed up as needed.

A separate MySQL user with read rights only should be created for the MySQL backup:

1
2
3
4

CREATE USER 'mysqlbackupuser'@'localhost' IDENTIFIED BY 'mysqlbackuppassword';
GRANT SELECT, RELOAD, LOCK TABLES, REPLICATION CLIENT, SHOW VIEW, EVENT, TRIGGER ON *.* TO 'mysqlbackupuser'@'localhost';
FLUSH PRIVILEGES;
QUIT;

Automatic prune on server

Backups are usually not needed indefinitely and lose their value over time. Data loss should be as low as possible and a backup should therefore be created relatively often.
Borg offers the possibility to clean up backups based on a schedule and to store certain backups according to a defined time table.

The following script keeps the last 7 backups per day, then 4 weekly backups, 6 monthly backups and 1 annual backup.
The times are added up. 7 daily + 4 weeks results in 5 weeks, 6 monthly in addition results in 7 months of backups.

More details and a more detailed description can be found in the Borg documentation for borg prune.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

#!/bin/bash
# Script from https://blog.bella.network/backups-using-borgbackup/
# Version 2021-07-20 - Initial release

# Only allow execution as remotebackup user
if [ "$EUID" -ne "$(id remotebackup -u)" ]; then
	echo "Please run as remotebackup user"
	exit 1
fi

# Skip encryption warning
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes
# Server and Client locations do not match because repo was created on backup server
export BORG_RELOCATED_REPO_ACCESS_IS_OK=yes

# In this example a repokey repository
echo "###### Pruning backup for <client-identifier> on $(date) ######"
export BORG_PASSPHRASE='<password for borg>'
borg prune -v /mnt/backup/<client-identifier> --list --show-rc --stats \
		--keep-daily 7 \
		--keep-weekly 4 \
		--keep-monthly 6 \
		--keep-yearly 1

# Verify consistency of repository
borg check /mnt/backup/<client-identifier>

echo "###### Pruning finished ######"

The execution of the script can be automated using crontab.
Attention: Prune deletes backup data and a restore is not possible!
Attention: The script must not be executed at the same time as backup task, as this can lead to conflicts and can cause data corruption.

1
2

root@Server ~ $ crontab -u remotebackup -e
1 0	* * *	~/borg-prune.sh

Restoring

Backups are created for emergencies in order to be able to restore lost or damaged data. Since it may have to be quick and for the sake of completeness, the restoration process is also part of a backup. The following options are available for restoring individual data up to and including a complete backup.

The following command can be used to list all backups stored in a repository by name:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

root@Server /mnt/backup $ borg list <client-identifier>
20200309-06                          Mon, 2020-03-09 06:25:23 [c2286e29afdaf14b599e695c440342837f319c270e577616f98c347fa4d236cd]
20200310-06                          Tue, 2020-03-10 06:25:21 [0ee5f2c415aa09a20871bf47244c605d810c7d47821bc257316079642e45a2ab]
20200311-06                          Wed, 2020-03-11 06:25:24 [50b77c55e2346a0cf09383ff2e877cd8c56413fabbbb0d88f4f7e153c8790814]
20200312-06                          Thu, 2020-03-12 06:25:23 [55fddf5d963249972168896a5525b1b1ec3d8afc3f117265eb151abf0c87ffe1]
20200313-06                          Fri, 2020-03-13 06:25:27 [e31451be099c15c3b1f0c38fdac640b0e7ec933e4e282a27aaead793dd77e71e]
20200314-06                          Sat, 2020-03-14 06:25:29 [5a4135761e7c85c90bc04deed5492058f6af190df68ad0942b03f0e1e5c117b4]
20200315-06                          Sun, 2020-03-15 06:25:28 [956c2d9eec2e46d2f931d5d7ae1477b08200895bff84646b3632f0cfe2ab467d]
20200316-06                          Mon, 2020-03-16 06:25:26 [0d8c973eaba01c98f6253c8f56b3be364f28035fb3f00b5f5c9a29df9b93588a]
20200317-06                          Tue, 2020-03-17 06:25:40 [2c964bce695e989547f5156ecc8d2eb9edfab1409bb3cf4e2660c78670bcee45]
20200318-06                          Wed, 2020-03-18 06:25:51 [9e79549c5a7bf117e1a5937785544c8b3bd114009c45c2c8edabd2e627674369]
20200319-06                          Thu, 2020-03-19 06:25:30 [c3489689694eaa0dddcf889865e18506f9ff03d2e7254420bf0b5ceec68ee5a9]
20200320-06                          Fri, 2020-03-20 06:25:37 [352eff48050d4456b6de2c680d3fab4378034afbc3dff3b19ff99b554ac1268c]
[...]

Details of a specific backup can be read out with the following command:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

root@Server /mnt/backup $ borg info <client-identifier>::20200314-06
Archive name: 20200314-06
Archive fingerprint: 5a4135761e7c85c90bc04deed5492058f6af190df68ad0942b03f0e1e5c117b4
Comment:
Hostname: WebServer
Username: root
Time (start): Sat, 2020-03-14 06:25:29
Time (end): Sat, 2020-03-14 06:31:14
Duration: 5 minutes 45.06 seconds
Number of files: 172741
Command line: /usr/bin/borg create --verbose --filter AME --list --stats --show-rc --compression auto,lzma --exclude-caches 'ssh://remotebackup/~/<client-identifier>::{now:%Y%m%d-%H}' / --exclude /dev --exclude /proc --exclude /sys --exclude /var/run --exclude /run --exclude /tmp --exclude /lost+found --exclude /mnt --exclude /media --exclude /root/.config/borg --exclude /var/cache/ --exclude /var/lib/apt/ --exclude /var/lib/dpkg/ --exclude /var/log/ --exclude '/home/*/.cache/*'
Utilization of maximum supported archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               11.22 GB              6.60 GB             65.57 MB
All archives:                4.32 TB              2.47 TB             72.65 GB

                       Unique chunks         Total chunks
Chunk index:                  535447             63613813

Using the command borg diff you can list the changes between backups:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

root@Server ~ $ borg diff webserver.bella.ml::20200313-06 20200314-06
 +15.4 kB  -15.0 kB root/.bash_history
  +1.6 MB   -1.6 MB root/.acme.sh/acme.sh.log
  +1.1 kB   -1.1 kB root/.acme.sh/http.header
   +512 B    -512 B var/lib/systemd/random-seed
  +1.5 kB   -1.5 kB var/lib/logrotate/status
[...]
added           0 B var/lib/php/sessions/sess_r92kqpom1100kqis7iv6ajii2u
[...]
removed         0 B var/lib/php/sessions/sess_5sbvpm47er61papesubm80b8jo
removed         0 B var/lib/php/sessions/sess_qjk3fpooognssaigf7rgtj9fbm

Mount as filesystem

Borg can mount a backup as a file system using fuse drivers. For this, the command borg mount is used as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

root@Server /mnt/backup $ borg mount <client-identifier>::20200313-06 /mnt/borg/
root@Server /mnt/backup $ ll /mnt/borg/
total 11K
drwxr-xr-x  1 root root  0 Mar 14 12:44 .
drwxr-xr-x  1 root root  0 Mar 14 12:44 .
drwxr-xr-x 12 root root 14 Apr 24  2020 ..
drwxr-xr-x  1 root root  0 Feb 17  2020 bin
drwxr-xr-x  1 root root  0 Mar 11  2020 boot
drwxr-xr-x  1 root root  0 Mar 11  2020 etc
drwxr-xr-x  1 root root  0 Jan 18  2020 home
lrwxrwxrwx  1 root root 33 Feb 17  2020 initrd.img -> boot/initrd.img-4.15.0-88-generic
lrwxrwxrwx  1 root root 33 Feb 17  2020 initrd.img.old -> boot/initrd.img-4.15.0-76-generic
drwxr-xr-x  1 root root  0 Feb  2  2020 lib
drwxr-xr-x  1 root root  0 Jan 18  2020 lib64
drwxr-xr-x  1 root root  0 Feb  2  2020 opt
drwx------  1 root root  0 Mar 11  2020 root
drwxr-xr-x  1 root root  0 Feb 17  2020 sbin
drwxr-xr-x  1 root root  0 Jan 18  2020 srv
drwxr-xr-x  1 root root  0 Jan 18  2020 usr
drwxr-xr-x  1 root root  0 Mar  8  2020 var
lrwxrwxrwx  1 root root 30 Feb 17  2020 vmlinuz -> boot/vmlinuz-4.15.0-88-generic
lrwxrwxrwx  1 root root 30 Feb 17  2020 vmlinuz.old -> boot/vmlinuz-4.15.0-76-generic

This allows individual data to be easily extracted from a backup and the complete file structure at the time of the backup can be viewed.
The mounted file system can be unmounted again with the following command:

1

root@Server /mnt/backup $ borg umount /mnt/borg

Extract files or folders

With the command borg extract, either the complete backup is extracted in the current folder or the specific path can be specified.
You should therefore switch to a different folder before using the command. More details on extract

1
2
3
4
5
6

root@Server /mnt/backup $ borg extract <client-identifier>::20200313-06 var/vhosts/cron/data.json
root@Server /mnt/backup $ tree var
var
└── vhosts
    └── cron
        └── data.json

Use of extracts

Extracted data with both mount and extract have the same permissions and user/group IDs as when the backup was created. A program that takes over the authorizations during the transfer can be used to restore an executable copy.
For example, rsync can be used for this:

1

rsync -vaP /mnt/borg/etc/nginx/ root@<server-to-restore>:/etc/nginx/

Stats from my backups

I am currently backup up multiple servers, running locally and remotely over VPN.
Besides the server running this page and uDomainFlag, I am also backing up my Proxmox hosts in my homelab, Mailcow email server, my desktop and laptop devices, Dropbox and Syncthing (shown below) folders, Minecraft and Factorio servers, multiple Raspberry Pi (read-only storage to extend SD lifespan) and my tiny storage server (shown below) which includes Nextcloud.

Storage server

1
2
3
4
5
6

                       Original size      Compressed size    Deduplicated size
This archive:                4.22 GB              2.47 GB              7.23 MB
All archives:                1.58 TB            259.21 GB             24.79 GB

                       Unique chunks         Total chunks
Chunk index:                  281494              4510398

The last backup consumed 7.23 MB effectively. Without deduplication and compression, the backup would consume 4.22 GB. In total of all backups of this client, 1.58 TB of storage would be required. Because of deduplication and compression only 24.79 GB are used.

Syncthing folder

1
2
3
4
5
6

                       Original size      Compressed size    Deduplicated size
This archive:                5.92 GB              5.48 GB              6.22 MB
All archives:              859.13 GB            786.93 GB             17.12 GB

                       Unique chunks         Total chunks
Chunk index:                  122793             11489780

I am using syncthing to synchronize my important daily data between clients. Every 3 hours, a backup using Borg is created on an independent host. As in the example above, the last backup consumed 6.22 MB and in total all backups would consume 859.13 GB of disk space. Instead, only 17.12 GB are used.
These 17.12 GB include 160 backups which reach from today July 2021 back to February 2018.

Conclusion

Finally, it is important to note that an untested backup is equivalent to a nonexistent backup.
The backup created should be tested regularly to see whether it can be restored. In addition, if it is functional, the backup should be transferred to another storage medium. The 3-2-1 rule should always be applied here:

  • 3 copies of the data - 1 original file and 2 backup copies
  • 2 different media - in addition to HDD / SSD also on tape, CD/DVD, network share, etc.
  • 1 offsite - A backup copy should be kept at another location, such as a bank or at a trusted cloud provider (different to the server provider, if used)

The backup script described in this article only backs up specific folders and not the entire system. Therefore, only these folders can be restored. The script has to be adapted as required, so that your data is backed up as needed and isn’t skipped in the backup process.

The following links offer very interesting content on the topic:

Have fun with the setup and don’t forget to test the backup after setup.
In my experience, Borgbackup has never disappointed me and restores have worked without any problems. Even backups executed every 3 hours work without any problems and regular prune keeps the storage space consumption low.