After installing a server and securing the operating system, I like to set up backup as early as possible. This prevents me from forgetting to create a backup at a later point in time or from losing previous configurations while setting things up.
For the sake of simplicity, I will refer to a system that creates backups and transfers them to another system as a “client” in this article. The central system on which the backups are stored is the “server”.
Because I use systems from different providers, I need a uniform backup system. This backup system must meet the following criteria:
- Secure data transfer - Data is transferred in untrustworthy environments/networks
- Push instead of pull - backups are actively transferred from the client to a central server, so the client can be behind a NAT
- Resistance to external access - Backups stored on the server can no longer be modified by the client
- Encryption - Backups are encrypted to enable safe storage
- Compression - Backups created are transmitted and stored in compressed form in order to save disk space
- Incremental - Only modified data should be backed up to reduce traffic and disk usage
- Deduplication - Backups and data in the backup are deduplicated to further save disk space
- Rotation - Old backups should be deleted on a specific and configurable schedule, for example annual, monthly and weekly backups are retained
- Restore individual files - Individual files can be restored from the backup
- Compatible with arm64 - In addition to amd64, arm64 systems such as Raspberry Pi should also be able to be backed up
- Automatic health checks - Health check of backups, whether they can be restored and whether there is any damage or repair is needed
After further research, I found and now stick to Borgbackup because it can meet all of my requirements. After getting in touch with it I had to admit, resistance is futile.
Borgbackup is an advanced backup tool which allows you to create secure, highly efficient backups of specific folders or your entire system.
BorgBackup (short: Borg) is a deduplicating backup program. Optionally, it supports compression and authenticated encryption.
The main goal of Borg is to provide an efficient and secure way to backup data. The data deduplication technique used makes Borg suitable for daily backups since only changes are stored. The authenticated encryption technique makes it suitable for backups to not fully trusted targets.
– Borg Documentation
A major limitation of Borg is the lack of support for Windows-based systems. The Linux Subsystem of Windows 10 can be used experimentally here, but I would not use it in productive environments.
Borg offers a lot of commands and functions that I won’t go into in this article. I’m focusing on the implementation in my environment and before you use Borg in your productive environment, you should read Quck Start and Frequently asked questions from the official Borg documentation.
In the following article I use some placeholders, which should be changed accordingly to your needs:
<client-identifier>- Unique client identification, preferably FQDN such as mywebserver1.bella.pm
<target-server>- target server, e.g. borgserver.bella.pm or 10.47.84.72
/mnt/backup/- destination folder for backups
<password for borg>- Repository password for encryption
At https://borgbackup.readthedocs.io/en/stable/installation.html there are several options for installing Borg.
In general, the version of Borg should be the same on every system used to avoid compatibility problems. Systems like Ubuntu & Debian have packages for Borg in the official repository, which I will also use here:
Borgbackup must be installed on both the client and the server.
Backups are created on the client as the root user so that all data on the system can be read. If this is not desired, the backup task can also be executed as a different user. Be aware that only files can be backed up which can be read by the selected user.
For the automated login to the server, a new Ed25519 key is created using the following command:
When entering a password, leave the input field empty and confirm with Enter.
To establish a connection to the server, the following content is added under
<target-server> with the IP address or host name of the server on which the backups are to be stored.
ServerAlive* parameters, unresponsive connections are closed after 10*30 seconds. This can occur with software problems or unstable connections.
Clients connect to this server to store backups, so this server is an interesting target for attackers and should be secured like any other server or even better.
As previously set up, clients establish an SSH connection for data transfer. Here we can use a simple user without extended rights.
The following command creates a new user named
remotebackup with a specific home
/mnt/backup/ in which backups are saved:
With the previous command, backups are stored at
/mnt/backup/. The folder can be adjusted accordingly and the destination should have enough space for all future backups.
For the next steps we will switch the user context to the newly created backup user:
In order to allow access via SSH and to enable access for clients, the folder
~/.ssh is created and the following content is added to the file
~/.ssh/authorized_keys (all in a single line!):
At the inserted line, replace
/mnt/backup/<client-identifier> with the desired target path where the backup should be stored and replace
ssh-ed25519 AAAA [...] with the content of
~/.ssh/id_ed25519_remotebackup.pub, which you generated on the client.
The inserted line contains several restrictions to the connecting client:
A client with the same SSH key only has append-only access to the repository stored at
/mnt/backup/<client-identifier>. All other SSH functions such as X11, port and agent forwarding are deactivated. Existing backups can neither be read, edited nor deleted.
Next, the Borg repository is created on the server as
There are several options for
- none - No encryption, not recommended
- repokey - key is stored on the server in the repo
- keyfile - the key is saved in the
$HOMEfolder on the client (must also be backed up, without the key on the server, backup is worthless!)
- authenticated (Borg 1.1+ only) - No encryption, but authentication with HMAC-SHA256 as with repokey & keyfile
- authenticated-blake2, repokey-blake2, keyfile-blake2 (Borg 1.1+ only) - Same as the previous options. Uses BLAKE2b-256 instead of SHA-256 as hash. Details
The following script is called once a day using crontab and can be stored under
/etc/cron.daily/borg-backup. The script is adapted for Debian / Ubuntu based systems, but can be adapted accordingly your system and specific requirements.
The script generates output to stdout and therefore generates an email every time it is executed.
Backups are created in the name format YYYYMMDD-HH such as 20210720-06.
Warning: Only the directories specified under
borg create will be backed up. This list should be maintained accordingly so that all application data is backed up as needed.
A separate MySQL user with read rights only should be created for the MySQL backup:
Automatic prune on server
Backups are usually not needed indefinitely and lose their value over time. Data loss should be as low as possible and a backup should therefore be created relatively often.
Borg offers the possibility to clean up backups based on a schedule and to store certain backups according to a defined time table.
The following script keeps the last 7 backups per day, then 4 weekly backups, 6 monthly backups and 1 annual backup.
The times are added up. 7 daily + 4 weeks results in 5 weeks, 6 monthly in addition results in 7 months of backups.
More details and a more detailed description can be found in the Borg documentation for borg prune.
The execution of the script can be automated using crontab.
Attention: Prune deletes backup data and a restore is not possible!
Attention: The script must not be executed at the same time as backup task, as this can lead to conflicts and can cause data corruption.
Backups are created for emergencies in order to be able to restore lost or damaged data. Since it may have to be quick and for the sake of completeness, the restoration process is also part of a backup. The following options are available for restoring individual data up to and including a complete backup.
The following command can be used to list all backups stored in a repository by name:
Details of a specific backup can be read out with the following command:
Using the command
borg diff you can list the changes between backups:
Mount as filesystem
Borg can mount a backup as a file system using fuse drivers. For this, the command
borg mount is used as follows:
This allows individual data to be easily extracted from a backup and the complete file structure at the time of the backup can be viewed.
The mounted file system can be unmounted again with the following command:
Extract files or folders
With the command
borg extract, either the complete backup is extracted in the current folder or the specific path can be specified.
You should therefore switch to a different folder before using the command. More details on extract
Use of extracts
Extracted data with both
extract have the same permissions and user/group IDs as when the backup was created. A program that takes over the authorizations during the transfer can be used to restore an executable copy.
rsync can be used for this:
Stats from my backups
I am currently backup up multiple servers, running locally and remotely over VPN.
Besides the server running this page and uDomainFlag, I am also backing up my Proxmox hosts in my homelab, Mailcow email server, my desktop and laptop devices, Dropbox and Syncthing (shown below) folders, Minecraft and Factorio servers, multiple Raspberry Pi (read-only storage to extend SD lifespan) and my tiny storage server (shown below) which includes Nextcloud.
The last backup consumed 7.23 MB effectively. Without deduplication and compression, the backup would consume 4.22 GB. In total of all backups of this client, 1.58 TB of storage would be required. Because of deduplication and compression only 24.79 GB are used.
I am using syncthing to synchronize my important daily data between clients. Every 3 hours, a backup using Borg is created on an independent host. As in the example above, the last backup consumed 6.22 MB and in total all backups would consume 859.13 GB of disk space. Instead, only 17.12 GB are used.
These 17.12 GB include 160 backups which reach from today July 2021 back to February 2018.
Finally, it is important to note that an untested backup is equivalent to a nonexistent backup.
The backup created should be tested regularly to see whether it can be restored. In addition, if it is functional, the backup should be transferred to another storage medium. The 3-2-1 rule should always be applied here:
- 3 copies of the data - 1 original file and 2 backup copies
- 2 different media - in addition to HDD / SSD also on tape, CD/DVD, network share, etc.
- 1 offsite - A backup copy should be kept at another location, such as a bank or at a trusted cloud provider (different to the server provider, if used)
The backup script described in this article only backs up specific folders and not the entire system. Therefore, only these folders can be restored. The script has to be adapted as required, so that your data is backed up as needed and isn’t skipped in the backup process.
The following links offer very interesting content on the topic:
Have fun with the setup and don’t forget to test the backup after setup.
In my experience, Borgbackup has never disappointed me and restores have worked without any problems. Even backups executed every 3 hours work without any problems and regular prune keeps the storage space consumption low.