Skip to content [Tab]
~/posts/devops/adding-backrest-restic-for-deduplicated-snapshots-and-historical-retention.md
$ stat -x adding-backrest-restic-for-deduplicated-snapshots-and-historical-retention.md POSIX filesystem
Access Mode: -rw-r--r-- (0644)
Author: uid:1000(@f7234bv9s8)
File Size: 16.2 KB (1304 words)
Modified:

Adding Backrest (Restic) for Deduplicated Snapshots and Historical Retention

attachment://featured-image.webp
Backrest

1. Why Semaphore Alone Is Not Enough

With Semaphore UI running pull_wp_backups.sh and pull_django_backups.sh, your Home Lab reliably pulls database dumps and file mirrors from your VPS fleet into /mnt/homelab_storage/backups/. However, a staging mirror by itself is not a true backupβ€”it only preserves a single point in time: the exact state of each website during the last pull.

Consider three real-world failure scenarios where a single mirror fails:

  1. Silent Data Corruption or Malware: A WordPress plugin introduces a subtle database corruption bug or injects malicious code on Tuesday, and you do not notice until Friday. By Friday, three nightly runs of rsync --delete and mv have overwritten your clean Monday files and SQL dump with the corrupted versions.
  2. Accidental Content Deletion: An editor deletes critical media directories or database records right before a scheduled backup runs, permanently removing those files from files_mirror/ via rsync --delete.
  3. Storage Bloat from Naive Archives: Creating uncompressed or .tar.gz copies of every website every single night quickly exhausts local hard drives, because standard compressed archives shift binary boundaries and prevent storage-level deduplication.

What Backrest (Restic) Solves

Backrest is a lightweight web interface and job scheduler built on top of Restic, an industry-standard, cryptographically secure backup engine. While Semaphore acts as your Orchestrator (extracting live data from remote Docker containers over SSH), Backrest acts as your Deduplicated Time Machine:

  • Block-Level Deduplication (Content-Defined Chunking): Restic splits every file in /mnt/homelab_storage/backups/ into variable-sized chunks and hashes them. When a new snapshot runs, Restic only writes newly created or modified chunks to disk. Keeping 30 days of historical snapshots for a 10 GB website typically consumes barely more disk space than a single copy.
  • Cross-Site Deduplication: Because your WordPress sites share identical core files, themes, and common plugins inside files_mirror/, a single Backrest repository stores those shared data blocks once across your entire fleet.
  • Uncompressed Staging Synergy: Because our Semaphore scripts write uncompressed .sql dumps and raw files_mirror/ directories rather than .tar.gz archives, Restic can inspect individual SQL rows and files directlyβ€”maximizing deduplication ratios and compressing the deduplicated blocks inside its own vault.
  • Encryption at Rest: Restic encrypts every chunk, index, and metadata file using AES-256 before writing it to disk, protecting sensitive database dumps and application configurations even if a physical drive is removed.
  • Automated Retention and Pruning: Backrest automatically enforces Grandfather-Father-Son retention policies (such as keeping 7 daily, 4 weekly, and 6 monthly snapshots) and purges expired data blocks so your storage pool never fills up unexpectedly.

Why Run Backrest as a Separate Container

Deploying Backrest in its own Docker container rather than bundling it into Semaphore provides clean operational isolation:

  • Independent Lifecycles: You can upgrade, restart, or troubleshoot Semaphore UI without interrupting scheduled Backrest repository checks or pruning jobs (and vice versa).
  • Privilege Separation: Semaphore runs as unprivileged user 1001 (semaphore) to execute SSH and rsync pulls, whereas Backrest runs with read access across the entire /mnt/homelab_storage/backups mount and exclusive write ownership over its encrypted repository volume (/data/restic_repo).

2. Deploying Backrest as a Separate Container

Step 1: Create Host Directories

Create a dedicated stack directory for Backrest’s configuration and local Restic repository on your Home Lab:

Bash

sudo mkdir -p /opt/backrest/backrest-config /opt/backrest/backrest-data

(Ensure /mnt/homelab_storage/backups already exists and is owned by 1001:0 with 775 permissions so Semaphore can write to it and Backrest can read every file inside it.)

Step 2: Create the Docker Compose Configuration

Create /opt/backrest/docker-compose.yml:

YAML

services:
  backrest:
    image: garethgeorge/backrest:latest
    container_name: backrest
    restart: unless-stopped
    ports:
      - "9898:9898"
    environment:
      - BACKREST_DATA=/data
      - BACKREST_CONFIG=/config/config.json
      - TZ=Europe/Kyiv
    volumes:
      - ./backrest-config:/config
      - ./backrest-data:/data
      - /mnt/homelab_storage/backups:/mnt/homelab_storage/backups:rslave

Why :rslave Mount Propagation Matters: Because /mnt/homelab_storage resides under /mnt/, an external drive, ZFS pool, mergerfs array, or network share may initialize slightly after the Docker daemon starts during a Home Lab reboot. Without :rslave, Docker pins the container to the underlying empty directory inode (readdirent: no such file or directory). Adding :rslave ensures host mount events propagate directly into the running backrest container.

Step 3: Start Backrest and Verify Mount Visibility

Launch the container from /opt/backrest:

Bash

cd /opt/backrest && docker compose up -d

Verify that Backrest can see your staged website folders inside its container namespace:

Bash

docker exec backrest ls -la /mnt/homelab_storage/backups

Confirm the output lists each SITE_NAME directory created by Semaphore rather than an empty directory.

3. Configuring the Encrypted Local Repository

  1. Open the Backrest Web UI in your browser at http://HOMELAB_IP:9898.
  2. On first launch, create your administrator username and password when prompted.
  3. In the left sidebar, navigate to Repos and click Add Repo.
  4. Configure the repository parameters:
    • Repo Name: homelab-local-vault
    • Type: Local
    • Repository Path: /data/restic_repo (this maps to /opt/backrest/backrest-data/restic_repo on your Home Lab host).
    • Password: Enter a strong encryption password (or click Generate).
  5. Click Test Configuration to verify Backrest can initialize the path, then click Save.

Critical Recovery Note: Back up this repository encryption password in an offline password manager. Restic encrypts all repository metadata and file chunks with this keyβ€”if you lose both /opt/backrest/backrest-config and this password, the backups inside /data/restic_repo cannot be decrypted.

4. Creating a Unified Multi-Site Backup Plan and Retention Policy

While Semaphore uses separate Variable Groups and Task Templates per website so you can run individual pre-update checkpoints, Backrest should use one unified Backup Plan covering the parent /mnt/homelab_storage/backups folder. This maximizes cross-site block deduplication while keeping every website cleanly separated into its own subfolder inside the snapshot tree.

Step 1: Configure the Backup Plan

  1. In the left sidebar, click Plans and select Add Plan.
  2. Configure the plan settings:
    • Plan Name: All-Websites-Daily
    • Repository: Select homelab-local-vault
    • Paths: Add /mnt/homelab_storage/backups
    • Excludes (Optional): Add *.tmp so transient database dump files are never captured if a manual Semaphore run overlaps with a snapshot.
  3. Set the Snapshot Schedule:
    • Configure the cron schedule to run after all staggered Semaphore pull tasks have finished. For example, if your Semaphore tasks run between 02:00 and 03:00, set Backrest’s cron schedule to 30 3 * * * (03:30 AM daily).

Step 2: Configure Grandfather-Father-Son Retention Rules

Inside the same Add Plan dialog, scroll to the Retention section to control how long snapshots are kept before Restic prunes old data blocks:

  • Policy Type: By Count (or Time Bucketed)
  • Keep Last: 7 (retains the 7 most recent snapshots regardless of age, protecting manual intraday runs)
  • Keep Daily: 14 (retains 1 snapshot per day for the past 2 weeks)
  • Keep Weekly: 4 (retains 1 snapshot per week for the past month)
  • Keep Monthly: 6 (retains 1 snapshot per month for half a year)

Click Save to activate the plan. Backrest automatically runs forget and prune cycles on schedule to reclaim disk space from expired snapshots without manual intervention.

5. Creating and Verifying Your First Snapshot

Step 1: Trigger an Initial Snapshot

  1. Open Plans in the Backrest UI, locate All-Websites-Daily, and click Backup Now.
  2. Monitor the active task bar at the top of the dashboard. Once finished, check the status badge:
    • A green Success badge confirms every file and SQL dump was read and deduplicated cleanly.
    • A yellow Warning badge (Partial backup) indicates Restic hit a permission denied error on one or more filesβ€”usually caused if rsync pulled 0600 files without the --chmod=Du=rwx,Dg=rx,Do=rx,Fu=rw,Fg=r,Fo=r flag covered in Article 1.

Step 2: Inspect the Snapshot Tree

  1. Click on your plan (All-Websites-Daily) or navigate to Snapshots in the left sidebar.
  2. Locate the new snapshot timestamp and click the Folder / Browse icon to open the Snapshot Browser.
  3. Expand the directory tree (mnt -> homelab_storage -> backups) to verify that every website directory appears with both its database/<db_name>.sql dump and files_mirror/ tree intact.

@f7234bv9s8 maintainer

Software engineer and writer at notes:own:vision:com.