Noah's Ark [(new, WIP)]

This Document is all about my backing up strategy.

Storage Mediums

Three physical storage mediums are used across the backup infrastructure. Each has a defined role and is not interchangeable.

NVMe (Ephemeral / Fast Scratch)

NVMe drives are used exclusively for operating system volumes, VM boot disks, and short-lived cache. They are explicitly not a backup target. TrueNAS and other storage VMs may have NVMe visibility for performance purposes, but no long-term backup data is written here.

• ✅ OS & boot volumes
• ✅ Cache and write-intent logs
• ❌ Long-term backup data
• ❌ Replicated datasets

HDD (Primary Backup Storage)

Spinning hard drives are the primary medium for all backup workloads. The current deployment is a 2×4 TB external HDD enclosure attached to the Proxmox Backup Server VM.

• ✅ PBS datastores
• ✅ TrueNAS bulk datasets
• ✅ Cold archive copies
• Higher capacity-to-cost ratio makes this the default for retention-heavy tiers

S3 (Offsite Object Storage)

S3 is the offsite layer of the backup strategy. It provides geographical separation from the physical homelab. Used for encrypted, versioned remote copies synced from local backup services.

• ✅ Offsite DR copy (Gold and Platinum tiers)
• ✅ Immutable/versioned bucket policies for ransomware resistance
• ✅ Lifecycle rules to tier cold data automatically
• ❌ Not a primary backup destination — always a sync target from local

Storage Services

Proxmox Backup Server (PBS)

PBS runs as a VM on Proxmox, managed with HA between pve-1 and pve-2 via Ceph. The VM has:

• 32 GB NVMe (OS/config disk — Ceph-backed for HA migration)
• 2×4 TB HDD via external enclosure (backup datastore — direct-attached to pve-2)

⚠️ Known HA Limitation — Datastore Availability After Failover

Problem: The HDD enclosure is physically connected to pve-2. If pve-2 fails and the PBS VM migrates to pve-1, the VM starts successfully but the /datastore mount is unavailable because the drives are no longer accessible. PBS enters a degraded state.

Assessed options:

Current recommendation: Implement the iSCSI approach as a medium-term fix. Target the HDD enclosure from pve-2 using tgt, connect the PBS VM via an iSCSI initiator, and mount as a block device. This makes the datastore network-addressable and removes the physical attachment constraint.

PBS Datastore Layout (Planned)

/mnt/datastore/
├── primary/        # Main backup target for all tiers
└── archive/        # Cold copies, infrequently pruned

TrueNAS (VM on Proxmox)

TrueNAS runs as a VM and provides ZFS-backed network storage for the homelab. Its role in the backup strategy is:

• SMB/NFS share target for file-level backups from workstations and containers
• ZFS snapshot source — periodic snapshots replicated to PBS or S3
• Staging area for large datasets before S3 upload

TrueNAS is itself a backup source as well as a storage service — its ZFS datasets must be included in backup tier assignments.

Backup Tier System

The tier system assigns a protection level to each service, VM, or dataset. Higher tiers mean more frequent backups, longer retention, more redundancy, and verified restores. Assign tiers based on criticality and acceptable data-loss window.

🥉 Bronze — Local Snapshot, Best-Effort

Use for: Non-critical VMs, dev/test environments, disposable workloads.
RPO (Recovery Point Objective): Up to 7 days
RTO (Recovery Time Objective): Best-effort, no SLA
Storage: PBS local datastore only
Offsite: None

Suitable for:

• Scratch VMs and containers
• Build/CI agents that can be reprovisioned from code
• Dev databases with no production data

🥈 Silver — Daily Local Backup, Short Retention

Use for: Standard homelab services — important but recoverable within a day.
RPO: 24 hours
RTO: < 4 hours
Storage: PBS local datastore + TrueNAS ZFS snapshots
Offsite: None

Suitable for:

• Standard self-hosted services (dashboards, monitoring stacks)
• Personal media servers
• Non-production databases
• TrueNAS datasets containing media/documents

🥇 Gold — Daily Local + S3 Offsite, Extended Retention

Use for: Important services with irreplaceable or hard-to-recreate data.
RPO: 24 hours
RTO: < 2 hours (local); < 8 hours (from S3)
Storage: PBS local datastore + S3 sync
Offsite: ✅ S3 (encrypted, versioned)

S3 Sync Method:

# Using proxmox-backup-client or rclone for datastore sync
rclone sync /mnt/datastore/primary s3:homelab-backups/primary \
  --s3-server-side-encryption AES256 \
  --transfers 4 \
  --log-file /var/log/rclone-sync.log

Suitable for:

• PBS VM itself (meta-backup of the backup service)
• TrueNAS configuration and critical datasets
• Identity/auth services (e.g. Authentik, LLDAP)
• Home automation state (Home Assistant)
• Network config (MikroTik export backups)

💎 Platinum — Frequent Local + Offsite + Verified, Maximum Retention

Use for: Critical services where data loss or extended downtime is unacceptable.
RPO: 4 hours
RTO: < 1 hour (local)
Storage: PBS local datastore + TrueNAS ZFS replication + S3 offsite
Offsite: ✅ S3 (encrypted, immutable object lock)

Restore Drill Procedure:

1. Spin up an isolated VLAN or test namespace on Proxmox
2. Restore latest backup from PBS to test VM
3. Validate service health (HTTP check, DB query, or equivalent)
4. Log result in homelab runbook with timestamp
5. Destroy test VM and clean up

Suitable for:

• Password manager (Vaultwarden)
• Certificate authority and PKI data
• Any VM storing financial records or irreplaceable personal documents
• DNS/DHCP config that underpins the entire network
• Wedding and personal photo archives

Tier Assignment Register

3-2-1 Compliance by Tier

A useful sanity check — the classic 3-2-1 rule states: 3 copies of data, on 2 different media types, with 1 offsite.

Key Outstanding TODOs

• Validate iSCSI datastore approach for PBS HA failover (see PBS section)
• Assign tiers to all running VMs and containers
• Configure S3 bucket with versioning + object lock for Gold/Platinum
• Implement rclone sync job and schedule via systemd timer or cron
• Write automated restore drill script for Platinum tier
• Document manual recovery runbook for PBS datastore unavailability
• Review PBS encryption key backup — keys must be stored independently of the PBS datastore (e.g. printed, in Vaultwarden, and in S3)