Linux Bare-Metal Server Backup SOP & Rapid Disaster Recovery

Managing dedicated physical bare-metal servers for high-throughput transactional databases or high-performance compute requires an enterprise-grade disaster recovery posture. When a motherboard fails, a RAID controller corrupts, or storage buses degrade, engineering teams cannot afford hours manually reinstalling operating system packages and reconfiguring kernel parameters.

This guide provides an actionable Standard Operating Procedure (SOP) for Linux bare-metal server backups, engineered to achieve a Recovery Time Objective (RTO) of under 45 minutes.

---

Decoupling System State from Application Data

A common architectural flaw is bundling the static operating system and dynamic transactional data into a single monolithic backup job. A verified bare-metal strategy decouples these concerns into two distinct execution paths:

+-------------------------------------------------------------------------------+
|                   BARE-METAL SERVER BACKUP ARCHITECTURE                       |
+-------------------------------------------------------------------------------+
|                                                                               |
|  [ PRODUCTION BARE-METAL SERVER ]                                             |
|         |                                                                     |
|         +---> PATH 1: OS State & Configuration (ReaR / Block Level)           |
|         |     - UEFI Bootloader, Root Partition /, /etc, Network Definitions  |
|         |     - Output: Bootable Rescue ISO + System Tar Archive              |
|         |     - Frequency: Weekly / On Configuration Changes                  |
|         |                                                                     |
|         +---> PATH 2: Stateful Application Tier (LVM Snapshot & DB Dumps)     |
|               - Relational Database (/var/lib/postgresql) & File Stores       |
|               - Output: Incremental Deduplicated Borg Repositories            |
|               - Frequency: Daily / Every 4 Hours                              |
|                                                                               |
|         v                                                                     |
|  [ STORAGE REPOSITORY: Local High-Throughput NFS/NAS + Offsite Cloud Vault ]  |
|                                                                               |
+-------------------------------------------------------------------------------+

---

Implementing Relax-and-Recover (ReaR)

Relax-and-Recover (ReaR) generates standalone bootable environments capable of re-partitioning raw hardware and restoring filesystems automatically.

1. Installation & Configuration (/etc/rear/local.conf)

# Install core packages on Debian/Ubuntu systems
apt-get install -y rear binutils isolinux syslinux-common

# Define the ReaR disaster recovery configuration
cat << 'EOF' > /etc/rear/local.conf
OUTPUT=ISO
OUTPUT_URL=nfs://192.168.10.100/mnt/backup/rear-iso
BACKUP=NETFS
BACKUP_URL=nfs://192.168.10.100/mnt/backup/rear-data
BACKUP_PROG_EXCLUDE=( '/tmp/*' '/var/tmp/*' '/var/lib/postgresql/*' '/mnt/*' )
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" btrfs gdisk )
EOF

2. Executing the Rescue Media & System Image Generation

# Generate the bootable recovery ISO and system archive
rear -v mkbackup

This command outputs a self-contained ISO containing the active kernel, modular device drivers, storage mappings, and network scripts required to reconstitute the physical server.

---

Consistent Database Backups via Atomic LVM Snapshots

For high-volume database engines, leverage Logical Volume Manager (LVM) snapshots to capture point-in-time state without database table locking:

#!/usr/bin/env bash
# /opt/scripts/backup-db-lvm.sh
set -euo pipefail

# 1. Place database in transactional backup mode
psql -U postgres -c "SELECT pg_backup_start('LVM_SNAPSHOT_BACKUP');"

# 2. Create the instantaneous LVM storage snapshot (1-2 seconds execution)
lvcreate -L 20G -s -n pg_snap /dev/vg_system/lv_postgres

# 3. Release the database transactional lock
psql -U postgres -c "SELECT pg_backup_stop();"

# 4. Mount the read-only snapshot volume and stream to deduplicated archive
mkdir -p /mnt/snapshot
mount -o ro /dev/vg_system/pg_snap /mnt/snapshot
borg create /mnt/nas-backup/borg::"{now}" /mnt/snapshot

# 5. Unmount and delete the temporary LVM snapshot
umount /mnt/snapshot
lvremove -y /dev/vg_system/pg_snap
echo "[$(date)] Consistent LVM database backup successfully archived."

---

Disaster Recovery Runbook: Bare-Metal Restore

When hardware failure strikes and a replacement chassis is mounted in the rack:

+-------------------------------------------------------------------------------+
|               BARE-METAL RESTORATION RUNBOOK (RTO < 45 MINUTES)               |
+-------------------------------------------------------------------------------+
|  1. Boot Target Node : Attach ReaR Rescue USB or initiate PXE network boot    |
|  2. Console Menu     : Select 'Recover [hostname]' on the bootstrap screen    |
|  3. Disk Partitioning: ReaR reconstructs GPT partition maps and LVM layouts   |
|  4. Filesystem Apply : ReaR extracts OS binaries, kernel configs, and drivers |
|  5. Initial Reboot   : Remove rescue media and reboot into the live OS        |
|  6. Database Hydrate : Pull the latest Borg snapshot to /var/lib/postgresql   |
|  7. Service Verify   : Confirm `systemctl status postgresql` & app endpoints  |
+-------------------------------------------------------------------------------+

---

Conclusion

Bare-metal infrastructure demands uncompromising recovery discipline. By pairing OS-level ReaR rescue generation with atomic LVM application snapshots, enterprise engineering teams transform severe physical hardware disasters into predictable, rapid restoration procedures.