Backup Plan

Backup Plan

This page replaces the original TODO with an actionable backup plan for the server that hosts wiki.mgaif.com, CloudPanel, Hermes Agent, and related applications. The goal is to produce one compressed archive that can be moved off-server, plus a checksum and manifests that make a rebuild less dependent on memory.

Executive summary

• Use the script at /home/mediawiki/tools/mgaif_server_backup.sh.
• Run it as root. It stages data under /var/backups/mgaif-server and creates a single .tar.zst archive plus a .sha256 checksum.
• The archive includes selected /home contents while explicitly excluding /home/.swap, all of /home/clp, /home/mysql, /home/linuxbrew, /home/mgaif-* logs/backups, and nginx log directories. CloudPanel-managed databases are still exported through clpctl into the backup staging area. Hermes data under /root/.hermes is included except logs, caches, sessions, the Hermes Agent checkout, and Web UI sessions.
• CloudPanel-managed databases are exported through clpctl db:export.
• Any additional non-system MySQL databases are exported with mysqldump after reading CloudPanel’s master database credentials in-process.
• The script performs conservative cleanup of old backup staging directories under /home only when they carry this workflow’s .mgaif-backup-staging marker.
• The archive contains secrets: database dumps, Hermes configuration, bot credentials, TLS material if present, and application environment files. Treat it as sensitive, transfer it over SSH/SFTP/rsync to encrypted off-server storage, and avoid leaving the only copy on this host.

Runbook

Pre-flight check

sudo /home/mediawiki/tools/mgaif_server_backup.sh --dry-run

The dry run should report the staging directory, output archive path, the CloudPanel databases that would be exported, and the major included/excluded paths. It should mention the selective /home scope and the /home/mediawiki changed-file rule.

Create the backup

sudo /home/mediawiki/tools/mgaif_server_backup.sh

By default, output files are written to /var/backups/mgaif-server/:

/var/backups/mgaif-server/mgaif-server-backup-YYYYMMDDTHHMMSSZ.tar.zst
/var/backups/mgaif-server/mgaif-server-backup-YYYYMMDDTHHMMSSZ.tar.zst.sha256
/var/backups/mgaif-server/mgaif-server-backup-YYYYMMDDTHHMMSSZ.log

Optional environment variables:

• BACKUP_ROOT=/path/to/backup-root changes the staging/output directory.
• STAMP=custom-stamp forces a deterministic timestamp in names.
• KEEP_STAGING=1 keeps the staging directory for inspection instead of deleting it after archive creation.

Move off-server

Example using rsync to another machine:

rsync -avP /var/backups/mgaif-server/mgaif-server-backup-*.tar.zst* backup-user@backup-host:/secure/backups/mgaif/

Verify on the destination:

sha256sum -c mgaif-server-backup-YYYYMMDDTHHMMSSZ.tar.zst.sha256
tar -I zstd -tf mgaif-server-backup-YYYYMMDDTHHMMSSZ.tar.zst | less

Backup scope

The script includes:

• Selected /home contents, excluding /home/.swap, all of /home/clp, /home/mysql, /home/linuxbrew, /home/mgaif-*/logs, /home/mgaif-*/backups, nginx log directories, and marked embedded backup staging directories. CloudPanel database exports remain included under the staging directory.
• /root/.hermes, excluding high-volume/generated paths: /root/.hermes/logs, /root/.hermes/caches, /root/.hermes/sessions, /root/.hermes/hermes-agent, and /root/.hermes/webui/sessions.
• CloudPanel database dumps from clpctl db:export.
• Extra non-system MySQL database dumps from mysqldump.
• Rebuild manifests: OS release, disk usage, installed Debian packages, systemd unit files, running services, listening ports, root crontab, CloudPanel site/database inventory, MediaWiki siteinfo, and Git revisions/status for Hermes Agent and Hermes Web UI.
• Selected configuration directories: /etc/nginx, /etc/systemd/system, /var/spool/cron, and /etc/letsencrypt when present. CloudPanel filesystem content under /home/clp is intentionally excluded; database exports remain included.

The script intentionally does not back up /home/.swap; it is a 2.1 GB swap file and should be recreated rather than restored. It excludes all of /home/clp from the filesystem archive, while still using CloudPanel tooling and SQLite metadata at runtime to export all CloudPanel-managed databases into the staging area. It also excludes /home/mysql because databases are captured through logical dumps, and excludes /home/linuxbrew because it should be reinstalled rather than restored wholesale. Under /home/mediawiki, the script includes only files and symlinks newer than the install baseline represented by the timestamp of /home/mediawiki/languages.

Database strategy

CloudPanel exposes the relevant backup command through clpctl:

clpctl db:export --databaseName=DATABASE --file=/path/to/DATABASE.sql.gz

Current CloudPanel-managed databases:

• test1-mgaif for test1.mgaif.com.
• wiki-mgaif for wiki.mgaif.com.

Current MySQL server inventory showed only the standard system schemas plus those two CloudPanel databases:

• information_schema
• mysql
• performance_schema
• sys
• test1-mgaif
• wiki-mgaif

The backup script still checks for additional non-system schemas on every run. If a future database exists outside CloudPanel’s database table, it is dumped to databases/mysql-extra/ in the archive.

Current server inventory

Captured during the TODO resolution on 2026-04-30.

Host and operating system

• Hostname: hal-server-723625.
• OS: Ubuntu 24.04.4 LTS (Noble Numbat).
• Kernel: Linux 6.8.0-90-generic x86_64.
• Root filesystem: ext4, 96 GB total, about 25 GB used and 72 GB available at capture time.
• /home size at capture time: about 6.0 GB including the 2.1 GB /home/.swap file.

Control panel and web stack

• CloudPanel package: cloudpanel 2.5.3-1+clp-noble.
• CloudPanel CLI: /usr/bin/clpctl.
• Web servers/reverse proxies: Nginx 1.28.3, CloudPanel Nginx, Varnish 7.5.0.
• Database server: Percona Server for MySQL 8.4.8-8, listening on 3306 and 33060.
• PHP: CloudPanel-provided PHP versions 7.1 through 8.5; active CLI PHP observed as 8.4.20.
• Node.js: v22.22.2 with npm 10.9.7.
• Python: system Python packages for 3.12 are installed; the active Hermes environment observed Python 3.11.15.
• Cache and supporting services: Redis 7.0.15, Memcached 1.6.24, Varnish, Postfix, ProFTPd, Fail2ban, Chrony, unattended-upgrades.

CloudPanel sites

• wiki.mgaif.com: CloudPanel reverse-proxy site owned by mgaif-wiki, reverse proxy target http://127.0.0.1:3411, Basic Auth configured in CloudPanel.
• test1.mgaif.com: CloudPanel static site owned by mgaif-test1; also hosts the WhiteBlaze Field Reports Next.js app files under /home/mgaif-test1/htdocs/test1.mgaif.com.
• claw.mgaif.com: CloudPanel reverse-proxy site owned by mgaif-claw, reverse proxy target http://127.0.0.1:3001.

Applications to account for during rebuild

• MediaWiki: MGAIF Wiki, MediaWiki 1.45.3, served by mediawiki-php.service from /home/mgaif-wiki/htdocs/wiki.mgaif.com on 127.0.0.1:3411. Default skin observed as Timeless. Installed skins/extensions observed through the API: Timeless, MinervaNeue, MonoBook, Vector, CategoryTree.
• Hermes Agent: repository at /root/.hermes/hermes-agent, branch main, commit observed as 69d4800db. The backup script records Git branch/commit/status in the manifests when the checkout is present, but excludes the full /root/.hermes/hermes-agent checkout from the archive.
• Hermes Gateway: /etc/systemd/system/hermes-gateway.service exists and is enabled but was inactive via systemd at capture time. A Hermes gateway process was still observed running from the Hermes virtual environment.
• Hermes Web UI: /home/hermes-webui, Python stdlib + PyYAML app using the Hermes virtual environment, observed listening on 127.0.0.1:3001.
• WhiteBlaze Field Reports: Next.js app under /home/mgaif-test1/htdocs/test1.mgaif.com, with whiteblaze-fieldreports.service running server.js. Dependencies include Next, React, Prisma, Tailwind, MapLibre, Zod, Vitest, Playwright, TypeScript, ESLint, and Prettier.
• CloudPanel itself: application data under /home/clp, CloudPanel SQLite state at /home/clp/htdocs/app/data/db.sq3, and CloudPanel Nginx/PHP-FPM services.

Important ports observed

• 21: ProFTPd.
• 22: SSH.
• 25: Postfix SMTP.
• 80 and 443: Nginx.
• 8443: CloudPanel UI Nginx.
• 3001: Hermes Web UI / claw.mgaif.com reverse-proxy target.
• 3411: local MediaWiki PHP server.
• 3306 and 33060: MySQL / MySQL X plugin.
• 6379: Redis on localhost.
• 11211: Memcached on localhost.
• 6081: Varnish.

Restore outline

Provision a clean Ubuntu 24.04 host and install CloudPanel/Percona/PHP/Node/Python tooling compatible with the captured package manifest.

Restore CloudPanel sites and users first, then copy back relevant /home directories with ownership preserved.

Reinstall or migrate CloudPanel itself deliberately rather than restoring /home/clp wholesale; the backup excludes /home/clp filesystem content but includes CloudPanel database exports and inventory manifests.

Import databases: CloudPanel-managed dumps with clpctl db:import --databaseName=NAME --file=NAME.sql.gz; extra MySQL dumps with mysql.

Restore and enable required systemd units, especially mediawiki-php.service, whiteblaze-fieldreports.service, and the desired Hermes gateway/service arrangement.

Reinstall Node dependencies for the WhiteBlaze app if node_modules is not restored cleanly, then rebuild/restart the service.

Restore Hermes from /root/.hermes, verify ~/.hermes/config.yaml, ~/.hermes/.env, cron jobs, skills, and gateway platform credentials.

Verify public endpoints: https://wiki.mgaif.com/, https://claw.mgaif.com/, https://test1.mgaif.com/, CloudPanel, SSH, and any messaging gateway integrations.

Rotate credentials if the backup was ever stored outside fully trusted encrypted storage.

Backup script

Installed path: /home/mediawiki/tools/mgaif_server_backup.sh

#!/usr/bin/env bash
# Create an off-server-transferable backup bundle for the MGAIF/CloudPanel server.
# Run as root from the server. The generated archive contains application data,
# CloudPanel and non-CloudPanel database dumps, rebuild manifests, and selected
# server configuration. Move the resulting .tar.zst file off the server.
set -Eeuo pipefail

BACKUP_ROOT="${BACKUP_ROOT:-/var/backups/mgaif-server}"
STAMP="${STAMP:-$(date -u +%Y%m%dT%H%M%SZ)}"
STAGE="$BACKUP_ROOT/staging-$STAMP"
ARCHIVE="$BACKUP_ROOT/mgaif-server-backup-$STAMP.tar.zst"
LOG="$BACKUP_ROOT/mgaif-server-backup-$STAMP.log"
KEEP_STAGING="${KEEP_STAGING:-0}"
DRY_RUN=0

if ["${1:-}" == "--dry-run"]("${1:-}" == "--dry-run"); then
  DRY_RUN=1
fi

msg() { printf '[%s] %s
' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" | tee -a "$LOG"; }
fail() { msg "ERROR: $*"; exit 1; }
need() { command -v "$1" >/dev/null 2>&1 || fail "Missing required command: $1"; }

require_root() {
  ["${EUID:-$(id -u)}" -eq 0]("${EUID:-$(id -u)}" -eq 0) || fail "Run as root so /home, /root/.hermes, configs, and DB dumps are readable."
}

cleanup_embedded_backup_dirs() {
  msg "Cleaning up previous backup staging directories under /home that carry the .mgaif-backup-staging marker."
  # Conservative cleanup: only remove directories explicitly marked by this backup workflow.
  while IFS= read -r -d '' marker; do
    dir="$(dirname "$marker")"
    case "$dir" in
      /home/*/mgaif-server-backup-*|/home/*/.mgaif-server-backup-*)
        msg "Removing marked embedded backup directory: $dir"
        rm -rf --one-file-system -- "$dir"
        ;;
      *)
        msg "Skipping marker outside expected pattern: $marker"
        ;;
    esac
  done < <(find /home -xdev -type f -name .mgaif-backup-staging -print0 2>/dev/null || true)
}

mysql_master_env() {
  # Emits shell assignments for MYSQL_HOST, MYSQL_PORT, MYSQL_USER, MYSQL_PWD.
  # The password is kept in-process and is not written to the archive.
  local out host port user pass
  out="$(clpctl db:show:master-credentials)"
  host="$(awk -F'|' '/Host/ {gsub(/^[   ]+|[   ]+$/, "", $3); print $3}' <<<"$out")"
  user="$(awk -F'|' '/User Name/ {gsub(/^[   ]+|[   ]+$/, "", $3); print $3}' <<<"$out")"
  pass="$(awk -F'|' '/Password/ {gsub(/^[   ]+|[   ]+$/, "", $3); print $3}' <<<"$out")"
  port="$(awk -F'|' '/Port/ {gsub(/^[   ]+|[   ]+$/, "", $3); print $3}' <<<"$out")"
  [-n "$host" && -n "$user" && -n "$pass" && -n "$port"](/-n-"$host"-&&--n-"$user"-&&--n-"$pass"-&&--n-"$port"/) || fail "Could not parse CloudPanel MySQL master credentials."
  printf 'MYSQL_HOST=%q
MYSQL_PORT=%q
MYSQL_USER=%q
MYSQL_PWD=%q
' "$host" "$port" "$user" "$pass"
}

mysql_cmd() {
  MYSQL_PWD="$MYSQL_PWD" mysql -h"$MYSQL_HOST" -P"$MYSQL_PORT" -u"$MYSQL_USER" "$@"
}

mysqldump_cmd() {
  MYSQL_PWD="$MYSQL_PWD" mysqldump -h"$MYSQL_HOST" -P"$MYSQL_PORT" -u"$MYSQL_USER" \
    --single-transaction --quick --routines --events --triggers "$@"
}

collect_manifests() {
  msg "Collecting rebuild manifests."
  mkdir -p "$STAGE/manifests"
  {
    echo "Backup timestamp UTC: $STAMP"
    echo "Hostname: $(hostname)"
    echo "Kernel: $(uname -a)"
    echo
    echo "OS release:"
    cat /etc/os-release || true
    echo
    echo "Disk usage:"
    df -hT || true
  } > "$STAGE/manifests/system.txt"

  dpkg-query -W -f='${Package}  ${Version}
' > "$STAGE/manifests/dpkg-packages.tsv" 2>/dev/null || true
  systemctl list-unit-files --no-pager > "$STAGE/manifests/systemd-unit-files.txt" 2>/dev/null || true
  systemctl list-units --type=service --state=running --no-pager > "$STAGE/manifests/running-services.txt" 2>/dev/null || true
  ss -ltnp > "$STAGE/manifests/listening-ports.txt" 2>/dev/null || true
  crontab -l > "$STAGE/manifests/root-crontab.txt" 2>/dev/null || true

  if [-d /home/clp/htdocs/app/data](/-d-/home/clp/htdocs/app/data/); then
    python3 - <<'PY' > "$STAGE/manifests/cloudpanel-inventory.txt" 2>/dev/null || true
import sqlite3, json
path='/home/clp/htdocs/app/data/db.sq3'
conn=sqlite3.connect(path); conn.row_factory=sqlite3.Row
cur=conn.cursor()
print('CloudPanel sites:')
for r in cur.execute("select type,domain_name,root_directory,user,application,reverse_proxy_url,varnish_cache,page_speed_enabled,basic_auth_id from site order by domain_name"):
    print(json.dumps(dict(r), sort_keys=True))
print('
CloudPanel databases:')
for r in cur.execute("select d.name as database_name, s.domain_name, ds.engine, ds.version, ds.host, ds.port from database d left join site s on d.site_id=s.id left join database_server ds on d.database_server_id=ds.id order by d.name"):
    print(json.dumps(dict(r), sort_keys=True))
PY
  fi

  if [-d /root/.hermes/hermes-agent/.git](/-d-/root/.hermes/hermes-agent/.git/); then
    {
      echo "Hermes agent git branch: $(git -C /root/.hermes/hermes-agent rev-parse --abbrev-ref HEAD 2>/dev/null || true)"
      echo "Hermes agent git commit: $(git -C /root/.hermes/hermes-agent rev-parse HEAD 2>/dev/null || true)"
      echo "Hermes agent status:"
      git -C /root/.hermes/hermes-agent status --short 2>/dev/null || true
    } > "$STAGE/manifests/hermes-agent-git.txt"
  fi

  if [-d /home/hermes-webui/.git](/-d-/home/hermes-webui/.git/); then
    {
      echo "Hermes Web UI git branch: $(git -C /home/hermes-webui rev-parse --abbrev-ref HEAD 2>/dev/null || true)"
      echo "Hermes Web UI git commit: $(git -C /home/hermes-webui rev-parse HEAD 2>/dev/null || true)"
      echo "Hermes Web UI status:"
      git -C /home/hermes-webui status --short 2>/dev/null || true
    } > "$STAGE/manifests/hermes-webui-git.txt"
  fi

  curl -fsS 'http://127.0.0.1:3411/api.php?action=query&meta=siteinfo&siprop=general|extensions|skins&format=json' \
    > "$STAGE/manifests/mediawiki-siteinfo.json" 2>/dev/null || true
}

cloudpanel_database_names() {
  if [-r /home/clp/htdocs/app/data/db.sq3](/-r-/home/clp/htdocs/app/data/db.sq3/); then
    sqlite3 /home/clp/htdocs/app/data/db.sq3 "select name from database order by name;" 2>/dev/null || true
  fi
}

dump_databases() {
  msg "Dumping CloudPanel databases with clpctl, then any other non-system MySQL databases with mysqldump."
  mkdir -p "$STAGE/databases/cloudpanel" "$STAGE/databases/mysql-extra"
  local db file
  mapfile -t CLP_DBS < <(cloudpanel_database_names)

  for db in "${CLP_DBS[@]:-}"; do
    [-n "$db"](/-n-"$db"/) || continue
    file="$STAGE/databases/cloudpanel/${db}.sql.gz"
    msg "CloudPanel export: $db -> $file"
    clpctl db:export --databaseName="$db" --file="$file"
  done

  eval "$(mysql_master_env)"
  mapfile -t ALL_DBS < <(mysql_cmd -NBe 'SHOW DATABASES;')
  for db in "${ALL_DBS[@]:-}"; do
    [-n "$db"](/-n-"$db"/) || continue
    case "$db" in information_schema|mysql|performance_schema|sys) continue ;; esac
    if printf '%s
' "${CLP_DBS[@]:-}" | grep -Fxq -- "$db"; then
      continue
    fi
    file="$STAGE/databases/mysql-extra/${db}.sql.gz"
    msg "Extra MySQL export: $db -> $file"
    mysqldump_cmd --databases "$db" | gzip -9 > "$file"
  done
}

create_archive() {
  msg "Creating archive: $ARCHIVE"
  local includes=()
  local path
  local mediawiki_changed_list="$STAGE/manifests/mediawiki-changed-files.txt"

  includes+=("$STAGE")

  # Include /home contents selectively. Large/generated directories are excluded here
  # rather than relying only on tar excludes, so dry-run output and archive scope stay clear.
  if [-d /home](/-d-/home/); then
    for path in /home/* /home/.[!.]* /home/..?*; do
      [-e "$path"](/-e-"$path"/) || continue
      case "$path" in
        /home/.|/home/..|/home/.swap|/home/mediawiki|/home/mysql|/home/linuxbrew|/home/clp|/home/hermes-webui)
          continue
          ;;
      esac
      includes+=("$path")
    done
  fi

  # /home/mediawiki is mostly upstream MediaWiki release files. Include only files
  # and symlinks changed after the install baseline, represented by the mtime of
  # /home/mediawiki/languages.
  : > "$mediawiki_changed_list"
  if [-d /home/mediawiki && -e /home/mediawiki/languages](/-d-/home/mediawiki-&&--e-/home/mediawiki/languages/); then
    find /home/mediawiki -xdev \( -type f -o -type l \) -newer /home/mediawiki/languages -print \
      > "$mediawiki_changed_list" 2>/dev/null || true
    msg "MediaWiki changed-file list: $mediawiki_changed_list ($(wc -l < "$mediawiki_changed_list") entries)."
  elif [-d /home/mediawiki](/-d-/home/mediawiki/); then
    msg "WARNING: /home/mediawiki exists but /home/mediawiki/languages was not found; /home/mediawiki is not included except for database dumps and linked site paths."
  fi

  [-d /root/.hermes](/-d-/root/.hermes/) && includes+=("/root/.hermes")
  [-d /etc/nginx](/-d-/etc/nginx/) && includes+=("/etc/nginx")
  [-d /etc/letsencrypt](/-d-/etc/letsencrypt/) && includes+=("/etc/letsencrypt")
  includes+=("/etc/systemd/system")
  [-d /var/spool/cron](/-d-/var/spool/cron/) && includes+=("/var/spool/cron")

  tar --create --preserve-permissions --xattrs --acls --one-file-system \
    --use-compress-program='zstd -19 -T0' \
    --file "$ARCHIVE" \
    --files-from="$mediawiki_changed_list" \
    --exclude=/home/.swap \
    --exclude=/home/mysql \
    --exclude=/home/mysql/* \
    --exclude=/home/linuxbrew \
    --exclude=/home/linuxbrew/* \
    --exclude=/home/clp \
    --exclude=/home/clp/* \
    --exclude=/root/.hermes/logs \
    --exclude=/root/.hermes/logs/* \
    --exclude=/root/.hermes/caches \
    --exclude=/root/.hermes/caches/* \
    --exclude=/root/.hermes/sessions \
    --exclude=/root/.hermes/sessions/* \
    --exclude=/root/.hermes/hermes-agent \
    --exclude=/root/.hermes/hermes-agent/* \
    --exclude=/root/.hermes/webui/sessions \
    --exclude=/root/.hermes/webui/sessions/* \
    --exclude=/home/mgaif-*/logs \
    --exclude=/home/mgaif-*/logs/* \
    --exclude=/home/mgaif-*/backups \
    --exclude=/home/mgaif-*/backups/* \
    --exclude='*/nginx/logs' \
    --exclude='*/nginx/logs/*' \
    --exclude='*/nginx/*/logs' \
    --exclude='*/nginx/*/logs/*' \
    --exclude=/home/*/mgaif-server-backup-* \
    --exclude=/home/*/.mgaif-server-backup-* \
    --exclude="$ARCHIVE" \
    "${includes[@]}"

  sha256sum "$ARCHIVE" > "$ARCHIVE.sha256"
  msg "Archive complete: $ARCHIVE"
  msg "Checksum: $ARCHIVE.sha256"
}

main() {
  require_root
  for c in clpctl sqlite3 mysql mysqldump gzip tar zstd sha256sum find curl git dpkg-query systemctl ss; do need "$c"; done
  mkdir -p "$BACKUP_ROOT" "$STAGE"
  : > "$STAGE/.mgaif-backup-staging"
  : > "$LOG"

  if ["$DRY_RUN" -eq 1](/"$dry_run"--eq-1/); then
    msg "Dry run only. Would stage in $STAGE and create $ARCHIVE."
    msg "Would export CloudPanel DBs: $(cloudpanel_database_names | paste -sd ', ' -)"
    msg "Would include selected /home contents excluding /home/.swap, /home/clp, /home/mysql, /home/linuxbrew, /home/mgaif-*/{logs,backups}, nginx logs directories, plus /home/mediawiki files newer than /home/mediawiki/languages. Would include /root/.hermes excluding logs, caches, sessions, hermes-agent, and webui/sessions. CloudPanel and extra MySQL databases would still be exported into the staging area."
    exit 0
  fi

  cleanup_embedded_backup_dirs
  collect_manifests
  dump_databases
  create_archive

  if ["$KEEP_STAGING" != "1"](/"$keep_staging"-!=-"1"/); then
    msg "Removing staging directory: $STAGE"
    rm -rf --one-file-system -- "$STAGE"
  else
    msg "Keeping staging directory because KEEP_STAGING=1: $STAGE"
  fi

  msg "Move $ARCHIVE and $ARCHIVE.sha256 off-server, then verify with: sha256sum -c $(basename "$ARCHIVE.sha256")"
}

main "$@"