admin/scripts/books-vds-backup-daily-kreknin/README.md

books-vds-backup-daily-kreknin

Daily backup pipeline: books VDS (89.253.255.133 / host4g.ru CentOS 7) → kreknin Synology (195.19.90.188:/volume1/NetBackup/books-vds/<date>/) via SSH + rsync.

Schedule: 0 6 * * * Europe/Moscow (after VDS-infra at 05:00, after windows-host at 05:30). Pattern mirrors scripts/vds-backup-rsync-kreknin/run.sh.

What's backed up

Component	Method	Source
MariaDB (books-db)	mariadb-dump --all-databases --single-transaction	container books-db
Mongo (shared)	mongodump --archive w/ root auth	container mongo
Mongo (job-scheduler)	mongodump --archive no-auth	container books-job-scheduler-mongo
Elasticsearch	REST snapshot API → fs repo kreknin → rsync	container elasticsearch (path.repo=/snapshots)
MinIO blobs	rsync direct (immutable content-addressed)	/usr/docker/minio/data
App configs	rsync	/opt/books/{api,job-scheduler,task-runner,ntfy,books-ops-mcp}
Traefik state	rsync	/usr/docker/traefik/{letsencrypt,data} (acme.json + config)
System config	rsync	/etc/{ssh,hosts,cron.d}, /root/.ssh

Retention: 7 daily snapshots on kreknin (+ on ES repo). Older — pruned at end of each run.

Notifications

Dual-channel after every run:

• ntfy → topic vds-backup on ntfy.vds.kzntsv.site (shared с VDS + RUVDS + windows-host backups).
  • Success: title BOOKS-VDS backup OK <date>, tag green_circle.
  • Fail: title BOOKS-VDS backup FAILED <date>, tag red_circle, priority high.
• Email via Yandex SMTP (noreply@snolla.com → OPS_NOTIFY_EMAIL).
  • Sent via curl --ssl-reqd --url smtp://...:587 (no msmtp dep on CentOS 7).
  • Subject: [BOOKS-VDS] backup OK <date> / [BOOKS-VDS] backup FAILED <date>.

If notify-channels fail — backup itself does NOT fail (|| true).

Files

• run.sh — backup logic. Synced to /opt/stacks/backup/run.sh on books VDS.
• .env.example — placeholder. Live .env built from pass show entries during install.

Initial install (rebuild from scratch)

# From workstation, with id_ed25519_books_ops authorized:
ssh root@89.253.255.133 'mkdir -p /opt/stacks/backup'
scp scripts/books-vds-backup-daily-kreknin/run.sh root@89.253.255.133:/opt/stacks/backup/

# Build .env locally from pass entries:
{
  echo "DEST_USER=vitya"
  echo "DEST_HOST=195.19.90.188"
  echo "DEST_BASE=/volume1/NetBackup/books-vds"
  echo "RETENTION_DAYS=7"
  echo "SSH_KEY=/opt/stacks/backup/kreknin-key"
  echo "ES_SNAPSHOT_REPO=kreknin"
  pass show books-vds/full-env | grep -E '^(BOOKS_DB_ROOT_PASSWORD|BOOKS_MONGO_)'
  pass show vds-kzntsv/full-env | grep -E '^NTFY_'
  pass show snolla-smtp/full-env | grep -E '^(SMTP_|OPS_NOTIFY_EMAIL)'
} | scp /dev/stdin root@89.253.255.133:/opt/stacks/backup/.env
ssh root@89.253.255.133 'chmod 600 /opt/stacks/backup/.env && chmod 755 /opt/stacks/backup/run.sh'

# Generate kreknin SSH key on books VDS:
ssh root@89.253.255.133 'ssh-keygen -t ed25519 -f /opt/stacks/backup/kreknin-key -N "" -C "books-vds-backup"'

# Get pubkey and authorize it on kreknin (workstation pivots — books VDS has no kreknin SSH yet):
PUBKEY=$(ssh root@89.253.255.133 cat /opt/stacks/backup/kreknin-key.pub)
ssh -i ~/.ssh/id_ed25519_kreknin vitya@195.19.90.188 \
  "echo '$PUBKEY' >> ~/.ssh/authorized_keys && mkdir -p /volume1/NetBackup/books-vds"

# Verify SSH books → kreknin
ssh root@89.253.255.133 'ssh -i /opt/stacks/backup/kreknin-key -o StrictHostKeyChecking=accept-new vitya@195.19.90.188 hostname'

# Smoke run (manual)
ssh root@89.253.255.133 '/opt/stacks/backup/run.sh'

# Install cron
ssh root@89.253.255.133 'echo "0 6 * * * root /opt/stacks/backup/run.sh" > /etc/cron.d/books-vds-backup && chmod 644 /etc/cron.d/books-vds-backup'

Decisions log

• ES snapshot via REST API, not data-dir rsync. Filesystem snapshot repo (type=fs, location=/snapshots, compress=true) registered as kreknin on 2026-05-25. Required one-time stack edit (add path.repo=/snapshots env + ./snapshots:/snapshots bind in /usr/docker/elasticsearch/docker-compose.yml).
• curl --ssl-reqd SMTP instead of msmtp — avoid CentOS 7 msmtp install (EOL repo issues). Native curl works on STARTTLS port 587.
• No tasks.yml-style auth on books-job-scheduler-mongo — container has no MONGO_INITDB_ROOT_* env. mongodump --archive без --uri достаточен.
• rsync /opt/books + /usr/docker/{minio,elasticsearch/snapshots,traefik} — bind paths из inspect 2026-05-25; не /var/lib/docker/volumes/ (named volumes для контейнеров пустые, кроме portainer/scheduler-mongo).
• books-vds-portainer/full-env consolidated into books-vds/full-env for one-file-per-host pattern parity with vds-kzntsv/full-env.
• No BackupConfig snapshot/git pre-commit-like step — VDS-infra run.sh тоже не делает, и нет аналогичной системы на books VDS.

Smoke run (manual)

ssh root@89.253.255.133 '/opt/stacks/backup/run.sh'
# Watch log:
ssh root@89.253.255.133 'tail -F /var/log/books-vds-backup/$(date +%Y-%m-%d).log'

Verify on kreknin:

ssh vitya@195.19.90.188 'du -sh /volume1/NetBackup/books-vds/*/'

Atomic revert (uninstall)

# On books VDS:
ssh root@89.253.255.133 'rm /etc/cron.d/books-vds-backup; rm -rf /opt/stacks/backup /var/log/books-vds-backup'

# On kreknin (via SSH):
ssh vitya@195.19.90.188 'rm -rf /volume1/NetBackup/books-vds'
# Remove the books-vds pubkey line from authorized_keys (one line with 'books-vds-backup' comment).

# On books VDS — revert ES path.repo change:
ssh root@89.253.255.133 'cd /usr/docker/elasticsearch && \
  cp docker-compose.yml.bak-pre-snapshots-2026-05-25 docker-compose.yml && \
  docker rm -f elasticsearch && docker-compose up -d'