backup(books-vds): daily 06:00 MSK pipeline to kreknin live 🟢

- scripts/books-vds-backup-daily-kreknin/ — run.sh + .env.example + README - ES path.repo bootstrap (one-time stack edit, snapshot repo 'kreknin' registered) - curl smtps://yandex:465 email (CRLF + Date headers, no msmtp dep on CentOS 7) - StrictHostKeyChecking=yes + pre-populated known_hosts (CentOS 7 no accept-new) - cron /etc/cron.d/books-vds-backup live, initial sync 4.87 GB in 13m28s, ntfy push sent - wiki: entities/books-vds.md + sources/books-vds-backup-daily-kreknin-2026-05-25.md created - wiki: concepts/books-ssh-access → vds-kzntsv-ssh-access (was lying about books on vds-kzntsv) - .tasks/books-vds-backup-daily-kreknin.md closed 🟢 pending phone-side verify Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 09:37:16 +03:00
parent 1eefb4d45a
commit fe41ee4422
10 changed files with 694 additions and 35 deletions
--- a/scripts/books-vds-backup-daily-kreknin/.env.example
+++ b/scripts/books-vds-backup-daily-kreknin/.env.example
@@ -0,0 +1,34 @@
+# /opt/stacks/backup/.env on books VDS — sourced by run.sh
+# Live values are NOT in git; sourced from pass entries:
+#   pass show books-vds/full-env     → BOOKS_DB_* + BOOKS_MONGO_*
+#   pass show vds-kzntsv/full-env    → NTFY_*
+#   pass show snolla-smtp/full-env   → SMTP_* + OPS_NOTIFY_EMAIL
+#   pass show kreknin/full-env       → DEST_USER (vitya), DEST_HOST (195.19.90.188)
+
+# === Backup destination ===
+DEST_USER=vitya
+DEST_HOST=195.19.90.188
+DEST_BASE=/volume1/NetBackup/books-vds
+RETENTION_DAYS=7
+SSH_KEY=/opt/stacks/backup/kreknin-key
+
+# === DB creds (books VDS containers) ===
+BOOKS_DB_ROOT_PASSWORD=...
+BOOKS_MONGO_USERNAME=root
+BOOKS_MONGO_PASSWORD=...
+
+# === Elasticsearch snapshot repo (file-based, registered 2026-05-25) ===
+ES_SNAPSHOT_REPO=kreknin
+
+# === NTFY (shared channel — vds-backup topic; ntfy.vds.kzntsv.site) ===
+NTFY_URL=https://ntfy.vds.kzntsv.site
+NTFY_USER=...
+NTFY_PASS=...
+
+# === SMTP (Yandex relay, noreply@snolla.com) ===
+SMTP_HOST=smtp.yandex.ru
+SMTP_PORT=587
+SMTP_USER=noreply@snolla.com
+SMTP_PASS=...
+SMTP_FROM=noreply@snolla.com
+OPS_NOTIFY_EMAIL=...
--- a/scripts/books-vds-backup-daily-kreknin/README.md
+++ b/scripts/books-vds-backup-daily-kreknin/README.md
@@ -0,0 +1,115 @@
+# books-vds-backup-daily-kreknin
+
+Daily backup pipeline: **books VDS** (`89.253.255.133` / host4g.ru CentOS 7) → **kreknin Synology** (`195.19.90.188:/volume1/NetBackup/books-vds/<date>/`) via SSH + rsync.
+
+Schedule: `0 6 * * *` Europe/Moscow (after VDS-infra at 05:00, after windows-host at 05:30). Pattern mirrors `scripts/vds-backup-rsync-kreknin/run.sh`.
+
+## What's backed up
+
+| Component | Method | Source |
+|---|---|---|
+| MariaDB (books-db) | `mariadb-dump --all-databases --single-transaction` | container `books-db` |
+| Mongo (shared) | `mongodump --archive` w/ root auth | container `mongo` |
+| Mongo (job-scheduler) | `mongodump --archive` no-auth | container `books-job-scheduler-mongo` |
+| Elasticsearch | REST snapshot API → fs repo `kreknin` → rsync | container `elasticsearch` (path.repo=/snapshots) |
+| MinIO blobs | rsync direct (immutable content-addressed) | `/usr/docker/minio/data` |
+| App configs | rsync | `/opt/books/{api,job-scheduler,task-runner,ntfy,books-ops-mcp}` |
+| Traefik state | rsync | `/usr/docker/traefik/{letsencrypt,data}` (acme.json + config) |
+| System config | rsync | `/etc/{ssh,hosts,cron.d}`, `/root/.ssh` |
+
+Retention: **7 daily snapshots** on kreknin (+ on ES repo). Older — pruned at end of each run.
+
+## Notifications
+
+Dual-channel after every run:
+
+- **ntfy** → topic `vds-backup` on `ntfy.vds.kzntsv.site` (shared с VDS + RUVDS + windows-host backups).
+  - Success: title `BOOKS-VDS backup OK <date>`, tag `green_circle`.
+  - Fail: title `BOOKS-VDS backup FAILED <date>`, tag `red_circle`, priority `high`.
+- **Email** via Yandex SMTP (`noreply@snolla.com` → `OPS_NOTIFY_EMAIL`).
+  - Sent via `curl --ssl-reqd --url smtp://...:587` (no msmtp dep on CentOS 7).
+  - Subject: `[BOOKS-VDS] backup OK <date>` / `[BOOKS-VDS] backup FAILED <date>`.
+
+If notify-channels fail — backup itself does NOT fail (`|| true`).
+
+## Files
+
+- `run.sh` — backup logic. Synced to `/opt/stacks/backup/run.sh` on books VDS.
+- `.env.example` — placeholder. Live `.env` built from `pass show` entries during install.
+
+## Initial install (rebuild from scratch)
+
+```bash
+# From workstation, with id_ed25519_books_ops authorized:
+ssh root@89.253.255.133 'mkdir -p /opt/stacks/backup'
+scp scripts/books-vds-backup-daily-kreknin/run.sh root@89.253.255.133:/opt/stacks/backup/
+
+# Build .env locally from pass entries:
+{
+  echo "DEST_USER=vitya"
+  echo "DEST_HOST=195.19.90.188"
+  echo "DEST_BASE=/volume1/NetBackup/books-vds"
+  echo "RETENTION_DAYS=7"
+  echo "SSH_KEY=/opt/stacks/backup/kreknin-key"
+  echo "ES_SNAPSHOT_REPO=kreknin"
+  pass show books-vds/full-env | grep -E '^(BOOKS_DB_ROOT_PASSWORD|BOOKS_MONGO_)'
+  pass show vds-kzntsv/full-env | grep -E '^NTFY_'
+  pass show snolla-smtp/full-env | grep -E '^(SMTP_|OPS_NOTIFY_EMAIL)'
+} | scp /dev/stdin root@89.253.255.133:/opt/stacks/backup/.env
+ssh root@89.253.255.133 'chmod 600 /opt/stacks/backup/.env && chmod 755 /opt/stacks/backup/run.sh'
+
+# Generate kreknin SSH key on books VDS:
+ssh root@89.253.255.133 'ssh-keygen -t ed25519 -f /opt/stacks/backup/kreknin-key -N "" -C "books-vds-backup"'
+
+# Get pubkey and authorize it on kreknin (workstation pivots — books VDS has no kreknin SSH yet):
+PUBKEY=$(ssh root@89.253.255.133 cat /opt/stacks/backup/kreknin-key.pub)
+ssh -i ~/.ssh/id_ed25519_kreknin vitya@195.19.90.188 \
+  "echo '$PUBKEY' >> ~/.ssh/authorized_keys && mkdir -p /volume1/NetBackup/books-vds"
+
+# Verify SSH books → kreknin
+ssh root@89.253.255.133 'ssh -i /opt/stacks/backup/kreknin-key -o StrictHostKeyChecking=accept-new vitya@195.19.90.188 hostname'
+
+# Smoke run (manual)
+ssh root@89.253.255.133 '/opt/stacks/backup/run.sh'
+
+# Install cron
+ssh root@89.253.255.133 'echo "0 6 * * * root /opt/stacks/backup/run.sh" > /etc/cron.d/books-vds-backup && chmod 644 /etc/cron.d/books-vds-backup'
+```
+
+## Decisions log
+
+- **ES snapshot via REST API**, not data-dir rsync. Filesystem snapshot repo (`type=fs`, `location=/snapshots`, `compress=true`) registered as `kreknin` on 2026-05-25. Required one-time stack edit (add `path.repo=/snapshots` env + `./snapshots:/snapshots` bind in `/usr/docker/elasticsearch/docker-compose.yml`).
+- **`curl --ssl-reqd` SMTP** instead of msmtp — avoid CentOS 7 msmtp install (EOL repo issues). Native `curl` works on STARTTLS port 587.
+- **No `tasks.yml`-style auth on `books-job-scheduler-mongo`** — container has no `MONGO_INITDB_ROOT_*` env. `mongodump --archive` без `--uri` достаточен.
+- **rsync /opt/books + /usr/docker/{minio,elasticsearch/snapshots,traefik}** — bind paths из inspect 2026-05-25; не `/var/lib/docker/volumes/` (named volumes для контейнеров пустые, кроме portainer/scheduler-mongo).
+- **`books-vds-portainer/full-env` consolidated** into `books-vds/full-env` for one-file-per-host pattern parity with `vds-kzntsv/full-env`.
+- **No `BackupConfig` snapshot/git pre-commit-like step** — VDS-infra `run.sh` тоже не делает, и нет аналогичной системы на books VDS.
+
+## Smoke run (manual)
+
+```bash
+ssh root@89.253.255.133 '/opt/stacks/backup/run.sh'
+# Watch log:
+ssh root@89.253.255.133 'tail -F /var/log/books-vds-backup/$(date +%Y-%m-%d).log'
+```
+
+Verify on kreknin:
+```bash
+ssh vitya@195.19.90.188 'du -sh /volume1/NetBackup/books-vds/*/'
+```
+
+## Atomic revert (uninstall)
+
+```bash
+# On books VDS:
+ssh root@89.253.255.133 'rm /etc/cron.d/books-vds-backup; rm -rf /opt/stacks/backup /var/log/books-vds-backup'
+
+# On kreknin (via SSH):
+ssh vitya@195.19.90.188 'rm -rf /volume1/NetBackup/books-vds'
+# Remove the books-vds pubkey line from authorized_keys (one line with 'books-vds-backup' comment).
+
+# On books VDS — revert ES path.repo change:
+ssh root@89.253.255.133 'cd /usr/docker/elasticsearch && \
+  cp docker-compose.yml.bak-pre-snapshots-2026-05-25 docker-compose.yml && \
+  docker rm -f elasticsearch && docker-compose up -d'
+```
--- a/scripts/books-vds-backup-daily-kreknin/run.sh
+++ b/scripts/books-vds-backup-daily-kreknin/run.sh
@@ -0,0 +1,216 @@
+#!/bin/bash
+# Daily books VDS → kreknin backup. Triggered by /etc/cron.d/books-vds-backup at 06:00 MSK.
+# Pattern: mirror VDS scripts/vds-backup-rsync-kreknin/run.sh, adapted for books VDS hybrid stack.
+set -euo pipefail
+
+ENV_FILE=/opt/stacks/backup/.env
+
+# shellcheck disable=SC1090
+source "$ENV_FILE"
+
+TODAY=$(date +%Y-%m-%d)
+LOG_DIR=/var/log/books-vds-backup
+LOG="$LOG_DIR/$TODAY.log"
+LOCK=/tmp/books-vds-backup.lock
+DUMP_DIR=/tmp/books-vds-backup-dumps/$TODAY
+DEST_PATH="$DEST_BASE/$TODAY"
+LATEST="$DEST_BASE/latest"
+
+mkdir -p "$LOG_DIR" "$DUMP_DIR"
+exec >> "$LOG" 2>&1
+echo "===== BOOKS-VDS backup start $TODAY $(date -Is) ====="
+
+# single-instance lock
+exec 9>"$LOCK"
+if ! flock -n 9; then echo "already running, exiting"; exit 0; fi
+
+START=$(date +%s)
+SSH_OPTS="-i $SSH_KEY -o StrictHostKeyChecking=yes -o ConnectTimeout=30 -o LogLevel=ERROR"
+# Note: kreknin host key must be pre-populated in /root/.ssh/known_hosts (CentOS 7 OpenSSH 7.4 has no accept-new option)
+
+HOST_LABEL=BOOKS-VDS
+SOURCE_DISPLAY="89.253.255.133 (books VDS)"
+
+notify() {
+  local title="$1" msg="$2" tags="${3:-}" prio="${4:-default}"
+  curl -sS --max-time 10 -u "$NTFY_USER:$NTFY_PASS" \
+    -H "Title: $title" -H "Tags: $tags" -H "Priority: $prio" \
+    -d "$msg" "$NTFY_URL/vds-backup" > /dev/null || true
+}
+
+email_send() {
+  local subj="$1" body="$2"
+  local tmpf
+  tmpf=$(mktemp)
+  # RFC 5322: CRLF headers + Date required by Yandex
+  {
+    printf 'From: %s\r\n' "$SMTP_FROM"
+    printf 'To: %s\r\n' "$OPS_NOTIFY_EMAIL"
+    printf 'Subject: %s\r\n' "$subj"
+    printf 'Date: %s\r\n' "$(date -R)"
+    printf 'MIME-Version: 1.0\r\n'
+    printf 'Content-Type: text/plain; charset=UTF-8\r\n'
+    printf '\r\n'
+    echo "$body" | sed 's/$/\r/'
+  } > "$tmpf"
+  # smtps:// (implicit TLS) for Yandex port 465; smtp:// (STARTTLS) would be port 587.
+  # SMTP_PORT in pass is 465 — use smtps:// scheme.
+  curl -sS --max-time 30 \
+    --ssl-reqd \
+    --url "smtps://$SMTP_HOST:$SMTP_PORT" \
+    --user "$SMTP_USER:$SMTP_PASS" \
+    --mail-from "$SMTP_FROM" \
+    --mail-rcpt "$OPS_NOTIFY_EMAIL" \
+    --upload-file "$tmpf" || true
+  rm -f "$tmpf"
+}
+
+fmt_duration() {
+  local s=$1
+  printf "%dm%02ds" $((s/60)) $((s%60))
+}
+
+on_error() {
+  local line=$1 rc=$2
+  local err="line $line, exit $rc"
+  local now
+  now=$(date +%s)
+  local dur=$((now - ${START:-now}))
+  local dur_h
+  dur_h=$(fmt_duration "$dur")
+  local tail_log
+  tail_log=$(tail -40 "$LOG" 2>/dev/null || true)
+
+  echo "FATAL: $HOST_LABEL backup FAILED — $err"
+
+  notify "$HOST_LABEL backup FAILED $TODAY" \
+         "After $dur_h: $err. See $LOG" \
+         red_circle high
+
+  email_send "[$HOST_LABEL] backup FAILED $TODAY" \
+"$HOST_LABEL daily backup FAILED.
+
+Date:     $TODAY
+Duration: $dur_h
+Error:    $err
+Source:   $SOURCE_DISPLAY
+Log:      $LOG
+
+Tail (last 40 lines):
+$tail_log"
+  exit "$rc"
+}
+trap 'on_error $LINENO $?' ERR
+
+# === Step 1: DB dumps ===
+echo "--- DB dumps to $DUMP_DIR ---"
+
+docker exec books-db mariadb-dump -uroot -p"$BOOKS_DB_ROOT_PASSWORD" \
+  --all-databases --single-transaction --quick 2>/dev/null \
+  | gzip > "$DUMP_DIR/mariadb.sql.gz"
+echo "mariadb: $(stat -c %s "$DUMP_DIR/mariadb.sql.gz") bytes"
+
+docker exec mongo mongodump --archive --quiet \
+  --uri="mongodb://$BOOKS_MONGO_USERNAME:$BOOKS_MONGO_PASSWORD@localhost:27017/?authSource=admin" 2>/dev/null \
+  | gzip > "$DUMP_DIR/mongo.archive.gz"
+echo "mongo (shared): $(stat -c %s "$DUMP_DIR/mongo.archive.gz") bytes"
+
+docker exec books-job-scheduler-mongo mongodump --archive --quiet 2>/dev/null \
+  | gzip > "$DUMP_DIR/job-scheduler-mongo.archive.gz"
+echo "job-scheduler-mongo: $(stat -c %s "$DUMP_DIR/job-scheduler-mongo.archive.gz") bytes"
+
+# === Step 2: ES snapshot via REST ===
+echo "--- ES snapshot daily-$TODAY in repo $ES_SNAPSHOT_REPO ---"
+ES_SNAP="daily-$TODAY"
+SNAP_RESULT=$(docker exec elasticsearch curl -sS -X PUT \
+  "http://localhost:9200/_snapshot/$ES_SNAPSHOT_REPO/$ES_SNAP?wait_for_completion=true" \
+  -H 'Content-Type: application/json')
+echo "ES snapshot result: $SNAP_RESULT"
+if ! echo "$SNAP_RESULT" | grep -q '"state":"SUCCESS"'; then
+  echo "ES snapshot did NOT report SUCCESS state — failing"
+  exit 3
+fi
+
+echo "--- prune ES snapshots older than $RETENTION_DAYS ---"
+# Pure bash parse — no jq/python dependency
+ALL_SNAPS=$(docker exec elasticsearch curl -sS "http://localhost:9200/_snapshot/$ES_SNAPSHOT_REPO/_all" \
+  | grep -oE '"snapshot":"daily-[0-9-]+"' \
+  | sed -E 's/.*"(daily-[0-9-]+)".*/\1/' \
+  | sort)
+SNAP_COUNT=$(echo "$ALL_SNAPS" | wc -l)
+PRUNE_COUNT=$((SNAP_COUNT - RETENTION_DAYS))
+if [ $PRUNE_COUNT -gt 0 ]; then
+  echo "$ALL_SNAPS" | head -n "$PRUNE_COUNT" | while read -r snap; do
+    [ -z "$snap" ] && continue
+    echo "deleting ES snapshot: $snap"
+    docker exec elasticsearch curl -sS -X DELETE \
+      "http://localhost:9200/_snapshot/$ES_SNAPSHOT_REPO/$snap" || true
+    echo
+  done
+fi
+
+# === Step 3: rsync to kreknin ===
+echo "--- rsync to $DEST_USER@$DEST_HOST:$DEST_PATH ---"
+ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "mkdir -p '$DEST_PATH'"
+
+LINK_DEST_ARG=""
+if ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "test -L '$LATEST' || test -d '$LATEST'" 2>/dev/null; then
+  LINK_DEST_ARG="--link-dest=$LATEST"
+fi
+
+rsync -aHh --info=stats2 --delete $LINK_DEST_ARG \
+  -e "ssh $SSH_OPTS" \
+  /opt/books \
+  /usr/docker/elasticsearch/snapshots \
+  /usr/docker/minio/data \
+  /usr/docker/traefik/letsencrypt \
+  /usr/docker/traefik/data \
+  /etc/ssh \
+  /etc/hosts \
+  /etc/cron.d \
+  /root/.ssh \
+  "$DUMP_DIR" \
+  "$DEST_USER@$DEST_HOST:$DEST_PATH/" || true
+
+ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "rm -f '$LATEST' && ln -s '$DEST_PATH' '$LATEST'"
+
+# === Step 4: retention prune on kreknin ===
+echo "--- prune snapshots older than $RETENTION_DAYS days on kreknin ---"
+ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" \
+  "cd '$DEST_BASE' && ls -1d 20*-*-* 2>/dev/null | sort | head -n -$RETENTION_DAYS | xargs -r rm -rf"
+
+# === Step 5: local cleanup ===
+rm -rf "$DUMP_DIR"
+
+# === Step 6: report ===
+END=$(date +%s)
+DURATION=$((END - START))
+SIZE=$(ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "du -sh '$DEST_PATH' 2>/dev/null | cut -f1" || echo "?")
+SNAPSHOT_COUNT=$(ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "ls -1d $DEST_BASE/20*-*-* 2>/dev/null | wc -l" || echo "?")
+DURATION_HUMAN=$(fmt_duration "$DURATION")
+
+NTFY_BODY="$DURATION_HUMAN, size=$SIZE, snapshots=$SNAPSHOT_COUNT, dest=kreknin:$DEST_PATH"
+notify "$HOST_LABEL backup OK $TODAY" "$NTFY_BODY" green_circle
+
+EMAIL_BODY="$HOST_LABEL daily backup completed successfully.
+
+Date:      $TODAY
+Duration:  $DURATION_HUMAN
+Size:      $SIZE
+Snapshots: $SNAPSHOT_COUNT
+Source:    $SOURCE_DISPLAY
+Dest:      kreknin:$DEST_PATH
+
+Components:
+- /opt/books/{api,job-scheduler,task-runner,ntfy,books-ops-mcp}
+- /usr/docker/{minio/data, elasticsearch/snapshots, traefik/{letsencrypt,data}}
+- /etc/{ssh,hosts,cron.d}
+- /root/.ssh
+- DB dumps: books-db (mariadb), mongo (shared), books-job-scheduler-mongo
+- ES snapshot: $ES_SNAP
+
+Log: $LOG"
+email_send "[$HOST_LABEL] backup OK $TODAY" "$EMAIL_BODY"
+
+echo "$HOST_LABEL backup OK: $NTFY_BODY"
+echo "===== $HOST_LABEL backup done $(date -Is) ====="