admin/scripts/books-vds-backup-daily-kreknin/README.md

# books-vds-backup-daily-kreknin

Daily backup pipeline: **books VDS** (`89.253.255.133` / host4g.ru CentOS 7) → **kreknin Synology** (`195.19.90.188:/volume1/NetBackup/books-vds/<date>/`) via SSH + rsync.

Schedule: `0 6 * * *` Europe/Moscow (after VDS-infra at 05:00, after windows-host at 05:30). Pattern mirrors `scripts/vds-backup-rsync-kreknin/run.sh`.

## What's backed up

| Component | Method | Source |
|---|---|---|
| MariaDB (books-db) | `mariadb-dump --all-databases --single-transaction` | container `books-db` |
| Mongo (shared) | `mongodump --archive` w/ root auth | container `mongo` |
| Mongo (job-scheduler) | `mongodump --archive` no-auth | container `books-job-scheduler-mongo` |
| Elasticsearch | REST snapshot API → fs repo `kreknin` → rsync | container `elasticsearch` (path.repo=/snapshots) |
| MinIO blobs | rsync direct (immutable content-addressed) | `/usr/docker/minio/data` |
| App configs | rsync | `/opt/books/{api,job-scheduler,task-runner,ntfy,books-ops-mcp}` |
| Traefik state | rsync | `/usr/docker/traefik/{letsencrypt,data}` (acme.json + config) |
| System config | rsync | `/etc/{ssh,hosts,cron.d}`, `/root/.ssh` |

Retention: **7 daily snapshots** on kreknin (+ on ES repo). Older — pruned at end of each run.

## Notifications

Dual-channel after every run:

- **ntfy** → topic `vds-backup` on `ntfy.vds.kzntsv.site` (shared с VDS + RUVDS + windows-host backups).
  - Success: title `BOOKS-VDS backup OK <date>`, tag `green_circle`.
  - Fail: title `BOOKS-VDS backup FAILED <date>`, tag `red_circle`, priority `high`.
- **Email** via Yandex SMTP (`noreply@snolla.com` → `OPS_NOTIFY_EMAIL`).
  - Sent via `curl --ssl-reqd --url smtp://...:587` (no msmtp dep on CentOS 7).
  - Subject: `[BOOKS-VDS] backup OK <date>` / `[BOOKS-VDS] backup FAILED <date>`.

If notify-channels fail — backup itself does NOT fail (`|| true`).

## Files

- `run.sh` — backup logic. Synced to `/opt/stacks/backup/run.sh` on books VDS.
- `.env.example` — placeholder. Live `.env` built from `pass show` entries during install.

## Initial install (rebuild from scratch)

```bash
# From workstation, with id_ed25519_books_ops authorized:
ssh root@89.253.255.133 'mkdir -p /opt/stacks/backup'
scp scripts/books-vds-backup-daily-kreknin/run.sh root@89.253.255.133:/opt/stacks/backup/

# Build .env locally from pass entries:
{
  echo "DEST_USER=vitya"
  echo "DEST_HOST=195.19.90.188"
  echo "DEST_BASE=/volume1/NetBackup/books-vds"
  echo "RETENTION_DAYS=7"
  echo "SSH_KEY=/opt/stacks/backup/kreknin-key"
  echo "ES_SNAPSHOT_REPO=kreknin"
  pass show books-vds/full-env | grep -E '^(BOOKS_DB_ROOT_PASSWORD|BOOKS_MONGO_)'
  pass show vds-kzntsv/full-env | grep -E '^NTFY_'
  pass show snolla-smtp/full-env | grep -E '^(SMTP_|OPS_NOTIFY_EMAIL)'
} | scp /dev/stdin root@89.253.255.133:/opt/stacks/backup/.env
ssh root@89.253.255.133 'chmod 600 /opt/stacks/backup/.env && chmod 755 /opt/stacks/backup/run.sh'

# Generate kreknin SSH key on books VDS:
ssh root@89.253.255.133 'ssh-keygen -t ed25519 -f /opt/stacks/backup/kreknin-key -N "" -C "books-vds-backup"'

# Get pubkey and authorize it on kreknin (workstation pivots — books VDS has no kreknin SSH yet):
PUBKEY=$(ssh root@89.253.255.133 cat /opt/stacks/backup/kreknin-key.pub)
ssh -i ~/.ssh/id_ed25519_kreknin vitya@195.19.90.188 \
  "echo '$PUBKEY' >> ~/.ssh/authorized_keys && mkdir -p /volume1/NetBackup/books-vds"

# Verify SSH books → kreknin
ssh root@89.253.255.133 'ssh -i /opt/stacks/backup/kreknin-key -o StrictHostKeyChecking=accept-new vitya@195.19.90.188 hostname'

# Smoke run (manual)
ssh root@89.253.255.133 '/opt/stacks/backup/run.sh'

# Install cron
ssh root@89.253.255.133 'echo "0 6 * * * root /opt/stacks/backup/run.sh" > /etc/cron.d/books-vds-backup && chmod 644 /etc/cron.d/books-vds-backup'
```

## Decisions log

- **ES snapshot via REST API**, not data-dir rsync. Filesystem snapshot repo (`type=fs`, `location=/snapshots`, `compress=true`) registered as `kreknin` on 2026-05-25. Required one-time stack edit (add `path.repo=/snapshots` env + `./snapshots:/snapshots` bind in `/usr/docker/elasticsearch/docker-compose.yml`).
- **`curl --ssl-reqd` SMTP** instead of msmtp — avoid CentOS 7 msmtp install (EOL repo issues). Native `curl` works on STARTTLS port 587.
- **No `tasks.yml`-style auth on `books-job-scheduler-mongo`** — container has no `MONGO_INITDB_ROOT_*` env. `mongodump --archive` без `--uri` достаточен.
- **rsync /opt/books + /usr/docker/{minio,elasticsearch/snapshots,traefik}** — bind paths из inspect 2026-05-25; не `/var/lib/docker/volumes/` (named volumes для контейнеров пустые, кроме portainer/scheduler-mongo).
- **`books-vds-portainer/full-env` consolidated** into `books-vds/full-env` for one-file-per-host pattern parity with `vds-kzntsv/full-env`.
- **No `BackupConfig` snapshot/git pre-commit-like step** — VDS-infra `run.sh` тоже не делает, и нет аналогичной системы на books VDS.

## Smoke run (manual)

```bash
ssh root@89.253.255.133 '/opt/stacks/backup/run.sh'
# Watch log:
ssh root@89.253.255.133 'tail -F /var/log/books-vds-backup/$(date +%Y-%m-%d).log'
```

Verify on kreknin:
```bash
ssh vitya@195.19.90.188 'du -sh /volume1/NetBackup/books-vds/*/'
```

## Atomic revert (uninstall)

```bash
# On books VDS:
ssh root@89.253.255.133 'rm /etc/cron.d/books-vds-backup; rm -rf /opt/stacks/backup /var/log/books-vds-backup'

# On kreknin (via SSH):
ssh vitya@195.19.90.188 'rm -rf /volume1/NetBackup/books-vds'
# Remove the books-vds pubkey line from authorized_keys (one line with 'books-vds-backup' comment).

# On books VDS — revert ES path.repo change:
ssh root@89.253.255.133 'cd /usr/docker/elasticsearch && \
  cp docker-compose.yml.bak-pre-snapshots-2026-05-25 docker-compose.yml && \
  docker rm -f elasticsearch && docker-compose up -d'
```