backup(books-vds): daily 06:00 MSK pipeline to kreknin live 🟢
- scripts/books-vds-backup-daily-kreknin/ — run.sh + .env.example + README - ES path.repo bootstrap (one-time stack edit, snapshot repo 'kreknin' registered) - curl smtps://yandex:465 email (CRLF + Date headers, no msmtp dep on CentOS 7) - StrictHostKeyChecking=yes + pre-populated known_hosts (CentOS 7 no accept-new) - cron /etc/cron.d/books-vds-backup live, initial sync 4.87 GB in 13m28s, ntfy push sent - wiki: entities/books-vds.md + sources/books-vds-backup-daily-kreknin-2026-05-25.md created - wiki: concepts/books-ssh-access → vds-kzntsv-ssh-access (was lying about books on vds-kzntsv) - .tasks/books-vds-backup-daily-kreknin.md closed 🟢 pending phone-side verify Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
111
.wiki/sources/books-vds-backup-daily-kreknin-2026-05-25.md
Normal file
111
.wiki/sources/books-vds-backup-daily-kreknin-2026-05-25.md
Normal file
@@ -0,0 +1,111 @@
|
||||
---
|
||||
title: books VDS daily backup → kreknin — pipeline standup 2026-05-25
|
||||
type: source
|
||||
tags: [backup, books-vds, kreknin, rsync, elasticsearch-snapshot, curl-smtp, cron, ntfy]
|
||||
ingested: 2026-05-25
|
||||
raw_path: ../../.tasks/books-vds-backup-daily-kreknin.md
|
||||
updated: 2026-05-25
|
||||
---
|
||||
|
||||
# books VDS daily backup → kreknin
|
||||
|
||||
Поднят ежедневный backup pipeline с [[../entities/books-vds]] (`89.253.255.133` / host4g.ru / CentOS 7) → [[../entities/kreknin-synology]] (`195.19.90.188`) в **06:00 MSK**. Параллель к [[vds-kzntsv-bootstrap-2026-05-20]] §backup и [[ruvds-backup-daily-kreknin-2026-05-24]] (pattern mirror), но source = CentOS 7 с hybrid SSH-compose + Portainer стек.
|
||||
|
||||
Источник истины — `.tasks/books-vds-backup-daily-kreknin.md` и `scripts/books-vds-backup-daily-kreknin/`.
|
||||
|
||||
## Background — почему отдельно
|
||||
|
||||
Прошлые сессии (и `[[../concepts/vds-kzntsv-ssh-access]]` до 2026-05-25) ошибочно утверждали что books-стек живёт на [[../entities/vds-kzntsv]]. **Это не так:** books на отдельном клиентском VDS `89.253.255.133`. User указал на ошибку явно: «books VDS - это совсем другой сервер, клиентский». См. [[../entities/books-vds]] для актуального factsheet.
|
||||
|
||||
## Scope
|
||||
|
||||
| Path on books VDS | Reason | Approx size |
|
||||
|---|---|---|
|
||||
| `/opt/books/{api,job-scheduler,task-runner,ntfy,books-ops-mcp}` | App configs (`config/default.json` overrides) | < 100 MB |
|
||||
| `/usr/docker/minio/data` | MinIO blobs (S3) — основная масса | ~4.5 GB |
|
||||
| `/usr/docker/elasticsearch/snapshots` | ES snapshot files (repo `kreknin`) | < 1 MB (read_me index only) |
|
||||
| `/usr/docker/traefik/{letsencrypt,data}` | acme.json + traefik.yml | < 5 MB |
|
||||
| `/etc/{ssh,hosts,cron.d}` | system config | < 50 KB |
|
||||
| `/root/.ssh` | root SSH state | < 5 KB |
|
||||
| `mariadb.sql.gz` (live dump) | books-db full dump | ~270 MB |
|
||||
| `mongo.archive.gz` (live dump) | shared mongo (root auth) | ~170 KB |
|
||||
| `job-scheduler-mongo.archive.gz` (live dump) | scheduler agenda (no auth) | ~3 MB |
|
||||
|
||||
Total после initial sync: **4.87 GB** (`du -sh` reported 4.8 GB после rsync).
|
||||
|
||||
Не бэкапятся: `/var/lib/docker/`, `/var/log/`, transient buildx volumes.
|
||||
|
||||
## Decisions log
|
||||
|
||||
- **Tool = bash + rsync + curl** (no rclone, no msmtp). Linux native, no extra installs. Pattern mirror VDS-infra `run.sh`.
|
||||
- **ES snapshot via REST API** (filesystem repo `kreknin`), не data-dir rsync. Required one-time stack edit `/usr/docker/elasticsearch/docker-compose.yml`: добавлены env `path.repo=/snapshots` + bind `./snapshots:/snapshots`, ES recreated. Снимки атомарны и Lucene-consistent.
|
||||
- **Email via `curl --ssl-reqd --url smtps://...:465`** (implicit TLS), не msmtp. CentOS 7 + EOL repos = msmtp install painful; curl native + Yandex SMTP works. Initial attempt с `smtp://...:465` + `--ssl-reqd` дала 30-sec timeout (curl попытался STARTTLS на TLS-only-from-start порту) — исправлено переходом на `smtps://`.
|
||||
- **RFC 5322 CRLF headers + Date + MIME-Version + Content-Type** в email-body. Yandex отвергает без Date. Initial body без них принимался при STARTTLS теста, но for production safety — full headers.
|
||||
- **`StrictHostKeyChecking=yes`** (НЕ `accept-new`) — CentOS 7 OpenSSH 7.4 не поддерживает `accept-new` (added in 7.6). kreknin host key pre-populated в `/root/.ssh/known_hosts` на bootstrap step.
|
||||
- **Retention 7 daily snapshots** (matches VDS + RUVDS pattern). ES snapshot pruning отдельно — via REST DELETE.
|
||||
- **`ntfy.vds.kzntsv.site/vds-backup` topic** (shared канал с VDS + RUVDS + windows-host). Title `BOOKS-VDS backup OK <date>` отличает от других хостов на phone-side.
|
||||
- **books-job-scheduler-mongo без auth** — `MONGO_INITDB_ROOT_*` env пустой, `mongodump --archive` без `--uri` достаточен. Shared `mongo` — root auth (urircreds).
|
||||
- **Cron 06:00 MSK** — после VDS-infra (05:00), RUVDS (04:30), windows-host (05:30). Break перед US-EU business start.
|
||||
- **`books-vds-portainer/full-env` consolidated** → `books-vds/full-env` (parity с `vds-kzntsv/full-env` one-file-per-host pattern).
|
||||
|
||||
## Хронология standup'а (2026-05-25)
|
||||
|
||||
- **08:30:** user-указание начать `stateful-split-volume-copy` под supervision. Probe `vds-ops` MCP показал что books-db нет на vds-kzntsv (только generic mariadb/mongo/minio). Wiki `[[../concepts/vds-kzntsv-ssh-access]]` (тогда `books-ssh-access.md`) врала что books живёт на vds-kzntsv.
|
||||
- **08:45:** user: «89.253.255.133 - books VDS, клиентский. Сделаем backup на kreknin». Pivot фокус.
|
||||
- **08:48:** SSH probe `id_ed25519_books_ops` → ready, root access OK. Inventory: hybrid Portainer + SSH-compose, hosting books-api/web/scheduler/task-runner/db + shared infra.
|
||||
- **08:55:** Portainer API token request — user создал `claude-code-automation` PAT в Portainer UI (`portainer.kzntsv.site`). Saved в pass `books-vds-portainer/full-env`. API tested OK. But `elasticsearch`/`mongo`/`minio`/`books-db`/`traefik` нет среди Portainer-managed stacks — they're SSH-compose. User: «после backup'а мигрируем максимум под Portainer».
|
||||
- **08:58:** ES probe — `path.repo` не настроен, snapshot API не работает без edit'а compose.
|
||||
- **09:00:** User approval на ES stack edit «сейчас, ES никто не использует». Sed edit `docker-compose.yml` + recreate. docker-compose 1.29 → docker 26 incompat (`KeyError: 'ContainerConfig'`) → manual `docker rm` renamed-stuck → `docker-compose up -d` clean → ES ready in 30s. Snapshot repo `kreknin` registered + test snap SUCCESS → deleted.
|
||||
- **09:14:** Wrote `scripts/books-vds-backup-daily-kreknin/{run.sh,.env.example,README.md}` локально. Built `.env` from pass entries (books-vds + vds-kzntsv NTFY + snolla-smtp SMTP). Deployed `/opt/stacks/backup/{run.sh,.env}` на VDS via scp. Generated `kreknin-key` ed25519 на VDS. Authorized pubkey on kreknin via workstation `id_ed25519_kreknin`. Pre-populated kreknin host key в `/root/.ssh/known_hosts` (CentOS 7 no accept-new).
|
||||
- **09:15:** Smoke run start. DB dumps OK (mariadb 270MB, mongo 170KB, scheduler-mongo 2.8MB). ES snapshot SUCCESS. rsync started.
|
||||
- **09:29:** rsync done, 4.87 GB in **13m28s** (7.6 MB/s). ntfy push отправлен. Email **timeout 30s** — bug в email_send.
|
||||
- **09:30:** Diagnose. SMTP_PORT=465 + `smtp://` + `--ssl-reqd` = curl попытался STARTTLS на implicit-TLS-only порту. Fix: `smtps://` schema. Plus CRLF headers + Date header. Re-tested: SMTP send OK.
|
||||
- **09:32:** Updated run.sh re-deployed. Cron installed `/etc/cron.d/books-vds-backup` (0 6 * * * root /opt/stacks/backup/run.sh). crond active.
|
||||
|
||||
## Pipeline state
|
||||
|
||||
- **Live:** cron `books-vds-backup`, daily 06:00 MSK, root user.
|
||||
- **Verified:** initial sync 4.87 GB end-to-end + ntfy push отправлен.
|
||||
- **Phone-side ntfy verify:** **pending user confirmation** (был ли push `BOOKS-VDS backup OK 2026-05-25` на ntfy app в общем канале `vds-backup`).
|
||||
- **Email verify:** sent после fix (см. 09:32 fix). User должен подтвердить получение `[BOOKS-VDS] backup OK <date>` или standalone fix-test message в inbox.
|
||||
- **Day-2 verify:** automatic at 06:00 MSK 2026-05-26 — first cron-trigger.
|
||||
- **Retention prune verify:** automatic at day-8 (когда snapshot count > 7).
|
||||
|
||||
## Bugs fixed during smoke
|
||||
|
||||
- **SMTP timeout 30s** — `smtp://...:465` + `--ssl-reqd` = curl STARTTLS на implicit-TLS-only порту. Server ждёт TLS handshake, curl ждёт plain EHLO, deadlock. Fix: `smtps://` scheme.
|
||||
- **email RFC 5322** — initial body had no Date/MIME headers, only From/To/Subject. Yandex отвергает без Date. Fix: добавлены Date, MIME-Version, Content-Type CRLF headers.
|
||||
- **CentOS 7 OpenSSH 7.4 no accept-new** — `StrictHostKeyChecking=accept-new` parse-error. Fix: `StrictHostKeyChecking=yes` + pre-populated known_hosts.
|
||||
- **docker-compose 1.29 ↔ docker 26 KeyError ContainerConfig** — recreate failed mid-way, container renamed-stuck. Manual `docker rm -f` + clean `up -d` rescued.
|
||||
|
||||
## Open follow-ups (не блокер)
|
||||
|
||||
- **Migrate SSH-compose стеки под Portainer** (отдельная таска, user-requested после backup live). Targets: elasticsearch, mongo, minio, books-db, traefik, proxy-chain.
|
||||
- **SSH-аудит books VDS** — отдельный аудит по pattern [[vds-kzntsv-ssh-access]]. На сейчас знаем только root key `id_ed25519_books_ops` под root — других пользователей не enum'ил. Pre-existing keys unknown.
|
||||
- **kreknin space audit** — quarterly check `du -sh /volume1/NetBackup/*` чтобы не упереться в 5.7T limit.
|
||||
- **wd40 backup**: рассмотреть второй backup target (B2 / Glacier) для resilience — см. [[../concepts/future-resilient-architecture-goals]].
|
||||
|
||||
## Атомарный revert
|
||||
|
||||
```bash
|
||||
# На books VDS:
|
||||
ssh root@89.253.255.133 'rm /etc/cron.d/books-vds-backup; rm -rf /opt/stacks/backup /var/log/books-vds-backup'
|
||||
|
||||
# Revert ES path.repo edit:
|
||||
ssh root@89.253.255.133 'cd /usr/docker/elasticsearch && \
|
||||
cp docker-compose.yml.bak-pre-snapshots-2026-05-25 docker-compose.yml && \
|
||||
docker rm -f elasticsearch && docker-compose up -d'
|
||||
|
||||
# На kreknin (через SSH):
|
||||
ssh vitya@195.19.90.188 'rm -rf /volume1/NetBackup/books-vds'
|
||||
# Remove the books-vds-backup-20260525 pubkey line from authorized_keys.
|
||||
```
|
||||
|
||||
## Cross-refs
|
||||
|
||||
- Source host: [[../entities/books-vds]]
|
||||
- Backup target: [[../entities/kreknin-synology]]
|
||||
- Pattern parent: [[vds-kzntsv-bootstrap-2026-05-20]] §backup (msmtp + rsync --link-dest)
|
||||
- Sibling pipeline: [[ruvds-backup-daily-kreknin-2026-05-24]] (rclone+SFTP)
|
||||
- Notification: общий ntfy topic `vds-backup` на `ntfy.vds.kzntsv.site`.
|
||||
- Misnaming legacy: [[../concepts/vds-kzntsv-ssh-access]] (бывший `books-ssh-access` до 2026-05-25).
|
||||
Reference in New Issue
Block a user