admin/.wiki/sources/books-vds-backup-daily-kreknin-2026-05-25.md

title, type, tags, ingested, raw_path, updated

title	type	tags	ingested	raw_path	updated
books VDS daily backup → kreknin — pipeline standup 2026-05-25	source	backup books-vds kreknin rsync elasticsearch-snapshot curl-smtp cron ntfy	2026-05-25	../../.tasks/books-vds-backup-daily-kreknin.md	2026-05-25

books VDS daily backup → kreknin

Поднят ежедневный backup pipeline с ../entities/books-vds (89.253.255.133 / host4g.ru / CentOS 7) → ../entities/kreknin-synology (195.19.90.188) в 06:00 MSK. Параллель к vds-kzntsv-bootstrap-2026-05-20 §backup и ruvds-backup-daily-kreknin-2026-05-24 (pattern mirror), но source = CentOS 7 с hybrid SSH-compose + Portainer стек.

Источник истины — .tasks/books-vds-backup-daily-kreknin.md и scripts/books-vds-backup-daily-kreknin/.

Background — почему отдельно

Прошлые сессии (и [[../concepts/vds-kzntsv-ssh-access]] до 2026-05-25) ошибочно утверждали что books-стек живёт на ../entities/vds-kzntsv. Это не так: books на отдельном клиентском VDS 89.253.255.133. User указал на ошибку явно: «books VDS - это совсем другой сервер, клиентский». См. ../entities/books-vds для актуального factsheet.

Scope

Path on books VDS	Reason	Approx size
/opt/books/{api,job-scheduler,task-runner,ntfy,books-ops-mcp}	App configs (config/default.json overrides)	< 100 MB
/usr/docker/minio/data	MinIO blobs (S3) — основная масса	~4.5 GB
/usr/docker/elasticsearch/snapshots	ES snapshot files (repo kreknin)	< 1 MB (read_me index only)
/usr/docker/traefik/{letsencrypt,data}	acme.json + traefik.yml	< 5 MB
/etc/{ssh,hosts,cron.d}	system config	< 50 KB
/root/.ssh	root SSH state	< 5 KB
mariadb.sql.gz (live dump)	books-db full dump	~270 MB
mongo.archive.gz (live dump)	shared mongo (root auth)	~170 KB
job-scheduler-mongo.archive.gz (live dump)	scheduler agenda (no auth)	~3 MB

Total после initial sync: 4.87 GB (du -sh reported 4.8 GB после rsync).

Не бэкапятся: /var/lib/docker/, /var/log/, transient buildx volumes.

Decisions log

• Tool = bash + rsync + curl (no rclone, no msmtp). Linux native, no extra installs. Pattern mirror VDS-infra run.sh.
• ES snapshot via REST API (filesystem repo kreknin), не data-dir rsync. Required one-time stack edit /usr/docker/elasticsearch/docker-compose.yml: добавлены env path.repo=/snapshots + bind ./snapshots:/snapshots, ES recreated. Снимки атомарны и Lucene-consistent.
• Email via curl --ssl-reqd --url smtps://...:465 (implicit TLS), не msmtp. CentOS 7 + EOL repos = msmtp install painful; curl native + Yandex SMTP works. Initial attempt с smtp://...:465 + --ssl-reqd дала 30-sec timeout (curl попытался STARTTLS на TLS-only-from-start порту) — исправлено переходом на smtps://.
• RFC 5322 CRLF headers + Date + MIME-Version + Content-Type в email-body. Yandex отвергает без Date. Initial body без них принимался при STARTTLS теста, но for production safety — full headers.
• StrictHostKeyChecking=yes (НЕ accept-new) — CentOS 7 OpenSSH 7.4 не поддерживает accept-new (added in 7.6). kreknin host key pre-populated в /root/.ssh/known_hosts на bootstrap step.
• Retention 7 daily snapshots (matches VDS + RUVDS pattern). ES snapshot pruning отдельно — via REST DELETE.
• ntfy.vds.kzntsv.site/vds-backup topic (shared канал с VDS + RUVDS + windows-host). Title BOOKS-VDS backup OK <date> отличает от других хостов на phone-side.
• books-job-scheduler-mongo без auth — MONGO_INITDB_ROOT_* env пустой, mongodump --archive без --uri достаточен. Shared mongo — root auth (urircreds).
• Cron 06:00 MSK — после VDS-infra (05:00), RUVDS (04:30), windows-host (05:30). Break перед US-EU business start.
• books-vds-portainer/full-env consolidated → books-vds/full-env (parity с vds-kzntsv/full-env one-file-per-host pattern).

Хронология standup'а (2026-05-25)

• 08:30: user-указание начать stateful-split-volume-copy под supervision. Probe vds-ops MCP показал что books-db нет на vds-kzntsv (только generic mariadb/mongo/minio). Wiki [[../concepts/vds-kzntsv-ssh-access]] (тогда books-ssh-access.md) врала что books живёт на vds-kzntsv.
• 08:45: user: «89.253.255.133 - books VDS, клиентский. Сделаем backup на kreknin». Pivot фокус.
• 08:48: SSH probe id_ed25519_books_ops → ready, root access OK. Inventory: hybrid Portainer + SSH-compose, hosting books-api/web/scheduler/task-runner/db + shared infra.
• 08:55: Portainer API token request — user создал claude-code-automation PAT в Portainer UI (portainer.kzntsv.site). Saved в pass books-vds-portainer/full-env. API tested OK. But elasticsearch/mongo/minio/books-db/traefik нет среди Portainer-managed stacks — they're SSH-compose. User: «после backup'а мигрируем максимум под Portainer».
• 08:58: ES probe — path.repo не настроен, snapshot API не работает без edit'а compose.
• 09:00: User approval на ES stack edit «сейчас, ES никто не использует». Sed edit docker-compose.yml + recreate. docker-compose 1.29 → docker 26 incompat (KeyError: 'ContainerConfig') → manual docker rm renamed-stuck → docker-compose up -d clean → ES ready in 30s. Snapshot repo kreknin registered + test snap SUCCESS → deleted.
• 09:14: Wrote scripts/books-vds-backup-daily-kreknin/{run.sh,.env.example,README.md} локально. Built .env from pass entries (books-vds + vds-kzntsv NTFY + snolla-smtp SMTP). Deployed /opt/stacks/backup/{run.sh,.env} на VDS via scp. Generated kreknin-key ed25519 на VDS. Authorized pubkey on kreknin via workstation id_ed25519_kreknin. Pre-populated kreknin host key в /root/.ssh/known_hosts (CentOS 7 no accept-new).
• 09:15: Smoke run start. DB dumps OK (mariadb 270MB, mongo 170KB, scheduler-mongo 2.8MB). ES snapshot SUCCESS. rsync started.
• 09:29: rsync done, 4.87 GB in 13m28s (7.6 MB/s). ntfy push отправлен. Email timeout 30s — bug в email_send.
• 09:30: Diagnose. SMTP_PORT=465 + smtp:// + --ssl-reqd = curl попытался STARTTLS на implicit-TLS-only порту. Fix: smtps:// schema. Plus CRLF headers + Date header. Re-tested: SMTP send OK.
• 09:32: Updated run.sh re-deployed. Cron installed /etc/cron.d/books-vds-backup (0 6 * * * root /opt/stacks/backup/run.sh). crond active.

Pipeline state

• Live: cron books-vds-backup, daily 06:00 MSK, root user.
• Verified: initial sync 4.87 GB end-to-end + ntfy push отправлен.
• Phone-side ntfy verify: pending user confirmation (был ли push BOOKS-VDS backup OK 2026-05-25 на ntfy app в общем канале vds-backup).
• Email verify: sent после fix (см. 09:32 fix). User должен подтвердить получение [BOOKS-VDS] backup OK <date> или standalone fix-test message в inbox.
• Day-2 verify: automatic at 06:00 MSK 2026-05-26 — first cron-trigger.
• Retention prune verify: automatic at day-8 (когда snapshot count > 7).

Bugs fixed during smoke

• SMTP timeout 30s — smtp://...:465 + --ssl-reqd = curl STARTTLS на implicit-TLS-only порту. Server ждёт TLS handshake, curl ждёт plain EHLO, deadlock. Fix: smtps:// scheme.
• email RFC 5322 — initial body had no Date/MIME headers, only From/To/Subject. Yandex отвергает без Date. Fix: добавлены Date, MIME-Version, Content-Type CRLF headers.
• CentOS 7 OpenSSH 7.4 no accept-new — StrictHostKeyChecking=accept-new parse-error. Fix: StrictHostKeyChecking=yes + pre-populated known_hosts.
• docker-compose 1.29 ↔ docker 26 KeyError ContainerConfig — recreate failed mid-way, container renamed-stuck. Manual docker rm -f + clean up -d rescued.

Open follow-ups (не блокер)

• Migrate SSH-compose стеки под Portainer (отдельная таска, user-requested после backup live). Targets: elasticsearch, mongo, minio, books-db, traefik, proxy-chain.
• SSH-аудит books VDS — отдельный аудит по pattern vds-kzntsv-ssh-access. На сейчас знаем только root key id_ed25519_books_ops под root — других пользователей не enum'ил. Pre-existing keys unknown.
• kreknin space audit — quarterly check du -sh /volume1/NetBackup/* чтобы не упереться в 5.7T limit.
• wd40 backup: рассмотреть второй backup target (B2 / Glacier) для resilience — см. ../concepts/future-resilient-architecture-goals.

Атомарный revert

# На books VDS:
ssh root@89.253.255.133 'rm /etc/cron.d/books-vds-backup; rm -rf /opt/stacks/backup /var/log/books-vds-backup'

# Revert ES path.repo edit:
ssh root@89.253.255.133 'cd /usr/docker/elasticsearch && \
  cp docker-compose.yml.bak-pre-snapshots-2026-05-25 docker-compose.yml && \
  docker rm -f elasticsearch && docker-compose up -d'

# На kreknin (через SSH):
ssh vitya@195.19.90.188 'rm -rf /volume1/NetBackup/books-vds'
# Remove the books-vds-backup-20260525 pubkey line from authorized_keys.

Cross-refs

• Source host: ../entities/books-vds
• Backup target: ../entities/kreknin-synology
• Pattern parent: vds-kzntsv-bootstrap-2026-05-20 §backup (msmtp + rsync --link-dest)
• Sibling pipeline: ruvds-backup-daily-kreknin-2026-05-24 (rclone+SFTP)
• Notification: общий ntfy topic vds-backup на ntfy.vds.kzntsv.site.
• Misnaming legacy: ../concepts/vds-kzntsv-ssh-access (бывший books-ssh-access до 2026-05-25).