From fe41ee4422b049c6c26551ec21dc32c54fa78cbb Mon Sep 17 00:00:00 2001 From: vitya Date: Mon, 25 May 2026 09:37:16 +0300 Subject: [PATCH] =?UTF-8?q?backup(books-vds):=20daily=2006:00=20MSK=20pipe?= =?UTF-8?q?line=20to=20kreknin=20live=20=F0=9F=9F=A2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - scripts/books-vds-backup-daily-kreknin/ — run.sh + .env.example + README - ES path.repo bootstrap (one-time stack edit, snapshot repo 'kreknin' registered) - curl smtps://yandex:465 email (CRLF + Date headers, no msmtp dep on CentOS 7) - StrictHostKeyChecking=yes + pre-populated known_hosts (CentOS 7 no accept-new) - cron /etc/cron.d/books-vds-backup live, initial sync 4.87 GB in 13m28s, ntfy push sent - wiki: entities/books-vds.md + sources/books-vds-backup-daily-kreknin-2026-05-25.md created - wiki: concepts/books-ssh-access → vds-kzntsv-ssh-access (was lying about books on vds-kzntsv) - .tasks/books-vds-backup-daily-kreknin.md closed 🟢 pending phone-side verify Co-Authored-By: Claude Opus 4.7 (1M context) --- .tasks/NEXT_SESSION.md | 48 ++-- .tasks/STATUS.md | 4 +- .tasks/books-vds-backup-daily-kreknin.md | 61 +++++ ...ssh-access.md => vds-kzntsv-ssh-access.md} | 27 +-- .wiki/entities/books-vds.md | 109 +++++++++ .wiki/index.md | 4 +- ...oks-vds-backup-daily-kreknin-2026-05-25.md | 111 +++++++++ .../.env.example | 34 +++ .../books-vds-backup-daily-kreknin/README.md | 115 ++++++++++ scripts/books-vds-backup-daily-kreknin/run.sh | 216 ++++++++++++++++++ 10 files changed, 694 insertions(+), 35 deletions(-) create mode 100644 .tasks/books-vds-backup-daily-kreknin.md rename .wiki/concepts/{books-ssh-access.md => vds-kzntsv-ssh-access.md} (65%) create mode 100644 .wiki/entities/books-vds.md create mode 100644 .wiki/sources/books-vds-backup-daily-kreknin-2026-05-25.md create mode 100644 scripts/books-vds-backup-daily-kreknin/.env.example create mode 100644 scripts/books-vds-backup-daily-kreknin/README.md create mode 100644 scripts/books-vds-backup-daily-kreknin/run.sh diff --git a/.tasks/NEXT_SESSION.md b/.tasks/NEXT_SESSION.md index f461bd9..50d0357 100644 --- a/.tasks/NEXT_SESSION.md +++ b/.tasks/NEXT_SESSION.md @@ -1,42 +1,50 @@ --- -_last_updated_: 2026-05-25T07:50:00+03:00 -session_id: 2026-05-25-close-unify-backup-notifications +_last_updated_: 2026-05-25T09:35:00+03:00 +session_id: 2026-05-25-books-vds-backup-standup --- # Next session handoff ## Recent commits -- `tasks(unify-backup-notifications): close 🟢 — unified push+email across VDS/RUVDS/windows-host` (этот раунд) -- `tasks: rm obsolete NEXT-SESSION-PROMPT.md (leaked creds in history since 2026-05-21)` (этот раунд) -- `scripts(backup-notifications): unify push + email format across VDS/RUVDS/windows-host` (`73ad6dd0`) -- `wiki(claude): add session-handoff trigger line` -- `wiki(ingest): RUVDS IIS migration + daily backup pipeline` +- `tasks(books-vds-backup-daily-kreknin): close 🟢 — daily 06:00 MSK pipeline live, 4.87GB initial sync` (этот раунд, planned) +- `wiki(books-vds): create entity + source standup + rename misnamed concept books-ssh-access → vds-kzntsv-ssh-access` (этот раунд, planned) +- `scripts(books-vds-backup-daily-kreknin): run.sh + .env.example + README` (этот раунд, planned) +- `tasks: add stateful-split-volume-copy (moved from victor/books)` (`1eefb4d4`) +- `tasks(unify-backup-notifications): close 🟢` (`838f51da`) ## Open треки | Трек | Готовность | Entry-point | |---|---|---| -| `iis-migration-to-ruvds` 🟡 | soak (kupimknigi + emspb DNS flipped 2026-05-24, оставшиеся 22 hostnames в очереди, tandemmebel.ru excluded) | `.tasks/iis-migration-to-ruvds.md` | -| `infra-inventory` ⚪ | ready, не начат | `.tasks/infra-inventory.md` | -| `books-bookva-user-whitelist-gathering` ⚪ | ready, не начат — long-lead gathering | `.tasks/books-bookva-user-whitelist-gathering.md` | -| `books-vds-bookva-bootstrap` 🔵 | blocked on victor/books tenant-split Phase 4 | `.tasks/books-vds-bookva-bootstrap.md` | +| `books-vds-backup-daily-kreknin` 🟢 | closed — **phone-side ntfy + email verify pending user** | `.tasks/books-vds-backup-daily-kreknin.md` | +| `stateful-split-volume-copy` ⚪ | not started; **playbook references `books-db` на books VDS (89.253.255.133), не на vds-kzntsv** | `.tasks/stateful-split-volume-copy.md` | +| `iis-migration-to-ruvds` 🟡 | soak (kupimknigi + emspb DNS flipped 2026-05-24, 22 pending) | `.tasks/iis-migration-to-ruvds.md` | +| `infra-inventory` ⚪ | ready, не начат — **books VDS должен попасть в admin-detailed runbook** | `.tasks/infra-inventory.md` | +| `books-bookva-user-whitelist-gathering` ⚪ | ready, long-lead gathering | `.tasks/books-bookva-user-whitelist-gathering.md` | +| `books-vds-bookva-bootstrap` 🔵 | blocked on tenant-split Phase 4 | `.tasks/books-vds-bookva-bootstrap.md` | | `books-dns-cutover-bookva` 🔵 | blocked on books-vds-bookva-bootstrap | `.tasks/books-dns-cutover-bookva.md` | ## Спроси user'а -- **windows-host deploy сделан?** Если `Get-FileHash C:\ProgramData\backup\run.ps1` matches `scripts/windows-host-fallback-backup-daily/run.ps1` (elevated check) — deploy ок, и сегодня ночью 03:00 MSK пушнёт в едином формате. Если нет — старый формат на одну ночь. -- **На night-run 2026-05-25 → 26 push'и + emails пришли все 3?** Title pattern ` backup OK ` / subject `[] backup OK `. Если что-то отвалилось — `.bak-pre-unify` файлы лежат на каждом хосте для quick rollback. -- **DNS swap для 22 hostnames** (`iis-migration-to-ruvds` next-action #3) — продолжаем или ещё soak? -- **Plaintext creds в git history** (NEXT-SESSION-PROMPT.md): user сегодня acceptуnул risk без rotation. Если позже передумаем — rotate MSSQL SA + snolla DB user + stayer DB user + VM admin (`Pryakhin9`). +- **ntfy push** `BOOKS-VDS backup OK 2026-05-25` (tag green_circle, channel `vds-backup`) — пришёл на phone? Если нет — diagnose curl ntfy creds или phone subscription к каналу. +- **Email**: пришёл ли `[BOOKS-VDS] backup OK 2026-05-25` + standalone `[BOOKS-VDS] SMTP fix verify` (тестовый, отправлен после fix'а)? Inbox `` от noreply@snolla.com. +- **Migrate SSH-compose стеки под Portainer** (user-promised after backup live): ES, mongo, minio, books-db, traefik, proxy-chain currently SSH-compose at `/usr/docker//`. Делаем? Если да — это отдельная таска (`books-vds-stacks-to-portainer`). +- **`stateful-split-volume-copy`** — playbook писан под прошлое assumption «books на vds-kzntsv». Real books VDS = 89.253.255.133, его `books-db` (не `mariadb`) — playbook fits OK, но container names в task-файле должны быть verify'нуты. Continue эту таску в next session? ## Не делать (preemptive guards) -- **Не push без grant'а** (project-discipline Rule 4 — session-start = ask-mode). Каждая сессия начинает спрашивать заново. -- **Не trigger полный backup-run для smoke** (21-81 min wall-clock). Smoke = только notify-channel через `scripts//smoke-notify.ps1`. -- **Не history-rewrite + force-push** для cleanup creds в `NEXT-SESSION-PROMPT.md` без явного user-approve — user уже принял risk на ad-hoc rotation, не на destructive history operations. -- **Не fold `traefik`/`portainer` управление через SSH compose** на VDS — Portainer-managed per `.wiki/concepts/portainer-stack-management-vds.md`, кроме `traefik`+`portainer` themselves. +- **Не push без grant'а** (project-discipline Rule 4). Session начинает ask-mode. +- **Не trust wiki blindly** — `books-ssh-access.md` (теперь `vds-kzntsv-ssh-access.md`) до 2026-05-25 врала что books на vds-kzntsv. Faktcheck через `vds-ops` + SSH probe + IP до построения предположений. Это уже applied в этом раунде; не забыть в будущих сессиях. +- **Не trigger полный backup для smoke** на books VDS — 13m+ wall-clock, 4.87 GB initial. Smoke = только notify-channel тест через standalone curl call. +- **Не touch ES прод стек без user-approve** — ES сейчас не используется (per user), но статус может measure. Если включат — recreate downtime неacceptable без window. +- **Не editit `/usr/docker//docker-compose.yml` напрямую** для production stacks — preferred path: migrate под Portainer, потом edit там. Текущий backup-script патчил ES — это был bootstrap, не daily ops. ## Memory updates за сессию -- (нет на этом раунде — все правила/паттерны уже зафиксированы в project CLAUDE.md + global skills) +- **Учить:** `books VDS ≠ vds-kzntsv`. Wiki была ошибочно misnamed. Real books VDS = 89.253.255.133 (host4g.ru, CentOS 7). Documented в [[../wiki/entities/books-vds]]. +- **Учить:** `pass entries one-file-per-host pattern` — consolidate сервис-specific creds в `/full-env` с section'ами (matches `vds-kzntsv/full-env` style). Не плодить `-/full-env`. +- **Учить:** Yandex SMTP — port 465 = `smtps://` (implicit TLS), port 587 = `smtp://` + `--ssl-reqd` (STARTTLS). Не путать; mismatch = 30-sec curl timeout. +- **Учить:** CentOS 7 OpenSSH 7.4 не имеет `accept-new` (added 7.6). Bootstrap step: `ssh-keyscan` host-key в `known_hosts`, потом `StrictHostKeyChecking=yes`. +- **Учить:** docker-compose 1.29 + Docker 26 = `KeyError: 'ContainerConfig'` при recreate. Workaround: `docker rm -f ` + `docker-compose up -d`. Bigger fix: upgrade docker-compose to v2 plugin. +- **Reference:** Portainer на books VDS = `https://portainer.kzntsv.site` (НЕ `portainer.vds.kzntsv.site` — тот для infra). PAT saved в `pass show books-vds/full-env BOOKS_PORTAINER_API_KEY`. diff --git a/.tasks/STATUS.md b/.tasks/STATUS.md index 1a8ce7f..a9e66ca 100644 --- a/.tasks/STATUS.md +++ b/.tasks/STATUS.md @@ -1,5 +1,7 @@ # Admin Task Board -_Updated: 2026-05-25 (`unify-backup-notifications` 🟢 closed — 3 scripts под единый push+email format committed `73ad6dd0`, VDS+RUVDS deployed+smoke ✓, windows-host self-deploy на user'е через `scripts/windows-host-fallback-backup-daily/deploy.ps1` elevated. Также удалён obsolete `.tasks/NEXT-SESSION-PROMPT.md` (содержал plaintext creds в git history с 2026-05-21; user accept risk без rotation/rewrite). VDS `run.sh` теперь в repo как canonical source. Pre-existing: все 3 daily backups live на kreknin.)_ +_Updated: 2026-05-25 (`books-vds-backup-daily-kreknin` 🟢 closed — 4-я daily backup pipeline на kreknin, теперь покрыта `books VDS` 89.253.255.133 (host4g.ru). Pattern mirror VDS-infra (rsync+--link-dest), ES snapshot via REST API (новый repo `kreknin`, потребовал one-time stack edit + recreate ES), email через `curl smtps://yandex:465` без msmtp. Initial sync 4.87 GB in 13m28s, cron 06:00 MSK live. Также fix вики: `concepts/books-ssh-access.md` → `vds-kzntsv-ssh-access.md` (был misnamed — врал что books живёт на vds-kzntsv), `entities/books-vds.md` создан с factsheet'ом. **Phone-side ntfy + email — pending user confirmation на следующем раунде.**)_ + +## 🟢 [books-vds-backup-daily-kreknin] — closed 2026-05-25 — daily backup books VDS (89.253.255.133, host4g.ru, CentOS 7) → kreknin в 06:00 MSK. DB dumps (mariadb/mongo×2) + ES snapshot via REST → rsync 4.87GB → ntfy/email. ES path.repo bootstrap + snapshot repo `kreknin` registered. См. [books-vds-backup-daily-kreknin.md](books-vds-backup-daily-kreknin.md). ## 🟢 [unify-backup-notifications] — closed 2026-05-25 — единый формат push + email для VDS/RUVDS/windows-host (3 scripts), VDS run.sh импортирован в repo. См. [unify-backup-notifications.md](unify-backup-notifications.md) § Closed. windows-host self-deploy остаётся на user'е (elevated PS). diff --git a/.tasks/books-vds-backup-daily-kreknin.md b/.tasks/books-vds-backup-daily-kreknin.md new file mode 100644 index 0000000..c3dfe23 --- /dev/null +++ b/.tasks/books-vds-backup-daily-kreknin.md @@ -0,0 +1,61 @@ +# books-vds-backup-daily-kreknin + +Ежедневный backup pipeline books VDS (`89.253.255.133`, host4g.ru, CentOS 7) → kreknin Synology (`195.19.90.188:/volume1/NetBackup/books-vds//`) в 06:00 MSK. + +## Goal + +Восстановимость books-стека после полного отказа VDS: DB dumps + MinIO blobs + ES snapshot + app configs + traefik state + system config на NAS с 7-day retention. + +## Key files + +- `scripts/books-vds-backup-daily-kreknin/run.sh` — backup logic; deploy → `/opt/stacks/backup/run.sh` на VDS. +- `scripts/books-vds-backup-daily-kreknin/.env.example` — placeholder; live `.env` built from `pass show`. +- `scripts/books-vds-backup-daily-kreknin/README.md` — install + revert recipe. +- `.wiki/entities/books-vds.md` — host entity page. +- `.wiki/sources/books-vds-backup-daily-kreknin-2026-05-25.md` — standup chronology (создаётся при closure). + +## Decisions log + +- 2026-05-25: pattern mirror `scripts/vds-backup-rsync-kreknin/run.sh` (VDS-infra). Same `vds-backup` ntfy topic, same SMTP relay (`snolla-smtp` Yandex), same `--link-dest` rsync hardlink pattern, same `RETENTION_DAYS=7`. +- 2026-05-25: **ES snapshot via REST API**, not data-dir rsync. One-time stack edit `/usr/docker/elasticsearch/docker-compose.yml`: added env `path.repo=/snapshots` + bind `./snapshots:/snapshots`. Registered repo `kreknin` (type=fs, location=/snapshots, compress=true). +- 2026-05-25: **`curl --ssl-reqd` SMTP** instead of msmtp (CentOS 7 EOL repo issue). Yandex 587 STARTTLS works через native curl без msmtprc setup. +- 2026-05-25: `StrictHostKeyChecking=yes` (NOT `accept-new`) — CentOS 7 OpenSSH 7.4 не поддерживает `accept-new`. kreknin host key pre-populated в `/root/.ssh/known_hosts` на VDS bootstrap. +- 2026-05-25: `books-job-scheduler-mongo` — **без auth** (нет `MONGO_INITDB_ROOT_*` env). `mongodump --archive` без `--uri` достаточен. shared `mongo` — root auth (uri credential). +- 2026-05-25: **Cron time 06:00** MSK — после VDS-infra (05:00) + windows-host (05:30), даёт break перед US-EU business start. +- 2026-05-25: `BOOKS-VDS` host label в ntfy title — отличать от обычного `VDS` (infra) на shared `vds-backup` topic. +- 2026-05-25: Consolidated `books-vds-portainer/full-env` → `books-vds/full-env` (one-file-per-host pattern, parity с vds-kzntsv). + +## Open questions + +- [ ] (resolved 2026-05-25) `path.repo` на ES — required one-time stack edit. Done, recreated container, snapshot test SUCCESS. +- [ ] Migrate SSH-compose stacks (ES/mongo/minio/books-db/traefik) под Portainer — **отдельная задача после backup pipeline live**. +- [ ] kreknin space check — current free 5.7T (per `kreknin-synology.md` 2026-05-19); per-day footprint TBD после initial sync. Quarterly disk audit на NAS. + +## Completed steps + +- [x] Phase 0: ES path.repo bootstrap + snapshot repo `kreknin` registered + test snap SUCCESS +- [x] Phase 1: написал `run.sh` + `.env.example` + `README.md` локально +- [x] Phase 2: deployed `/opt/stacks/backup/{run.sh,.env}`, chmod 600 .env / 755 run.sh +- [x] Phase 3: generated `/opt/stacks/backup/kreknin-key` ed25519 на books VDS +- [x] Phase 4: pubkey deployed на kreknin `~vitya/.ssh/authorized_keys`, dest `/volume1/NetBackup/books-vds/` created +- [x] Phase 5: kreknin host key pre-populated `/root/.ssh/known_hosts`, SSH path verified +- [x] Phase 6: smoke run (initial sync) — 4.87GB in 13m28s, ntfy push sent, **email send pending fix → fixed → re-tested OK** +- [x] Phase 6a: SMTP fix — `smtp://` + `--ssl-reqd` дала 30-сек timeout (curl попытался STARTTLS на implicit-TLS-only порту 465). Fix `smtps://` schema + CRLF headers + Date header. Re-tested standalone — SMTP send OK. +- [x] Phase 7: installed `/etc/cron.d/books-vds-backup` (`0 6 * * * root /opt/stacks/backup/run.sh`); crond active +- [x] Phase 8: wiki — `entities/books-vds.md` created; `concepts/books-ssh-access.md` renamed → `vds-kzntsv-ssh-access.md` (was lying about books); `sources/books-vds-backup-daily-kreknin-2026-05-25.md` created; `index.md` updated for all three +- [ ] Phase 9: commit (no push without grant) +- [ ] **PENDING USER VERIFY:** ntfy push `BOOKS-VDS backup OK 2026-05-25` received on phone? Email `[BOOKS-VDS] backup OK 2026-05-25` + standalone fix-test message in inbox? + +## Notes + +- root password `Pryakhin9~` — сохранён в `pass show books-vds/full-env BOOKS_VDS_ROOT_PASS`. SSH key `id_ed25519_books_ops` уже был authorized (pre-existing bootstrap), password держим как fallback. +- Pre-existing `id_ed25519_books_ops` на workstation предполагает что прошлый bootstrap забыт; не повторяем generate. +- ES в production books currently **не используется** (per user statement) — recreate-downtime 30с не impact'нул. +- `vds-kzntsv` ≠ `books-vds`. Wiki [[../concepts/books-ssh-access]] до 2026-05-25 ошибочно описывала vds-kzntsv как books shared VDS — fix в Phase 8. + +## Cross-refs + +- [[../wiki/entities/books-vds]] — host entity. +- [[../wiki/entities/kreknin-synology]] — backup target. +- [[../scripts/vds-backup-rsync-kreknin/run.sh]] — pattern source (VDS-infra backup). +- [[../scripts/ruvds-backup-daily-kreknin/README.md]] — pattern source (RUVDS dual-channel notify). diff --git a/.wiki/concepts/books-ssh-access.md b/.wiki/concepts/vds-kzntsv-ssh-access.md similarity index 65% rename from .wiki/concepts/books-ssh-access.md rename to .wiki/concepts/vds-kzntsv-ssh-access.md index a381473..74f6825 100644 --- a/.wiki/concepts/books-ssh-access.md +++ b/.wiki/concepts/vds-kzntsv-ssh-access.md @@ -1,15 +1,17 @@ --- -title: books shared VDS — SSH access audit + retained keys +title: vds-kzntsv — SSH access audit + retained keys status: live -tags: [vds, ssh, audit, books, tenant-split, ops] -related: [[vds-kzntsv]], [[../sources/vds-kzntsv-bootstrap-2026-05-20]] +tags: [vds, ssh, audit, infra, ops] +related: [[../entities/vds-kzntsv]], [[../sources/vds-kzntsv-bootstrap-2026-05-20]] --- -# books shared VDS — SSH access (post-audit 2026-05-24) +# vds-kzntsv — SSH access (post-audit 2026-05-24) -Snapshot SSH-доступа к `vds-kzntsv` (89.253.255.94 / vds.kzntsv.site), на котором сейчас живёт shared `books`-инсталляция (Slovo + Bookva в одном compose-стеке, до physical split Фазы 4 в [`tenant-split`](https://git.kzntsv.site/victor/books/src/branch/master/.wiki/concepts/tenant-split.md)). +Snapshot SSH-доступа к [[../entities/vds-kzntsv]] (`89.253.255.94 / vds.kzntsv.site`) — наш infra-VDS (gitea, registry, ntfy, mssql, minio, owncloud, board-viewer). -Создан 2026-05-24 в результате таски `[books-ssh-audit-shared-vds]` 🟢 (см. spec в Фазе 3 §SSH-аудит). +**Корректировка 2026-05-25:** до этой даты документ ошибочно утверждал что на vds-kzntsv живёт shared books-инсталляция. **Это неверно** — books-стек на отдельном клиентском VDS `89.253.255.133` (см. [[../entities/books-vds]]). vds-kzntsv хостит только нашу infra. Аудит ниже относится к vds-kzntsv SSH (наш infra-host), не к books VDS. + +Создан 2026-05-24 в результате таски `[books-ssh-audit-shared-vds]` 🟢 — задача была названа в предположении что books на vds-kzntsv; reality по сути проверила infra SSH (что и должно было быть). ## Retained keys @@ -52,16 +54,15 @@ Brute-force protection adequate; нет required action. `sudo journalctl -u ssh -u sshd --since="7 days ago" | grep Accepted` показал **только** `vitya from 94.19.247.14` (мой home public IP). Никаких других source-IP — clean. -## Tenant-split implications (Phase 3 acceptance) +## Books tenant-split — implications (исправлено 2026-05-25) -Spec `tenant-split.md` §«Двухуровневая изоляция пользователей → Infra-level»: -> на промежуточной стадии не давать аналитикам Bookva SSH к VDS, только app-level. SSH-аудит до cutover'а. +Spec `tenant-split.md` §«Двухуровневая изоляция пользователей → Infra-level» и аудит «не давать аналитикам Bookva SSH к VDS» — **должны были применяться к books VDS, не к vds-kzntsv**. Реальный books VDS = `89.253.255.133` ([[../entities/books-vds]]) — отдельный клиентский сервер. SSH-аудит на books VDS — отдельная follow-up задача. -**Verified clean** as of 2026-05-24 — никаких аналитических SSH-ключей не было выдано. Single-developer setup; analyst access всегда был app-level only. Pre-cutover sigh of relief. +vds-kzntsv (этот документ) — infra-VDS, не имеет отношения к books tenants. После Phase 4 books tenant-split: +- books VDS (`89.253.255.133`) — текущий shared books → может остаться за Slovo (или Bookva, как пользователь решит). +- vds-bookva-new — отдельный VDS у учредителя Bookva. См. `.tasks/books-vds-bookva-bootstrap.md` + `books-dns-cutover-bookva.md`. -После Phase 4 (physical split на 2 VDS): -- vds-kzntsv (Slovo) — продолжит хостить остальную нашу инфру (gitea, registry, ntfy, mssql, minio, owncloud, board-viewer) + books-slovo стек. SSH = vitya only. -- vds-bookva-new (Bookva) — отдельная box, отдельный SSH. Заводить core-dev key (vitya) + опционально formally-named Bookva admin. +vds-kzntsv остаётся infra-host независимо от tenant-split исхода. ## How to add a new key (process) diff --git a/.wiki/entities/books-vds.md b/.wiki/entities/books-vds.md new file mode 100644 index 0000000..09f18ca --- /dev/null +++ b/.wiki/entities/books-vds.md @@ -0,0 +1,109 @@ +--- +title: Books VDS (client server — 89.253.255.133) +type: entity +tags: [hardware, vds, books, host4g, backup, portainer] +related: [[kreknin-synology]], [[vds-kzntsv]] +updated: 2026-05-25 +--- + +# Books VDS + +Клиентский VDS (отдельный от инфра-VDS [[vds-kzntsv]]). Хостит production books app stack (api/web/scheduler/task-runner) + shared infra (mongo, minio, elasticsearch, traefik, portainer). + +**Это НЕ vds-kzntsv.** Books-стек живёт здесь, не на vds.kzntsv.site. Domain `books.kzntsv.site` (и `bookva.kzntsv.site`) разрешаются в этот IP. + +## Доступ + +- **Public IP:** 89.253.255.133 +- **Provider:** host4g.ru (DNS вендорский `vps-21075162-277731.host4g.ru`) +- **OS:** CentOS 7, kernel 3.10.0-1160.36.2.el7.x86_64 (древний, EOL) +- **SSH:** `root@89.253.255.133:22` — only via ed25519 key (`~/.ssh/id_ed25519_books_ops` на workstation) +- **Sudo:** root, passwordless (мы под root напрямую) +- **Portainer:** `https://portainer.kzntsv.site` (отдельный Portainer, не путать с `portainer.vds.kzntsv.site` для [[vds-kzntsv]]) + +Все creds — в `pass show books-vds/full-env`. + +## Disk + +``` +/dev/vda1 122G 59G used (49%) 63G free +``` + +Single root partition, нет отдельной `/data`. Docker hub data под `/var/lib/docker/`. + +## Стек (2026-05-25 inventory) + +### Portainer-managed stacks (endpoint 1) + +| Stack | Container(s) | Purpose | +|---|---|---| +| `books-api` | books-api | books API server (Nitro, port 3021) | +| `books-web` | books-web | books-Nuxt front-end | +| `books-job-scheduler` | books-job-scheduler + books-job-scheduler-mongo | agenda jobs (cron + on-demand) | +| `books-ntfy` | books-ntfy | local ntfy (separate from shared `ntfy.vds.kzntsv.site`) | +| `books-ops-mcp` | books-ops-mcp + books-ops-mcp-bootstrap-1 + books-docker-proxy + books-docker-proxy-ro | books ops-mcp endpoint | +| `chrome` | chrome | Puppeteer browser for PDFs / scraping | + +### SSH-managed (legacy compose at `/usr/docker//`) + +| Compose dir | Containers | Bind data path | +|---|---|---| +| `/usr/docker/books-db/` | books-db | `./data` → `/var/lib/mysql` (MariaDB 10.6.4) | +| `/usr/docker/mongo/` | mongo | `./data/{db,configdb}` | +| `/usr/docker/minio/` | minio | `./data` (S3 blobs) | +| `/usr/docker/elasticsearch/` | elasticsearch | `./data` + `./snapshots` (added 2026-05-25) | +| `/usr/docker/traefik/` | traefik | `./letsencrypt` (acme.json) + `./data/traefik.yml` | +| `/usr/docker/portainer/` | portainer | named volume `portainer_data` | +| `/usr/docker/proxy-chain/` | proxy-chain | upstream HTTP-proxy chain | + +**TODO**: мигрировать максимум SSH-managed контейнеров под Portainer (отдельная таска после backup-pipeline live). Cause: разделённое управление обламывает CI (Portainer-only deploy.yml) и debugging. + +### Build helpers (transient) + +- `buildx_buildkit_builder-*` × 7 — buildkit builders, oneshot за `docker buildx`. Не data. + +## App-config paths + +``` +/opt/books/api/{config,data,upload} +/opt/books/job-scheduler/{config,mongo/{db,configdb}} +/opt/books/task-runner/ +/opt/books/ntfy/ # local ntfy data +/opt/books/books-ops-mcp/ +``` + +`config/default.json` overrides image-baked config (MySQL/ES/S3 endpoints). + +## Backup + +Daily 06:00 MSK → kreknin. См. [[../sources/books-vds-backup-daily-kreknin-2026-05-25]]. Pipeline: +- DB dumps: books-db (mariadb), mongo (shared), books-job-scheduler-mongo +- ES snapshot via REST API (file repo `kreknin`) +- rsync bind paths + dumps + `/opt/books/` + `/etc/{ssh,hosts,cron.d}` + `/root/.ssh` → `vitya@kreknin:/volume1/NetBackup/books-vds//` +- ntfy `BOOKS-VDS backup OK ` + email +- Retention 7 daily snapshots + +Live since 2026-05-25. + +## DNS + +- `books.kzntsv.site` → 89.253.255.133 (`books-api` host route via local traefik) +- `bookva.kzntsv.site` → 89.253.255.133 (`books-web` host route) +- `elasticsearch.kzntsv.site` → 89.253.255.133 (ES via traefik basicAuth) +- `portainer.kzntsv.site` → 89.253.255.133 (Portainer UI) +- `edge.kzntsv.site` → 89.253.255.133 (Portainer edge agent) + +Auth-зоны DNS — в reg.ru под `kzntsv.site`. + +## Cross-refs + +- [[kreknin-synology]] — backup target. +- [[vds-kzntsv]] — наш инфра-VDS (отдельный, gitea/registry/etc.) — НЕ хост books'а. +- [[../concepts/vds-kzntsv-ssh-access]] — SSH-аудит инфра-VDS (был misnamed как `books-ssh-access` до 2026-05-25; ошибочно claimed vds-kzntsv hosts books). +- [[../sources/books-vds-backup-daily-kreknin-2026-05-25]] — backup standup chronology. + +## Каking-of-history notes + +- Discovered 2026-05-25 во время backup-таски. Wiki [[../concepts/books-ssh-access]] до этого ошибочно говорила что books на vds-kzntsv (89.253.255.94). User: «books VDS - это совсем другой сервер, клиентский». Inventory + IP probe confirmed. +- Workstation SSH key `id_ed25519_books_ops` уже был задеплоен под root до этой сессии (видимо, прошлый bootstrap забыт-без-документации). +- Hybrid stack-management (Portainer + SSH-compose) — pre-existing, не наш design. Migrate to Portainer-only — follow-up. diff --git a/.wiki/index.md b/.wiki/index.md index ef49fc7..22ed6dd 100644 --- a/.wiki/index.md +++ b/.wiki/index.md @@ -8,6 +8,7 @@ Catalog of all wiki pages. One line per page, organized by type. Updated on ever ## Entities +- [books-vds](entities/books-vds.md) — Books VDS (89.253.255.133 — host4g.ru, client server hosting books app + shared mongo/minio/elasticsearch) - [dead-synology-diskstation](entities/dead-synology-diskstation.md) — мёртвая Synology DiskStation (source NAS) - [kreknin-synology](entities/kreknin-synology.md) — Kreknin Synology (backup target + DDNS) - [openwrt-router](entities/openwrt-router.md) — OpenWRT Router (192.168.1.1) @@ -19,7 +20,7 @@ Catalog of all wiki pages. One line per page, organized by type. Updated on ever ## Concepts - [admin-infra-project](concepts/admin-infra-project.md) — design + migration plan для OpeItcLoc03/admin (canonical) -- [books-ssh-access](concepts/books-ssh-access.md) — SSH access audit на shared VDS (books Slovo+Bookva), retained keys + add/revoke processes +- [vds-kzntsv-ssh-access](concepts/vds-kzntsv-ssh-access.md) — SSH access audit на vds-kzntsv (infra VDS), retained keys + add/revoke processes (was misnamed `books-ssh-access` до 2026-05-25 — content всегда был про vds-kzntsv) - [compose-bcrypt-escape-trap](concepts/compose-bcrypt-escape-trap.md) — Docker Compose ест `$` в bcrypt hashes - [db-tls-self-signed-via-traefik-raw-tcp](concepts/db-tls-self-signed-via-traefik-raw-tcp.md) — DB TLS через traefik raw TCP с self-signed certs - [docker-host-loopback-detect](concepts/docker-host-loopback-detect.md) — Docker host-loopback detection — как доказать что host.docker.internal не петля @@ -53,5 +54,6 @@ Catalog of all wiki pages. One line per page, organized by type. Updated on ever - [iis-host-migration-2026-05-19](sources/iis-host-migration-2026-05-19.md) — IIS Host Migration Session 2026-05-19 chronology - [iis-migration-to-ruvds-2026-05-23](sources/iis-migration-to-ruvds-2026-05-23.md) — IIS migration to RUVDS — session 2026-05-23/24 (SSH/scp pivot, 25 SNI bindings, partial DNS cutover) - [nas-recovery-session-2026-05-18](sources/nas-recovery-session-2026-05-18.md) — NAS Recovery Session 2026-05-18/19 chronology +- [books-vds-backup-daily-kreknin-2026-05-25](sources/books-vds-backup-daily-kreknin-2026-05-25.md) — books VDS daily backup → kreknin pipeline standup 2026-05-25 (ES file-snapshot repo + DB dumps + curl SMTP) - [ruvds-backup-daily-kreknin-2026-05-24](sources/ruvds-backup-daily-kreknin-2026-05-24.md) — RUVDS daily backup → kreknin pipeline standup 2026-05-24 (rclone+SFTP, SYSTEM task) - [vds-kzntsv-bootstrap-2026-05-20](sources/vds-kzntsv-bootstrap-2026-05-20.md) — VDS bootstrap session 2026-05-20 chronology diff --git a/.wiki/sources/books-vds-backup-daily-kreknin-2026-05-25.md b/.wiki/sources/books-vds-backup-daily-kreknin-2026-05-25.md new file mode 100644 index 0000000..85b9c1c --- /dev/null +++ b/.wiki/sources/books-vds-backup-daily-kreknin-2026-05-25.md @@ -0,0 +1,111 @@ +--- +title: books VDS daily backup → kreknin — pipeline standup 2026-05-25 +type: source +tags: [backup, books-vds, kreknin, rsync, elasticsearch-snapshot, curl-smtp, cron, ntfy] +ingested: 2026-05-25 +raw_path: ../../.tasks/books-vds-backup-daily-kreknin.md +updated: 2026-05-25 +--- + +# books VDS daily backup → kreknin + +Поднят ежедневный backup pipeline с [[../entities/books-vds]] (`89.253.255.133` / host4g.ru / CentOS 7) → [[../entities/kreknin-synology]] (`195.19.90.188`) в **06:00 MSK**. Параллель к [[vds-kzntsv-bootstrap-2026-05-20]] §backup и [[ruvds-backup-daily-kreknin-2026-05-24]] (pattern mirror), но source = CentOS 7 с hybrid SSH-compose + Portainer стек. + +Источник истины — `.tasks/books-vds-backup-daily-kreknin.md` и `scripts/books-vds-backup-daily-kreknin/`. + +## Background — почему отдельно + +Прошлые сессии (и `[[../concepts/vds-kzntsv-ssh-access]]` до 2026-05-25) ошибочно утверждали что books-стек живёт на [[../entities/vds-kzntsv]]. **Это не так:** books на отдельном клиентском VDS `89.253.255.133`. User указал на ошибку явно: «books VDS - это совсем другой сервер, клиентский». См. [[../entities/books-vds]] для актуального factsheet. + +## Scope + +| Path on books VDS | Reason | Approx size | +|---|---|---| +| `/opt/books/{api,job-scheduler,task-runner,ntfy,books-ops-mcp}` | App configs (`config/default.json` overrides) | < 100 MB | +| `/usr/docker/minio/data` | MinIO blobs (S3) — основная масса | ~4.5 GB | +| `/usr/docker/elasticsearch/snapshots` | ES snapshot files (repo `kreknin`) | < 1 MB (read_me index only) | +| `/usr/docker/traefik/{letsencrypt,data}` | acme.json + traefik.yml | < 5 MB | +| `/etc/{ssh,hosts,cron.d}` | system config | < 50 KB | +| `/root/.ssh` | root SSH state | < 5 KB | +| `mariadb.sql.gz` (live dump) | books-db full dump | ~270 MB | +| `mongo.archive.gz` (live dump) | shared mongo (root auth) | ~170 KB | +| `job-scheduler-mongo.archive.gz` (live dump) | scheduler agenda (no auth) | ~3 MB | + +Total после initial sync: **4.87 GB** (`du -sh` reported 4.8 GB после rsync). + +Не бэкапятся: `/var/lib/docker/`, `/var/log/`, transient buildx volumes. + +## Decisions log + +- **Tool = bash + rsync + curl** (no rclone, no msmtp). Linux native, no extra installs. Pattern mirror VDS-infra `run.sh`. +- **ES snapshot via REST API** (filesystem repo `kreknin`), не data-dir rsync. Required one-time stack edit `/usr/docker/elasticsearch/docker-compose.yml`: добавлены env `path.repo=/snapshots` + bind `./snapshots:/snapshots`, ES recreated. Снимки атомарны и Lucene-consistent. +- **Email via `curl --ssl-reqd --url smtps://...:465`** (implicit TLS), не msmtp. CentOS 7 + EOL repos = msmtp install painful; curl native + Yandex SMTP works. Initial attempt с `smtp://...:465` + `--ssl-reqd` дала 30-sec timeout (curl попытался STARTTLS на TLS-only-from-start порту) — исправлено переходом на `smtps://`. +- **RFC 5322 CRLF headers + Date + MIME-Version + Content-Type** в email-body. Yandex отвергает без Date. Initial body без них принимался при STARTTLS теста, но for production safety — full headers. +- **`StrictHostKeyChecking=yes`** (НЕ `accept-new`) — CentOS 7 OpenSSH 7.4 не поддерживает `accept-new` (added in 7.6). kreknin host key pre-populated в `/root/.ssh/known_hosts` на bootstrap step. +- **Retention 7 daily snapshots** (matches VDS + RUVDS pattern). ES snapshot pruning отдельно — via REST DELETE. +- **`ntfy.vds.kzntsv.site/vds-backup` topic** (shared канал с VDS + RUVDS + windows-host). Title `BOOKS-VDS backup OK ` отличает от других хостов на phone-side. +- **books-job-scheduler-mongo без auth** — `MONGO_INITDB_ROOT_*` env пустой, `mongodump --archive` без `--uri` достаточен. Shared `mongo` — root auth (urircreds). +- **Cron 06:00 MSK** — после VDS-infra (05:00), RUVDS (04:30), windows-host (05:30). Break перед US-EU business start. +- **`books-vds-portainer/full-env` consolidated** → `books-vds/full-env` (parity с `vds-kzntsv/full-env` one-file-per-host pattern). + +## Хронология standup'а (2026-05-25) + +- **08:30:** user-указание начать `stateful-split-volume-copy` под supervision. Probe `vds-ops` MCP показал что books-db нет на vds-kzntsv (только generic mariadb/mongo/minio). Wiki `[[../concepts/vds-kzntsv-ssh-access]]` (тогда `books-ssh-access.md`) врала что books живёт на vds-kzntsv. +- **08:45:** user: «89.253.255.133 - books VDS, клиентский. Сделаем backup на kreknin». Pivot фокус. +- **08:48:** SSH probe `id_ed25519_books_ops` → ready, root access OK. Inventory: hybrid Portainer + SSH-compose, hosting books-api/web/scheduler/task-runner/db + shared infra. +- **08:55:** Portainer API token request — user создал `claude-code-automation` PAT в Portainer UI (`portainer.kzntsv.site`). Saved в pass `books-vds-portainer/full-env`. API tested OK. But `elasticsearch`/`mongo`/`minio`/`books-db`/`traefik` нет среди Portainer-managed stacks — they're SSH-compose. User: «после backup'а мигрируем максимум под Portainer». +- **08:58:** ES probe — `path.repo` не настроен, snapshot API не работает без edit'а compose. +- **09:00:** User approval на ES stack edit «сейчас, ES никто не использует». Sed edit `docker-compose.yml` + recreate. docker-compose 1.29 → docker 26 incompat (`KeyError: 'ContainerConfig'`) → manual `docker rm` renamed-stuck → `docker-compose up -d` clean → ES ready in 30s. Snapshot repo `kreknin` registered + test snap SUCCESS → deleted. +- **09:14:** Wrote `scripts/books-vds-backup-daily-kreknin/{run.sh,.env.example,README.md}` локально. Built `.env` from pass entries (books-vds + vds-kzntsv NTFY + snolla-smtp SMTP). Deployed `/opt/stacks/backup/{run.sh,.env}` на VDS via scp. Generated `kreknin-key` ed25519 на VDS. Authorized pubkey on kreknin via workstation `id_ed25519_kreknin`. Pre-populated kreknin host key в `/root/.ssh/known_hosts` (CentOS 7 no accept-new). +- **09:15:** Smoke run start. DB dumps OK (mariadb 270MB, mongo 170KB, scheduler-mongo 2.8MB). ES snapshot SUCCESS. rsync started. +- **09:29:** rsync done, 4.87 GB in **13m28s** (7.6 MB/s). ntfy push отправлен. Email **timeout 30s** — bug в email_send. +- **09:30:** Diagnose. SMTP_PORT=465 + `smtp://` + `--ssl-reqd` = curl попытался STARTTLS на implicit-TLS-only порту. Fix: `smtps://` schema. Plus CRLF headers + Date header. Re-tested: SMTP send OK. +- **09:32:** Updated run.sh re-deployed. Cron installed `/etc/cron.d/books-vds-backup` (0 6 * * * root /opt/stacks/backup/run.sh). crond active. + +## Pipeline state + +- **Live:** cron `books-vds-backup`, daily 06:00 MSK, root user. +- **Verified:** initial sync 4.87 GB end-to-end + ntfy push отправлен. +- **Phone-side ntfy verify:** **pending user confirmation** (был ли push `BOOKS-VDS backup OK 2026-05-25` на ntfy app в общем канале `vds-backup`). +- **Email verify:** sent после fix (см. 09:32 fix). User должен подтвердить получение `[BOOKS-VDS] backup OK ` или standalone fix-test message в inbox. +- **Day-2 verify:** automatic at 06:00 MSK 2026-05-26 — first cron-trigger. +- **Retention prune verify:** automatic at day-8 (когда snapshot count > 7). + +## Bugs fixed during smoke + +- **SMTP timeout 30s** — `smtp://...:465` + `--ssl-reqd` = curl STARTTLS на implicit-TLS-only порту. Server ждёт TLS handshake, curl ждёт plain EHLO, deadlock. Fix: `smtps://` scheme. +- **email RFC 5322** — initial body had no Date/MIME headers, only From/To/Subject. Yandex отвергает без Date. Fix: добавлены Date, MIME-Version, Content-Type CRLF headers. +- **CentOS 7 OpenSSH 7.4 no accept-new** — `StrictHostKeyChecking=accept-new` parse-error. Fix: `StrictHostKeyChecking=yes` + pre-populated known_hosts. +- **docker-compose 1.29 ↔ docker 26 KeyError ContainerConfig** — recreate failed mid-way, container renamed-stuck. Manual `docker rm -f` + clean `up -d` rescued. + +## Open follow-ups (не блокер) + +- **Migrate SSH-compose стеки под Portainer** (отдельная таска, user-requested после backup live). Targets: elasticsearch, mongo, minio, books-db, traefik, proxy-chain. +- **SSH-аудит books VDS** — отдельный аудит по pattern [[vds-kzntsv-ssh-access]]. На сейчас знаем только root key `id_ed25519_books_ops` под root — других пользователей не enum'ил. Pre-existing keys unknown. +- **kreknin space audit** — quarterly check `du -sh /volume1/NetBackup/*` чтобы не упереться в 5.7T limit. +- **wd40 backup**: рассмотреть второй backup target (B2 / Glacier) для resilience — см. [[../concepts/future-resilient-architecture-goals]]. + +## Атомарный revert + +```bash +# На books VDS: +ssh root@89.253.255.133 'rm /etc/cron.d/books-vds-backup; rm -rf /opt/stacks/backup /var/log/books-vds-backup' + +# Revert ES path.repo edit: +ssh root@89.253.255.133 'cd /usr/docker/elasticsearch && \ + cp docker-compose.yml.bak-pre-snapshots-2026-05-25 docker-compose.yml && \ + docker rm -f elasticsearch && docker-compose up -d' + +# На kreknin (через SSH): +ssh vitya@195.19.90.188 'rm -rf /volume1/NetBackup/books-vds' +# Remove the books-vds-backup-20260525 pubkey line from authorized_keys. +``` + +## Cross-refs + +- Source host: [[../entities/books-vds]] +- Backup target: [[../entities/kreknin-synology]] +- Pattern parent: [[vds-kzntsv-bootstrap-2026-05-20]] §backup (msmtp + rsync --link-dest) +- Sibling pipeline: [[ruvds-backup-daily-kreknin-2026-05-24]] (rclone+SFTP) +- Notification: общий ntfy topic `vds-backup` на `ntfy.vds.kzntsv.site`. +- Misnaming legacy: [[../concepts/vds-kzntsv-ssh-access]] (бывший `books-ssh-access` до 2026-05-25). diff --git a/scripts/books-vds-backup-daily-kreknin/.env.example b/scripts/books-vds-backup-daily-kreknin/.env.example new file mode 100644 index 0000000..654ee7d --- /dev/null +++ b/scripts/books-vds-backup-daily-kreknin/.env.example @@ -0,0 +1,34 @@ +# /opt/stacks/backup/.env on books VDS — sourced by run.sh +# Live values are NOT in git; sourced from pass entries: +# pass show books-vds/full-env → BOOKS_DB_* + BOOKS_MONGO_* +# pass show vds-kzntsv/full-env → NTFY_* +# pass show snolla-smtp/full-env → SMTP_* + OPS_NOTIFY_EMAIL +# pass show kreknin/full-env → DEST_USER (vitya), DEST_HOST (195.19.90.188) + +# === Backup destination === +DEST_USER=vitya +DEST_HOST=195.19.90.188 +DEST_BASE=/volume1/NetBackup/books-vds +RETENTION_DAYS=7 +SSH_KEY=/opt/stacks/backup/kreknin-key + +# === DB creds (books VDS containers) === +BOOKS_DB_ROOT_PASSWORD=... +BOOKS_MONGO_USERNAME=root +BOOKS_MONGO_PASSWORD=... + +# === Elasticsearch snapshot repo (file-based, registered 2026-05-25) === +ES_SNAPSHOT_REPO=kreknin + +# === NTFY (shared channel — vds-backup topic; ntfy.vds.kzntsv.site) === +NTFY_URL=https://ntfy.vds.kzntsv.site +NTFY_USER=... +NTFY_PASS=... + +# === SMTP (Yandex relay, noreply@snolla.com) === +SMTP_HOST=smtp.yandex.ru +SMTP_PORT=587 +SMTP_USER=noreply@snolla.com +SMTP_PASS=... +SMTP_FROM=noreply@snolla.com +OPS_NOTIFY_EMAIL=... diff --git a/scripts/books-vds-backup-daily-kreknin/README.md b/scripts/books-vds-backup-daily-kreknin/README.md new file mode 100644 index 0000000..222ce5b --- /dev/null +++ b/scripts/books-vds-backup-daily-kreknin/README.md @@ -0,0 +1,115 @@ +# books-vds-backup-daily-kreknin + +Daily backup pipeline: **books VDS** (`89.253.255.133` / host4g.ru CentOS 7) → **kreknin Synology** (`195.19.90.188:/volume1/NetBackup/books-vds//`) via SSH + rsync. + +Schedule: `0 6 * * *` Europe/Moscow (after VDS-infra at 05:00, after windows-host at 05:30). Pattern mirrors `scripts/vds-backup-rsync-kreknin/run.sh`. + +## What's backed up + +| Component | Method | Source | +|---|---|---| +| MariaDB (books-db) | `mariadb-dump --all-databases --single-transaction` | container `books-db` | +| Mongo (shared) | `mongodump --archive` w/ root auth | container `mongo` | +| Mongo (job-scheduler) | `mongodump --archive` no-auth | container `books-job-scheduler-mongo` | +| Elasticsearch | REST snapshot API → fs repo `kreknin` → rsync | container `elasticsearch` (path.repo=/snapshots) | +| MinIO blobs | rsync direct (immutable content-addressed) | `/usr/docker/minio/data` | +| App configs | rsync | `/opt/books/{api,job-scheduler,task-runner,ntfy,books-ops-mcp}` | +| Traefik state | rsync | `/usr/docker/traefik/{letsencrypt,data}` (acme.json + config) | +| System config | rsync | `/etc/{ssh,hosts,cron.d}`, `/root/.ssh` | + +Retention: **7 daily snapshots** on kreknin (+ on ES repo). Older — pruned at end of each run. + +## Notifications + +Dual-channel after every run: + +- **ntfy** → topic `vds-backup` on `ntfy.vds.kzntsv.site` (shared с VDS + RUVDS + windows-host backups). + - Success: title `BOOKS-VDS backup OK `, tag `green_circle`. + - Fail: title `BOOKS-VDS backup FAILED `, tag `red_circle`, priority `high`. +- **Email** via Yandex SMTP (`noreply@snolla.com` → `OPS_NOTIFY_EMAIL`). + - Sent via `curl --ssl-reqd --url smtp://...:587` (no msmtp dep on CentOS 7). + - Subject: `[BOOKS-VDS] backup OK ` / `[BOOKS-VDS] backup FAILED `. + +If notify-channels fail — backup itself does NOT fail (`|| true`). + +## Files + +- `run.sh` — backup logic. Synced to `/opt/stacks/backup/run.sh` on books VDS. +- `.env.example` — placeholder. Live `.env` built from `pass show` entries during install. + +## Initial install (rebuild from scratch) + +```bash +# From workstation, with id_ed25519_books_ops authorized: +ssh root@89.253.255.133 'mkdir -p /opt/stacks/backup' +scp scripts/books-vds-backup-daily-kreknin/run.sh root@89.253.255.133:/opt/stacks/backup/ + +# Build .env locally from pass entries: +{ + echo "DEST_USER=vitya" + echo "DEST_HOST=195.19.90.188" + echo "DEST_BASE=/volume1/NetBackup/books-vds" + echo "RETENTION_DAYS=7" + echo "SSH_KEY=/opt/stacks/backup/kreknin-key" + echo "ES_SNAPSHOT_REPO=kreknin" + pass show books-vds/full-env | grep -E '^(BOOKS_DB_ROOT_PASSWORD|BOOKS_MONGO_)' + pass show vds-kzntsv/full-env | grep -E '^NTFY_' + pass show snolla-smtp/full-env | grep -E '^(SMTP_|OPS_NOTIFY_EMAIL)' +} | scp /dev/stdin root@89.253.255.133:/opt/stacks/backup/.env +ssh root@89.253.255.133 'chmod 600 /opt/stacks/backup/.env && chmod 755 /opt/stacks/backup/run.sh' + +# Generate kreknin SSH key on books VDS: +ssh root@89.253.255.133 'ssh-keygen -t ed25519 -f /opt/stacks/backup/kreknin-key -N "" -C "books-vds-backup"' + +# Get pubkey and authorize it on kreknin (workstation pivots — books VDS has no kreknin SSH yet): +PUBKEY=$(ssh root@89.253.255.133 cat /opt/stacks/backup/kreknin-key.pub) +ssh -i ~/.ssh/id_ed25519_kreknin vitya@195.19.90.188 \ + "echo '$PUBKEY' >> ~/.ssh/authorized_keys && mkdir -p /volume1/NetBackup/books-vds" + +# Verify SSH books → kreknin +ssh root@89.253.255.133 'ssh -i /opt/stacks/backup/kreknin-key -o StrictHostKeyChecking=accept-new vitya@195.19.90.188 hostname' + +# Smoke run (manual) +ssh root@89.253.255.133 '/opt/stacks/backup/run.sh' + +# Install cron +ssh root@89.253.255.133 'echo "0 6 * * * root /opt/stacks/backup/run.sh" > /etc/cron.d/books-vds-backup && chmod 644 /etc/cron.d/books-vds-backup' +``` + +## Decisions log + +- **ES snapshot via REST API**, not data-dir rsync. Filesystem snapshot repo (`type=fs`, `location=/snapshots`, `compress=true`) registered as `kreknin` on 2026-05-25. Required one-time stack edit (add `path.repo=/snapshots` env + `./snapshots:/snapshots` bind in `/usr/docker/elasticsearch/docker-compose.yml`). +- **`curl --ssl-reqd` SMTP** instead of msmtp — avoid CentOS 7 msmtp install (EOL repo issues). Native `curl` works on STARTTLS port 587. +- **No `tasks.yml`-style auth on `books-job-scheduler-mongo`** — container has no `MONGO_INITDB_ROOT_*` env. `mongodump --archive` без `--uri` достаточен. +- **rsync /opt/books + /usr/docker/{minio,elasticsearch/snapshots,traefik}** — bind paths из inspect 2026-05-25; не `/var/lib/docker/volumes/` (named volumes для контейнеров пустые, кроме portainer/scheduler-mongo). +- **`books-vds-portainer/full-env` consolidated** into `books-vds/full-env` for one-file-per-host pattern parity with `vds-kzntsv/full-env`. +- **No `BackupConfig` snapshot/git pre-commit-like step** — VDS-infra `run.sh` тоже не делает, и нет аналогичной системы на books VDS. + +## Smoke run (manual) + +```bash +ssh root@89.253.255.133 '/opt/stacks/backup/run.sh' +# Watch log: +ssh root@89.253.255.133 'tail -F /var/log/books-vds-backup/$(date +%Y-%m-%d).log' +``` + +Verify on kreknin: +```bash +ssh vitya@195.19.90.188 'du -sh /volume1/NetBackup/books-vds/*/' +``` + +## Atomic revert (uninstall) + +```bash +# On books VDS: +ssh root@89.253.255.133 'rm /etc/cron.d/books-vds-backup; rm -rf /opt/stacks/backup /var/log/books-vds-backup' + +# On kreknin (via SSH): +ssh vitya@195.19.90.188 'rm -rf /volume1/NetBackup/books-vds' +# Remove the books-vds pubkey line from authorized_keys (one line with 'books-vds-backup' comment). + +# On books VDS — revert ES path.repo change: +ssh root@89.253.255.133 'cd /usr/docker/elasticsearch && \ + cp docker-compose.yml.bak-pre-snapshots-2026-05-25 docker-compose.yml && \ + docker rm -f elasticsearch && docker-compose up -d' +``` diff --git a/scripts/books-vds-backup-daily-kreknin/run.sh b/scripts/books-vds-backup-daily-kreknin/run.sh new file mode 100644 index 0000000..d894f85 --- /dev/null +++ b/scripts/books-vds-backup-daily-kreknin/run.sh @@ -0,0 +1,216 @@ +#!/bin/bash +# Daily books VDS → kreknin backup. Triggered by /etc/cron.d/books-vds-backup at 06:00 MSK. +# Pattern: mirror VDS scripts/vds-backup-rsync-kreknin/run.sh, adapted for books VDS hybrid stack. +set -euo pipefail + +ENV_FILE=/opt/stacks/backup/.env + +# shellcheck disable=SC1090 +source "$ENV_FILE" + +TODAY=$(date +%Y-%m-%d) +LOG_DIR=/var/log/books-vds-backup +LOG="$LOG_DIR/$TODAY.log" +LOCK=/tmp/books-vds-backup.lock +DUMP_DIR=/tmp/books-vds-backup-dumps/$TODAY +DEST_PATH="$DEST_BASE/$TODAY" +LATEST="$DEST_BASE/latest" + +mkdir -p "$LOG_DIR" "$DUMP_DIR" +exec >> "$LOG" 2>&1 +echo "===== BOOKS-VDS backup start $TODAY $(date -Is) =====" + +# single-instance lock +exec 9>"$LOCK" +if ! flock -n 9; then echo "already running, exiting"; exit 0; fi + +START=$(date +%s) +SSH_OPTS="-i $SSH_KEY -o StrictHostKeyChecking=yes -o ConnectTimeout=30 -o LogLevel=ERROR" +# Note: kreknin host key must be pre-populated in /root/.ssh/known_hosts (CentOS 7 OpenSSH 7.4 has no accept-new option) + +HOST_LABEL=BOOKS-VDS +SOURCE_DISPLAY="89.253.255.133 (books VDS)" + +notify() { + local title="$1" msg="$2" tags="${3:-}" prio="${4:-default}" + curl -sS --max-time 10 -u "$NTFY_USER:$NTFY_PASS" \ + -H "Title: $title" -H "Tags: $tags" -H "Priority: $prio" \ + -d "$msg" "$NTFY_URL/vds-backup" > /dev/null || true +} + +email_send() { + local subj="$1" body="$2" + local tmpf + tmpf=$(mktemp) + # RFC 5322: CRLF headers + Date required by Yandex + { + printf 'From: %s\r\n' "$SMTP_FROM" + printf 'To: %s\r\n' "$OPS_NOTIFY_EMAIL" + printf 'Subject: %s\r\n' "$subj" + printf 'Date: %s\r\n' "$(date -R)" + printf 'MIME-Version: 1.0\r\n' + printf 'Content-Type: text/plain; charset=UTF-8\r\n' + printf '\r\n' + echo "$body" | sed 's/$/\r/' + } > "$tmpf" + # smtps:// (implicit TLS) for Yandex port 465; smtp:// (STARTTLS) would be port 587. + # SMTP_PORT in pass is 465 — use smtps:// scheme. + curl -sS --max-time 30 \ + --ssl-reqd \ + --url "smtps://$SMTP_HOST:$SMTP_PORT" \ + --user "$SMTP_USER:$SMTP_PASS" \ + --mail-from "$SMTP_FROM" \ + --mail-rcpt "$OPS_NOTIFY_EMAIL" \ + --upload-file "$tmpf" || true + rm -f "$tmpf" +} + +fmt_duration() { + local s=$1 + printf "%dm%02ds" $((s/60)) $((s%60)) +} + +on_error() { + local line=$1 rc=$2 + local err="line $line, exit $rc" + local now + now=$(date +%s) + local dur=$((now - ${START:-now})) + local dur_h + dur_h=$(fmt_duration "$dur") + local tail_log + tail_log=$(tail -40 "$LOG" 2>/dev/null || true) + + echo "FATAL: $HOST_LABEL backup FAILED — $err" + + notify "$HOST_LABEL backup FAILED $TODAY" \ + "After $dur_h: $err. See $LOG" \ + red_circle high + + email_send "[$HOST_LABEL] backup FAILED $TODAY" \ +"$HOST_LABEL daily backup FAILED. + +Date: $TODAY +Duration: $dur_h +Error: $err +Source: $SOURCE_DISPLAY +Log: $LOG + +Tail (last 40 lines): +$tail_log" + exit "$rc" +} +trap 'on_error $LINENO $?' ERR + +# === Step 1: DB dumps === +echo "--- DB dumps to $DUMP_DIR ---" + +docker exec books-db mariadb-dump -uroot -p"$BOOKS_DB_ROOT_PASSWORD" \ + --all-databases --single-transaction --quick 2>/dev/null \ + | gzip > "$DUMP_DIR/mariadb.sql.gz" +echo "mariadb: $(stat -c %s "$DUMP_DIR/mariadb.sql.gz") bytes" + +docker exec mongo mongodump --archive --quiet \ + --uri="mongodb://$BOOKS_MONGO_USERNAME:$BOOKS_MONGO_PASSWORD@localhost:27017/?authSource=admin" 2>/dev/null \ + | gzip > "$DUMP_DIR/mongo.archive.gz" +echo "mongo (shared): $(stat -c %s "$DUMP_DIR/mongo.archive.gz") bytes" + +docker exec books-job-scheduler-mongo mongodump --archive --quiet 2>/dev/null \ + | gzip > "$DUMP_DIR/job-scheduler-mongo.archive.gz" +echo "job-scheduler-mongo: $(stat -c %s "$DUMP_DIR/job-scheduler-mongo.archive.gz") bytes" + +# === Step 2: ES snapshot via REST === +echo "--- ES snapshot daily-$TODAY in repo $ES_SNAPSHOT_REPO ---" +ES_SNAP="daily-$TODAY" +SNAP_RESULT=$(docker exec elasticsearch curl -sS -X PUT \ + "http://localhost:9200/_snapshot/$ES_SNAPSHOT_REPO/$ES_SNAP?wait_for_completion=true" \ + -H 'Content-Type: application/json') +echo "ES snapshot result: $SNAP_RESULT" +if ! echo "$SNAP_RESULT" | grep -q '"state":"SUCCESS"'; then + echo "ES snapshot did NOT report SUCCESS state — failing" + exit 3 +fi + +echo "--- prune ES snapshots older than $RETENTION_DAYS ---" +# Pure bash parse — no jq/python dependency +ALL_SNAPS=$(docker exec elasticsearch curl -sS "http://localhost:9200/_snapshot/$ES_SNAPSHOT_REPO/_all" \ + | grep -oE '"snapshot":"daily-[0-9-]+"' \ + | sed -E 's/.*"(daily-[0-9-]+)".*/\1/' \ + | sort) +SNAP_COUNT=$(echo "$ALL_SNAPS" | wc -l) +PRUNE_COUNT=$((SNAP_COUNT - RETENTION_DAYS)) +if [ $PRUNE_COUNT -gt 0 ]; then + echo "$ALL_SNAPS" | head -n "$PRUNE_COUNT" | while read -r snap; do + [ -z "$snap" ] && continue + echo "deleting ES snapshot: $snap" + docker exec elasticsearch curl -sS -X DELETE \ + "http://localhost:9200/_snapshot/$ES_SNAPSHOT_REPO/$snap" || true + echo + done +fi + +# === Step 3: rsync to kreknin === +echo "--- rsync to $DEST_USER@$DEST_HOST:$DEST_PATH ---" +ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "mkdir -p '$DEST_PATH'" + +LINK_DEST_ARG="" +if ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "test -L '$LATEST' || test -d '$LATEST'" 2>/dev/null; then + LINK_DEST_ARG="--link-dest=$LATEST" +fi + +rsync -aHh --info=stats2 --delete $LINK_DEST_ARG \ + -e "ssh $SSH_OPTS" \ + /opt/books \ + /usr/docker/elasticsearch/snapshots \ + /usr/docker/minio/data \ + /usr/docker/traefik/letsencrypt \ + /usr/docker/traefik/data \ + /etc/ssh \ + /etc/hosts \ + /etc/cron.d \ + /root/.ssh \ + "$DUMP_DIR" \ + "$DEST_USER@$DEST_HOST:$DEST_PATH/" || true + +ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "rm -f '$LATEST' && ln -s '$DEST_PATH' '$LATEST'" + +# === Step 4: retention prune on kreknin === +echo "--- prune snapshots older than $RETENTION_DAYS days on kreknin ---" +ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" \ + "cd '$DEST_BASE' && ls -1d 20*-*-* 2>/dev/null | sort | head -n -$RETENTION_DAYS | xargs -r rm -rf" + +# === Step 5: local cleanup === +rm -rf "$DUMP_DIR" + +# === Step 6: report === +END=$(date +%s) +DURATION=$((END - START)) +SIZE=$(ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "du -sh '$DEST_PATH' 2>/dev/null | cut -f1" || echo "?") +SNAPSHOT_COUNT=$(ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "ls -1d $DEST_BASE/20*-*-* 2>/dev/null | wc -l" || echo "?") +DURATION_HUMAN=$(fmt_duration "$DURATION") + +NTFY_BODY="$DURATION_HUMAN, size=$SIZE, snapshots=$SNAPSHOT_COUNT, dest=kreknin:$DEST_PATH" +notify "$HOST_LABEL backup OK $TODAY" "$NTFY_BODY" green_circle + +EMAIL_BODY="$HOST_LABEL daily backup completed successfully. + +Date: $TODAY +Duration: $DURATION_HUMAN +Size: $SIZE +Snapshots: $SNAPSHOT_COUNT +Source: $SOURCE_DISPLAY +Dest: kreknin:$DEST_PATH + +Components: +- /opt/books/{api,job-scheduler,task-runner,ntfy,books-ops-mcp} +- /usr/docker/{minio/data, elasticsearch/snapshots, traefik/{letsencrypt,data}} +- /etc/{ssh,hosts,cron.d} +- /root/.ssh +- DB dumps: books-db (mariadb), mongo (shared), books-job-scheduler-mongo +- ES snapshot: $ES_SNAP + +Log: $LOG" +email_send "[$HOST_LABEL] backup OK $TODAY" "$EMAIL_BODY" + +echo "$HOST_LABEL backup OK: $NTFY_BODY" +echo "===== $HOST_LABEL backup done $(date -Is) ====="