OpeItcLoc03/admin
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

backup(books-vds): daily 06:00 MSK pipeline to kreknin live 🟢

Browse Source 
- scripts/books-vds-backup-daily-kreknin/ — run.sh + .env.example + README
- ES path.repo bootstrap (one-time stack edit, snapshot repo 'kreknin' registered)
- curl smtps://yandex:465 email (CRLF + Date headers, no msmtp dep on CentOS 7)
- StrictHostKeyChecking=yes + pre-populated known_hosts (CentOS 7 no accept-new)
- cron /etc/cron.d/books-vds-backup live, initial sync 4.87 GB in 13m28s, ntfy push sent
- wiki: entities/books-vds.md + sources/books-vds-backup-daily-kreknin-2026-05-25.md created
- wiki: concepts/books-ssh-access → vds-kzntsv-ssh-access (was lying about books on vds-kzntsv)
- .tasks/books-vds-backup-daily-kreknin.md closed 🟢 pending phone-side verify

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
vitya
2026-05-25 09:37:16 +03:00
parent 1eefb4d45a
commit fe41ee4422
10 changed files with 694 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files

48
.tasks/NEXT_SESSION.md
View File

@@ -1,42 +1,50 @@
--- ---
_last_updated_: 2026-05-25T07:50:00+03:00 _last_updated_: 2026-05-25T09:35:00+03:00
session_id: 2026-05-25-close-unify-backup-notifications session_id: 2026-05-25-books-vds-backup-standup
--- ---
# Next session handoff # Next session handoff
## Recent commits ## Recent commits
- `tasks(unify-backup-notifications): close 🟢 — unified push+email across VDS/RUVDS/windows-host` (этот раунд) - `tasks(books-vds-backup-daily-kreknin): close 🟢 — daily 06:00 MSK pipeline live, 4.87GB initial sync` (этот раунд, planned)
- `tasks: rm obsolete NEXT-SESSION-PROMPT.md (leaked creds in history since 2026-05-21)` (этот раунд) - `wiki(books-vds): create entity + source standup + rename misnamed concept books-ssh-access → vds-kzntsv-ssh-access` (этот раунд, planned)
- `scripts(backup-notifications): unify push + email format across VDS/RUVDS/windows-host` (`73ad6dd0`) - `scripts(books-vds-backup-daily-kreknin): run.sh + .env.example + README` (этот раунд, planned)
- `wiki(claude): add session-handoff trigger line` - `tasks: add stateful-split-volume-copy (moved from victor/books)` (`1eefb4d4`)
- `wiki(ingest): RUVDS IIS migration + daily backup pipeline` - `tasks(unify-backup-notifications): close 🟢` (`838f51da`)
## Open треки ## Open треки
| Трек | Готовность | Entry-point | | Трек | Готовность | Entry-point |
|---|---|---| |---|---|---|
| `iis-migration-to-ruvds` 🟡 | soak (kupimknigi + emspb DNS flipped 2026-05-24, оставшиеся 22 hostnames в очереди, tandemmebel.ru excluded) | `.tasks/iis-migration-to-ruvds.md` | | `books-vds-backup-daily-kreknin` 🟢 | closed — **phone-side ntfy + email verify pending user** | `.tasks/books-vds-backup-daily-kreknin.md` |
| `infra-inventory` ⚪ | ready, не начат | `.tasks/infra-inventory.md` | | `stateful-split-volume-copy` ⚪ | not started; **playbook references `books-db` на books VDS (89.253.255.133), не на vds-kzntsv** | `.tasks/stateful-split-volume-copy.md` |
| `books-bookva-user-whitelist-gathering` ⚪ | ready, не начат — long-lead gathering | `.tasks/books-bookva-user-whitelist-gathering.md` | | `iis-migration-to-ruvds` 🟡 | soak (kupimknigi + emspb DNS flipped 2026-05-24, 22 pending) | `.tasks/iis-migration-to-ruvds.md` |
| `books-vds-bookva-bootstrap` 🔵 | blocked on victor/books tenant-split Phase 4 | `.tasks/books-vds-bookva-bootstrap.md` | | `infra-inventory` ⚪ | ready, не начат — **books VDS должен попасть в admin-detailed runbook** | `.tasks/infra-inventory.md` |
| `books-bookva-user-whitelist-gathering` ⚪ | ready, long-lead gathering | `.tasks/books-bookva-user-whitelist-gathering.md` |
| `books-vds-bookva-bootstrap` 🔵 | blocked on tenant-split Phase 4 | `.tasks/books-vds-bookva-bootstrap.md` |
| `books-dns-cutover-bookva` 🔵 | blocked on books-vds-bookva-bootstrap | `.tasks/books-dns-cutover-bookva.md` | | `books-dns-cutover-bookva` 🔵 | blocked on books-vds-bookva-bootstrap | `.tasks/books-dns-cutover-bookva.md` |
## Спроси user'а ## Спроси user'а
- **windows-host deploy сделан?** Если `Get-FileHash C:\ProgramData\backup\run.ps1` matches `scripts/windows-host-fallback-backup-daily/run.ps1` (elevated check) — deploy ок, и сегодня ночью 03:00 MSK пушнёт в едином формате. Если нет — старый формат на одну ночь. - **ntfy push** `BOOKS-VDS backup OK 2026-05-25` (tag green_circle, channel `vds-backup`) — пришёл на phone? Если нет — diagnose curl ntfy creds или phone subscription к каналу.
- **На night-run 2026-05-25 → 26 push'и + emails пришли все 3?** Title pattern `<HOST> backup OK <date>` / subject `[<HOST>] backup OK <date>`. Если что-то отвалилось — `.bak-pre-unify` файлы лежат на каждом хосте для quick rollback. - **Email**: пришёл ли `[BOOKS-VDS] backup OK 2026-05-25` + standalone `[BOOKS-VDS] SMTP fix verify` (тестовый, отправлен после fix'а)? Inbox `<OPS_NOTIFY_EMAIL>` от noreply@snolla.com.
- **DNS swap для 22 hostnames** (`iis-migration-to-ruvds` next-action #3) — продолжаем или ещё soak? - **Migrate SSH-compose стеки под Portainer** (user-promised after backup live): ES, mongo, minio, books-db, traefik, proxy-chain currently SSH-compose at `/usr/docker/<svc>/`. Делаем? Если да — это отдельная таска (`books-vds-stacks-to-portainer`).
- **Plaintext creds в git history** (NEXT-SESSION-PROMPT.md): user сегодня acceptуnул risk без rotation. Если позже передумаем — rotate MSSQL SA + snolla DB user + stayer DB user + VM admin (`Pryakhin9`). - **`stateful-split-volume-copy`** — playbook писан под прошлое assumption «books на vds-kzntsv». Real books VDS = 89.253.255.133, его `books-db` (не `mariadb`) — playbook fits OK, но container names в task-файле должны быть verify'нуты. Continue эту таску в next session?
## Не делать (preemptive guards) ## Не делать (preemptive guards)
- **Не push без grant'а** (project-discipline Rule 4 — session-start = ask-mode). Каждая сессия начинает спрашивать заново. - **Не push без grant'а** (project-discipline Rule 4). Session начинает ask-mode.
- **Не trigger полный backup-run для smoke** (21-81 min wall-clock). Smoke = только notify-channel через `scripts/<slug>/smoke-notify.ps1`. - **Не trust wiki blindly** — `books-ssh-access.md` (теперь `vds-kzntsv-ssh-access.md`) до 2026-05-25 врала что books на vds-kzntsv. Faktcheck через `vds-ops` + SSH probe + IP до построения предположений. Это уже applied в этом раунде; не забыть в будущих сессиях.
- **Не history-rewrite + force-push** для cleanup creds в `NEXT-SESSION-PROMPT.md` без явного user-approve — user уже принял risk на ad-hoc rotation, не на destructive history operations. - **Не trigger полный backup для smoke** на books VDS — 13m+ wall-clock, 4.87 GB initial. Smoke = только notify-channel тест через standalone curl call.
- **Не fold `traefik`/`portainer` управление через SSH compose** на VDS — Portainer-managed per `.wiki/concepts/portainer-stack-management-vds.md`, кроме `traefik`+`portainer` themselves. - **Не touch ES прод стек без user-approve** — ES сейчас не используется (per user), но статус может measure. Если включат — recreate downtime неacceptable без window.
- **Не editit `/usr/docker/<svc>/docker-compose.yml` напрямую** для production stacks — preferred path: migrate под Portainer, потом edit там. Текущий backup-script патчил ES — это был bootstrap, не daily ops.
## Memory updates за сессию ## Memory updates за сессию
- (нет на этом раунде — все правила/паттерны уже зафиксированы в project CLAUDE.md + global skills) - **Учить:** `books VDS ≠ vds-kzntsv`. Wiki была ошибочно misnamed. Real books VDS = 89.253.255.133 (host4g.ru, CentOS 7). Documented в [[../wiki/entities/books-vds]].
- **Учить:** `pass entries one-file-per-host pattern` — consolidate сервис-specific creds в `<host>/full-env` с section'ами (matches `vds-kzntsv/full-env` style). Не плодить `<host>-<service>/full-env`.
- **Учить:** Yandex SMTP — port 465 = `smtps://` (implicit TLS), port 587 = `smtp://` + `--ssl-reqd` (STARTTLS). Не путать; mismatch = 30-sec curl timeout.
- **Учить:** CentOS 7 OpenSSH 7.4 не имеет `accept-new` (added 7.6). Bootstrap step: `ssh-keyscan` host-key в `known_hosts`, потом `StrictHostKeyChecking=yes`.
- **Учить:** docker-compose 1.29 + Docker 26 = `KeyError: 'ContainerConfig'` при recreate. Workaround: `docker rm -f <renamed-stuck>` + `docker-compose up -d`. Bigger fix: upgrade docker-compose to v2 plugin.
- **Reference:** Portainer на books VDS = `https://portainer.kzntsv.site` (НЕ `portainer.vds.kzntsv.site` — тот для infra). PAT saved в `pass show books-vds/full-env BOOKS_PORTAINER_API_KEY`.

4
.tasks/STATUS.md
View File

@@ -1,5 +1,7 @@
# Admin Task Board # Admin Task Board
_Updated: 2026-05-25 (`unify-backup-notifications` 🟢 closed — 3 scripts под единый push+email format committed `73ad6dd0`, VDS+RUVDS deployed+smoke ✓, windows-host self-deploy на user'е через `scripts/windows-host-fallback-backup-daily/deploy.ps1` elevated. Также удалён obsolete `.tasks/NEXT-SESSION-PROMPT.md` (содержал plaintext creds в git history с 2026-05-21; user accept risk без rotation/rewrite). VDS `run.sh` теперь в repo как canonical source. Pre-existing: все 3 daily backups live на kreknin.)_ _Updated: 2026-05-25 (`books-vds-backup-daily-kreknin` 🟢 closed — 4-я daily backup pipeline на kreknin, теперь покрыта `books VDS` 89.253.255.133 (host4g.ru). Pattern mirror VDS-infra (rsync+--link-dest), ES snapshot via REST API (новый repo `kreknin`, потребовал one-time stack edit + recreate ES), email через `curl smtps://yandex:465` без msmtp. Initial sync 4.87 GB in 13m28s, cron 06:00 MSK live. Также fix вики: `concepts/books-ssh-access.md` → `vds-kzntsv-ssh-access.md` (был misnamed — врал что books живёт на vds-kzntsv), `entities/books-vds.md` создан с factsheet'ом. **Phone-side ntfy + email — pending user confirmation на следующем раунде.**)_
## 🟢 [books-vds-backup-daily-kreknin] — closed 2026-05-25 — daily backup books VDS (89.253.255.133, host4g.ru, CentOS 7) → kreknin в 06:00 MSK. DB dumps (mariadb/mongo×2) + ES snapshot via REST → rsync 4.87GB → ntfy/email. ES path.repo bootstrap + snapshot repo `kreknin` registered. См. [books-vds-backup-daily-kreknin.md](books-vds-backup-daily-kreknin.md).
## 🟢 [unify-backup-notifications] — closed 2026-05-25 — единый формат push + email для VDS/RUVDS/windows-host (3 scripts), VDS run.sh импортирован в repo. См. [unify-backup-notifications.md](unify-backup-notifications.md) § Closed. windows-host self-deploy остаётся на user'е (elevated PS). ## 🟢 [unify-backup-notifications] — closed 2026-05-25 — единый формат push + email для VDS/RUVDS/windows-host (3 scripts), VDS run.sh импортирован в repo. См. [unify-backup-notifications.md](unify-backup-notifications.md) § Closed. windows-host self-deploy остаётся на user'е (elevated PS).

61
.tasks/books-vds-backup-daily-kreknin.md Normal file
View File

@@ -0,0 +1,61 @@
# books-vds-backup-daily-kreknin
Ежедневный backup pipeline books VDS (`89.253.255.133`, host4g.ru, CentOS 7) → kreknin Synology (`195.19.90.188:/volume1/NetBackup/books-vds/<date>/`) в 06:00 MSK.
## Goal
Восстановимость books-стека после полного отказа VDS: DB dumps + MinIO blobs + ES snapshot + app configs + traefik state + system config на NAS с 7-day retention.
## Key files
- `scripts/books-vds-backup-daily-kreknin/run.sh` — backup logic; deploy → `/opt/stacks/backup/run.sh` на VDS.
- `scripts/books-vds-backup-daily-kreknin/.env.example` — placeholder; live `.env` built from `pass show`.
- `scripts/books-vds-backup-daily-kreknin/README.md` — install + revert recipe.
- `.wiki/entities/books-vds.md` — host entity page.
- `.wiki/sources/books-vds-backup-daily-kreknin-2026-05-25.md` — standup chronology (создаётся при closure).
## Decisions log
- 2026-05-25: pattern mirror `scripts/vds-backup-rsync-kreknin/run.sh` (VDS-infra). Same `vds-backup` ntfy topic, same SMTP relay (`snolla-smtp` Yandex), same `--link-dest` rsync hardlink pattern, same `RETENTION_DAYS=7`.
- 2026-05-25: **ES snapshot via REST API**, not data-dir rsync. One-time stack edit `/usr/docker/elasticsearch/docker-compose.yml`: added env `path.repo=/snapshots` + bind `./snapshots:/snapshots`. Registered repo `kreknin` (type=fs, location=/snapshots, compress=true).
- 2026-05-25: **`curl --ssl-reqd` SMTP** instead of msmtp (CentOS 7 EOL repo issue). Yandex 587 STARTTLS works через native curl без msmtprc setup.
- 2026-05-25: `StrictHostKeyChecking=yes` (NOT `accept-new`) — CentOS 7 OpenSSH 7.4 не поддерживает `accept-new`. kreknin host key pre-populated в `/root/.ssh/known_hosts` на VDS bootstrap.
- 2026-05-25: `books-job-scheduler-mongo` — **без auth** (нет `MONGO_INITDB_ROOT_*` env). `mongodump --archive` без `--uri` достаточен. shared `mongo` — root auth (uri credential).
- 2026-05-25: **Cron time 06:00** MSK — после VDS-infra (05:00) + windows-host (05:30), даёт break перед US-EU business start.
- 2026-05-25: `BOOKS-VDS` host label в ntfy title — отличать от обычного `VDS` (infra) на shared `vds-backup` topic.
- 2026-05-25: Consolidated `books-vds-portainer/full-env` → `books-vds/full-env` (one-file-per-host pattern, parity с vds-kzntsv).
## Open questions
- [ ] (resolved 2026-05-25) `path.repo` на ES — required one-time stack edit. Done, recreated container, snapshot test SUCCESS.
- [ ] Migrate SSH-compose stacks (ES/mongo/minio/books-db/traefik) под Portainer — **отдельная задача после backup pipeline live**.
- [ ] kreknin space check — current free 5.7T (per `kreknin-synology.md` 2026-05-19); per-day footprint TBD после initial sync. Quarterly disk audit на NAS.
## Completed steps
- [x] Phase 0: ES path.repo bootstrap + snapshot repo `kreknin` registered + test snap SUCCESS
- [x] Phase 1: написал `run.sh` + `.env.example` + `README.md` локально
- [x] Phase 2: deployed `/opt/stacks/backup/{run.sh,.env}`, chmod 600 .env / 755 run.sh
- [x] Phase 3: generated `/opt/stacks/backup/kreknin-key` ed25519 на books VDS
- [x] Phase 4: pubkey deployed на kreknin `~vitya/.ssh/authorized_keys`, dest `/volume1/NetBackup/books-vds/` created
- [x] Phase 5: kreknin host key pre-populated `/root/.ssh/known_hosts`, SSH path verified
- [x] Phase 6: smoke run (initial sync) — 4.87GB in 13m28s, ntfy push sent, **email send pending fix → fixed → re-tested OK**
- [x] Phase 6a: SMTP fix — `smtp://` + `--ssl-reqd` дала 30-сек timeout (curl попытался STARTTLS на implicit-TLS-only порту 465). Fix `smtps://` schema + CRLF headers + Date header. Re-tested standalone — SMTP send OK.
- [x] Phase 7: installed `/etc/cron.d/books-vds-backup` (`0 6 * * * root /opt/stacks/backup/run.sh`); crond active
- [x] Phase 8: wiki — `entities/books-vds.md` created; `concepts/books-ssh-access.md` renamed → `vds-kzntsv-ssh-access.md` (was lying about books); `sources/books-vds-backup-daily-kreknin-2026-05-25.md` created; `index.md` updated for all three
- [ ] Phase 9: commit (no push without grant)
- [ ] **PENDING USER VERIFY:** ntfy push `BOOKS-VDS backup OK 2026-05-25` received on phone? Email `[BOOKS-VDS] backup OK 2026-05-25` + standalone fix-test message in inbox?
## Notes
- root password `Pryakhin9~` — сохранён в `pass show books-vds/full-env BOOKS_VDS_ROOT_PASS`. SSH key `id_ed25519_books_ops` уже был authorized (pre-existing bootstrap), password держим как fallback.
- Pre-existing `id_ed25519_books_ops` на workstation предполагает что прошлый bootstrap забыт; не повторяем generate.
- ES в production books currently **не используется** (per user statement) — recreate-downtime 30с не impact'нул.
- `vds-kzntsv` ≠ `books-vds`. Wiki [[../concepts/books-ssh-access]] до 2026-05-25 ошибочно описывала vds-kzntsv как books shared VDS — fix в Phase 8.
## Cross-refs
- [[../wiki/entities/books-vds]] — host entity.
- [[../wiki/entities/kreknin-synology]] — backup target.
- [[../scripts/vds-backup-rsync-kreknin/run.sh]] — pattern source (VDS-infra backup).
- [[../scripts/ruvds-backup-daily-kreknin/README.md]] — pattern source (RUVDS dual-channel notify).

27
.wiki/concepts/books-ssh-access.md → .wiki/concepts/vds-kzntsv-ssh-access.md
View File

@@ -1,15 +1,17 @@
--- ---
title: books shared VDS — SSH access audit + retained keys title: vds-kzntsv — SSH access audit + retained keys
status: live status: live
tags: [vds, ssh, audit, books, tenant-split, ops] tags: [vds, ssh, audit, infra, ops]
related: [[vds-kzntsv]], [[../sources/vds-kzntsv-bootstrap-2026-05-20]] related: [[../entities/vds-kzntsv]], [[../sources/vds-kzntsv-bootstrap-2026-05-20]]
--- ---
# books shared VDS — SSH access (post-audit 2026-05-24) # vds-kzntsv — SSH access (post-audit 2026-05-24)
Snapshot SSH-доступа к `vds-kzntsv` (89.253.255.94 / vds.kzntsv.site), на котором сейчас живёт shared `books`-инсталляция (Slovo + Bookva в одном compose-стеке, до physical split Фазы 4 в [`tenant-split`](https://git.kzntsv.site/victor/books/src/branch/master/.wiki/concepts/tenant-split.md)). Snapshot SSH-доступа к [[../entities/vds-kzntsv]] (`89.253.255.94 / vds.kzntsv.site`) — наш infra-VDS (gitea, registry, ntfy, mssql, minio, owncloud, board-viewer).
Создан 2026-05-24 в результате таски `[books-ssh-audit-shared-vds]` 🟢 (см. spec в Фазе 3 §SSH-аудит). **Корректировка 2026-05-25:** до этой даты документ ошибочно утверждал что на vds-kzntsv живёт shared books-инсталляция. **Это неверно** — books-стек на отдельном клиентском VDS `89.253.255.133` (см. [[../entities/books-vds]]). vds-kzntsv хостит только нашу infra. Аудит ниже относится к vds-kzntsv SSH (наш infra-host), не к books VDS.
Создан 2026-05-24 в результате таски `[books-ssh-audit-shared-vds]` 🟢 — задача была названа в предположении что books на vds-kzntsv; reality по сути проверила infra SSH (что и должно было быть).
## Retained keys ## Retained keys
@@ -52,16 +54,15 @@ Brute-force protection adequate; нет required action.
`sudo journalctl -u ssh -u sshd --since="7 days ago" | grep Accepted` показал **только** `vitya from 94.19.247.14` (мой home public IP). Никаких других source-IP — clean. `sudo journalctl -u ssh -u sshd --since="7 days ago" | grep Accepted` показал **только** `vitya from 94.19.247.14` (мой home public IP). Никаких других source-IP — clean.
## Tenant-split implications (Phase 3 acceptance) ## Books tenant-split — implications (исправлено 2026-05-25)
Spec `tenant-split.md` §«Двухуровневая изоляция пользователей → Infra-level»: Spec `tenant-split.md` §«Двухуровневая изоляция пользователей → Infra-level» и аудит «не давать аналитикам Bookva SSH к VDS» — **должны были применяться к books VDS, не к vds-kzntsv**. Реальный books VDS = `89.253.255.133` ([[../entities/books-vds]]) — отдельный клиентский сервер. SSH-аудит на books VDS — отдельная follow-up задача.
> на промежуточной стадии не давать аналитикам Bookva SSH к VDS, только app-level. SSH-аудит до cutover'а.
**Verified clean** as of 2026-05-24 — никаких аналитических SSH-ключей не было выдано. Single-developer setup; analyst access всегда был app-level only. Pre-cutover sigh of relief. vds-kzntsv (этот документ) — infra-VDS, не имеет отношения к books tenants. После Phase 4 books tenant-split:
- books VDS (`89.253.255.133`) — текущий shared books → может остаться за Slovo (или Bookva, как пользователь решит).
- vds-bookva-new — отдельный VDS у учредителя Bookva. См. `.tasks/books-vds-bookva-bootstrap.md` + `books-dns-cutover-bookva.md`.
После Phase 4 (physical split на 2 VDS): vds-kzntsv остаётся infra-host независимо от tenant-split исхода.
- vds-kzntsv (Slovo) — продолжит хостить остальную нашу инфру (gitea, registry, ntfy, mssql, minio, owncloud, board-viewer) + books-slovo стек. SSH = vitya only.
- vds-bookva-new (Bookva) — отдельная box, отдельный SSH. Заводить core-dev key (vitya) + опционально formally-named Bookva admin.
## How to add a new key (process) ## How to add a new key (process)

109
.wiki/entities/books-vds.md Normal file
View File

@@ -0,0 +1,109 @@
---
title: Books VDS (client server — 89.253.255.133)
type: entity
tags: [hardware, vds, books, host4g, backup, portainer]
related: [[kreknin-synology]], [[vds-kzntsv]]
updated: 2026-05-25
---
# Books VDS
Клиентский VDS (отдельный от инфра-VDS [[vds-kzntsv]]). Хостит production books app stack (api/web/scheduler/task-runner) + shared infra (mongo, minio, elasticsearch, traefik, portainer).
**Это НЕ vds-kzntsv.** Books-стек живёт здесь, не на vds.kzntsv.site. Domain `books.kzntsv.site` (и `bookva.kzntsv.site`) разрешаются в этот IP.
## Доступ
- **Public IP:** 89.253.255.133
- **Provider:** host4g.ru (DNS вендорский `vps-21075162-277731.host4g.ru`)
- **OS:** CentOS 7, kernel 3.10.0-1160.36.2.el7.x86_64 (древний, EOL)
- **SSH:** `root@89.253.255.133:22` — only via ed25519 key (`~/.ssh/id_ed25519_books_ops` на workstation)
- **Sudo:** root, passwordless (мы под root напрямую)
- **Portainer:** `https://portainer.kzntsv.site` (отдельный Portainer, не путать с `portainer.vds.kzntsv.site` для [[vds-kzntsv]])
Все creds — в `pass show books-vds/full-env`.
## Disk
```
/dev/vda1 122G 59G used (49%) 63G free
```
Single root partition, нет отдельной `/data`. Docker hub data под `/var/lib/docker/`.
## Стек (2026-05-25 inventory)
### Portainer-managed stacks (endpoint 1)
| Stack | Container(s) | Purpose |
|---|---|---|
| `books-api` | books-api | books API server (Nitro, port 3021) |
| `books-web` | books-web | books-Nuxt front-end |
| `books-job-scheduler` | books-job-scheduler + books-job-scheduler-mongo | agenda jobs (cron + on-demand) |
| `books-ntfy` | books-ntfy | local ntfy (separate from shared `ntfy.vds.kzntsv.site`) |
| `books-ops-mcp` | books-ops-mcp + books-ops-mcp-bootstrap-1 + books-docker-proxy + books-docker-proxy-ro | books ops-mcp endpoint |
| `chrome` | chrome | Puppeteer browser for PDFs / scraping |
### SSH-managed (legacy compose at `/usr/docker/<svc>/`)
| Compose dir | Containers | Bind data path |
|---|---|---|
| `/usr/docker/books-db/` | books-db | `./data` → `/var/lib/mysql` (MariaDB 10.6.4) |
| `/usr/docker/mongo/` | mongo | `./data/{db,configdb}` |
| `/usr/docker/minio/` | minio | `./data` (S3 blobs) |
| `/usr/docker/elasticsearch/` | elasticsearch | `./data` + `./snapshots` (added 2026-05-25) |
| `/usr/docker/traefik/` | traefik | `./letsencrypt` (acme.json) + `./data/traefik.yml` |
| `/usr/docker/portainer/` | portainer | named volume `portainer_data` |
| `/usr/docker/proxy-chain/` | proxy-chain | upstream HTTP-proxy chain |
**TODO**: мигрировать максимум SSH-managed контейнеров под Portainer (отдельная таска после backup-pipeline live). Cause: разделённое управление обламывает CI (Portainer-only deploy.yml) и debugging.
### Build helpers (transient)
- `buildx_buildkit_builder-*` × 7 — buildkit builders, oneshot за `docker buildx`. Не data.
## App-config paths
```
/opt/books/api/{config,data,upload}
/opt/books/job-scheduler/{config,mongo/{db,configdb}}
/opt/books/task-runner/
/opt/books/ntfy/ # local ntfy data
/opt/books/books-ops-mcp/
```
`config/default.json` overrides image-baked config (MySQL/ES/S3 endpoints).
## Backup
Daily 06:00 MSK → kreknin. См. [[../sources/books-vds-backup-daily-kreknin-2026-05-25]]. Pipeline:
- DB dumps: books-db (mariadb), mongo (shared), books-job-scheduler-mongo
- ES snapshot via REST API (file repo `kreknin`)
- rsync bind paths + dumps + `/opt/books/` + `/etc/{ssh,hosts,cron.d}` + `/root/.ssh` → `vitya@kreknin:/volume1/NetBackup/books-vds/<date>/`
- ntfy `BOOKS-VDS backup OK <date>` + email
- Retention 7 daily snapshots
Live since 2026-05-25.
## DNS
- `books.kzntsv.site` → 89.253.255.133 (`books-api` host route via local traefik)
- `bookva.kzntsv.site` → 89.253.255.133 (`books-web` host route)
- `elasticsearch.kzntsv.site` → 89.253.255.133 (ES via traefik basicAuth)
- `portainer.kzntsv.site` → 89.253.255.133 (Portainer UI)
- `edge.kzntsv.site` → 89.253.255.133 (Portainer edge agent)
Auth-зоны DNS — в reg.ru под `kzntsv.site`.
## Cross-refs
- [[kreknin-synology]] — backup target.
- [[vds-kzntsv]] — наш инфра-VDS (отдельный, gitea/registry/etc.) — НЕ хост books'а.
- [[../concepts/vds-kzntsv-ssh-access]] — SSH-аудит инфра-VDS (был misnamed как `books-ssh-access` до 2026-05-25; ошибочно claimed vds-kzntsv hosts books).
- [[../sources/books-vds-backup-daily-kreknin-2026-05-25]] — backup standup chronology.
## Каking-of-history notes
- Discovered 2026-05-25 во время backup-таски. Wiki [[../concepts/books-ssh-access]] до этого ошибочно говорила что books на vds-kzntsv (89.253.255.94). User: «books VDS - это совсем другой сервер, клиентский». Inventory + IP probe confirmed.
- Workstation SSH key `id_ed25519_books_ops` уже был задеплоен под root до этой сессии (видимо, прошлый bootstrap забыт-без-документации).
- Hybrid stack-management (Portainer + SSH-compose) — pre-existing, не наш design. Migrate to Portainer-only — follow-up.

4
.wiki/index.md
View File

@@ -8,6 +8,7 @@ Catalog of all wiki pages. One line per page, organized by type. Updated on ever
## Entities ## Entities
- [books-vds](entities/books-vds.md) — Books VDS (89.253.255.133 — host4g.ru, client server hosting books app + shared mongo/minio/elasticsearch)
- [dead-synology-diskstation](entities/dead-synology-diskstation.md) — мёртвая Synology DiskStation (source NAS) - [dead-synology-diskstation](entities/dead-synology-diskstation.md) — мёртвая Synology DiskStation (source NAS)
- [kreknin-synology](entities/kreknin-synology.md) — Kreknin Synology (backup target + DDNS) - [kreknin-synology](entities/kreknin-synology.md) — Kreknin Synology (backup target + DDNS)
- [openwrt-router](entities/openwrt-router.md) — OpenWRT Router (192.168.1.1) - [openwrt-router](entities/openwrt-router.md) — OpenWRT Router (192.168.1.1)
@@ -19,7 +20,7 @@ Catalog of all wiki pages. One line per page, organized by type. Updated on ever
## Concepts ## Concepts
- [admin-infra-project](concepts/admin-infra-project.md) — design + migration plan для OpeItcLoc03/admin (canonical) - [admin-infra-project](concepts/admin-infra-project.md) — design + migration plan для OpeItcLoc03/admin (canonical)
- [books-ssh-access](concepts/books-ssh-access.md) — SSH access audit на shared VDS (books Slovo+Bookva), retained keys + add/revoke processes - [vds-kzntsv-ssh-access](concepts/vds-kzntsv-ssh-access.md) — SSH access audit на vds-kzntsv (infra VDS), retained keys + add/revoke processes (was misnamed `books-ssh-access` до 2026-05-25 — content всегда был про vds-kzntsv)
- [compose-bcrypt-escape-trap](concepts/compose-bcrypt-escape-trap.md) — Docker Compose ест `$` в bcrypt hashes - [compose-bcrypt-escape-trap](concepts/compose-bcrypt-escape-trap.md) — Docker Compose ест `$` в bcrypt hashes
- [db-tls-self-signed-via-traefik-raw-tcp](concepts/db-tls-self-signed-via-traefik-raw-tcp.md) — DB TLS через traefik raw TCP с self-signed certs - [db-tls-self-signed-via-traefik-raw-tcp](concepts/db-tls-self-signed-via-traefik-raw-tcp.md) — DB TLS через traefik raw TCP с self-signed certs
- [docker-host-loopback-detect](concepts/docker-host-loopback-detect.md) — Docker host-loopback detection — как доказать что host.docker.internal не петля - [docker-host-loopback-detect](concepts/docker-host-loopback-detect.md) — Docker host-loopback detection — как доказать что host.docker.internal не петля
@@ -53,5 +54,6 @@ Catalog of all wiki pages. One line per page, organized by type. Updated on ever
- [iis-host-migration-2026-05-19](sources/iis-host-migration-2026-05-19.md) — IIS Host Migration Session 2026-05-19 chronology - [iis-host-migration-2026-05-19](sources/iis-host-migration-2026-05-19.md) — IIS Host Migration Session 2026-05-19 chronology
- [iis-migration-to-ruvds-2026-05-23](sources/iis-migration-to-ruvds-2026-05-23.md) — IIS migration to RUVDS — session 2026-05-23/24 (SSH/scp pivot, 25 SNI bindings, partial DNS cutover) - [iis-migration-to-ruvds-2026-05-23](sources/iis-migration-to-ruvds-2026-05-23.md) — IIS migration to RUVDS — session 2026-05-23/24 (SSH/scp pivot, 25 SNI bindings, partial DNS cutover)
- [nas-recovery-session-2026-05-18](sources/nas-recovery-session-2026-05-18.md) — NAS Recovery Session 2026-05-18/19 chronology - [nas-recovery-session-2026-05-18](sources/nas-recovery-session-2026-05-18.md) — NAS Recovery Session 2026-05-18/19 chronology
- [books-vds-backup-daily-kreknin-2026-05-25](sources/books-vds-backup-daily-kreknin-2026-05-25.md) — books VDS daily backup → kreknin pipeline standup 2026-05-25 (ES file-snapshot repo + DB dumps + curl SMTP)
- [ruvds-backup-daily-kreknin-2026-05-24](sources/ruvds-backup-daily-kreknin-2026-05-24.md) — RUVDS daily backup → kreknin pipeline standup 2026-05-24 (rclone+SFTP, SYSTEM task) - [ruvds-backup-daily-kreknin-2026-05-24](sources/ruvds-backup-daily-kreknin-2026-05-24.md) — RUVDS daily backup → kreknin pipeline standup 2026-05-24 (rclone+SFTP, SYSTEM task)
- [vds-kzntsv-bootstrap-2026-05-20](sources/vds-kzntsv-bootstrap-2026-05-20.md) — VDS bootstrap session 2026-05-20 chronology - [vds-kzntsv-bootstrap-2026-05-20](sources/vds-kzntsv-bootstrap-2026-05-20.md) — VDS bootstrap session 2026-05-20 chronology

111
.wiki/sources/books-vds-backup-daily-kreknin-2026-05-25.md Normal file
View File

@@ -0,0 +1,111 @@
---
title: books VDS daily backup → kreknin — pipeline standup 2026-05-25
type: source
tags: [backup, books-vds, kreknin, rsync, elasticsearch-snapshot, curl-smtp, cron, ntfy]
ingested: 2026-05-25
raw_path: ../../.tasks/books-vds-backup-daily-kreknin.md
updated: 2026-05-25
---
# books VDS daily backup → kreknin
Поднят ежедневный backup pipeline с [[../entities/books-vds]] (`89.253.255.133` / host4g.ru / CentOS 7) → [[../entities/kreknin-synology]] (`195.19.90.188`) в **06:00 MSK**. Параллель к [[vds-kzntsv-bootstrap-2026-05-20]] §backup и [[ruvds-backup-daily-kreknin-2026-05-24]] (pattern mirror), но source = CentOS 7 с hybrid SSH-compose + Portainer стек.
Источник истины — `.tasks/books-vds-backup-daily-kreknin.md` и `scripts/books-vds-backup-daily-kreknin/`.
## Background — почему отдельно
Прошлые сессии (и `[[../concepts/vds-kzntsv-ssh-access]]` до 2026-05-25) ошибочно утверждали что books-стек живёт на [[../entities/vds-kzntsv]]. **Это не так:** books на отдельном клиентском VDS `89.253.255.133`. User указал на ошибку явно: «books VDS - это совсем другой сервер, клиентский». См. [[../entities/books-vds]] для актуального factsheet.
## Scope
| Path on books VDS | Reason | Approx size |
|---|---|---|
| `/opt/books/{api,job-scheduler,task-runner,ntfy,books-ops-mcp}` | App configs (`config/default.json` overrides) | < 100 MB |
| `/usr/docker/minio/data` | MinIO blobs (S3) — основная масса | ~4.5 GB |
| `/usr/docker/elasticsearch/snapshots` | ES snapshot files (repo `kreknin`) | < 1 MB (read_me index only) |
| `/usr/docker/traefik/{letsencrypt,data}` | acme.json + traefik.yml | < 5 MB |
| `/etc/{ssh,hosts,cron.d}` | system config | < 50 KB |
| `/root/.ssh` | root SSH state | < 5 KB |
| `mariadb.sql.gz` (live dump) | books-db full dump | ~270 MB |
| `mongo.archive.gz` (live dump) | shared mongo (root auth) | ~170 KB |
| `job-scheduler-mongo.archive.gz` (live dump) | scheduler agenda (no auth) | ~3 MB |
Total после initial sync: **4.87 GB** (`du -sh` reported 4.8 GB после rsync).
Не бэкапятся: `/var/lib/docker/`, `/var/log/`, transient buildx volumes.
## Decisions log
- **Tool = bash + rsync + curl** (no rclone, no msmtp). Linux native, no extra installs. Pattern mirror VDS-infra `run.sh`.
- **ES snapshot via REST API** (filesystem repo `kreknin`), не data-dir rsync. Required one-time stack edit `/usr/docker/elasticsearch/docker-compose.yml`: добавлены env `path.repo=/snapshots` + bind `./snapshots:/snapshots`, ES recreated. Снимки атомарны и Lucene-consistent.
- **Email via `curl --ssl-reqd --url smtps://...:465`** (implicit TLS), не msmtp. CentOS 7 + EOL repos = msmtp install painful; curl native + Yandex SMTP works. Initial attempt с `smtp://...:465` + `--ssl-reqd` дала 30-sec timeout (curl попытался STARTTLS на TLS-only-from-start порту) — исправлено переходом на `smtps://`.
- **RFC 5322 CRLF headers + Date + MIME-Version + Content-Type** в email-body. Yandex отвергает без Date. Initial body без них принимался при STARTTLS теста, но for production safety — full headers.
- **`StrictHostKeyChecking=yes`** (НЕ `accept-new`) — CentOS 7 OpenSSH 7.4 не поддерживает `accept-new` (added in 7.6). kreknin host key pre-populated в `/root/.ssh/known_hosts` на bootstrap step.
- **Retention 7 daily snapshots** (matches VDS + RUVDS pattern). ES snapshot pruning отдельно — via REST DELETE.
- **`ntfy.vds.kzntsv.site/vds-backup` topic** (shared канал с VDS + RUVDS + windows-host). Title `BOOKS-VDS backup OK <date>` отличает от других хостов на phone-side.
- **books-job-scheduler-mongo без auth** — `MONGO_INITDB_ROOT_*` env пустой, `mongodump --archive` без `--uri` достаточен. Shared `mongo` — root auth (urircreds).
- **Cron 06:00 MSK** — после VDS-infra (05:00), RUVDS (04:30), windows-host (05:30). Break перед US-EU business start.
- **`books-vds-portainer/full-env` consolidated** → `books-vds/full-env` (parity с `vds-kzntsv/full-env` one-file-per-host pattern).
## Хронология standup'а (2026-05-25)
- **08:30:** user-указание начать `stateful-split-volume-copy` под supervision. Probe `vds-ops` MCP показал что books-db нет на vds-kzntsv (только generic mariadb/mongo/minio). Wiki `[[../concepts/vds-kzntsv-ssh-access]]` (тогда `books-ssh-access.md`) врала что books живёт на vds-kzntsv.
- **08:45:** user: «89.253.255.133 - books VDS, клиентский. Сделаем backup на kreknin». Pivot фокус.
- **08:48:** SSH probe `id_ed25519_books_ops` → ready, root access OK. Inventory: hybrid Portainer + SSH-compose, hosting books-api/web/scheduler/task-runner/db + shared infra.
- **08:55:** Portainer API token request — user создал `claude-code-automation` PAT в Portainer UI (`portainer.kzntsv.site`). Saved в pass `books-vds-portainer/full-env`. API tested OK. But `elasticsearch`/`mongo`/`minio`/`books-db`/`traefik` нет среди Portainer-managed stacks — they're SSH-compose. User: «после backup'а мигрируем максимум под Portainer».
- **08:58:** ES probe — `path.repo` не настроен, snapshot API не работает без edit'а compose.
- **09:00:** User approval на ES stack edit «сейчас, ES никто не использует». Sed edit `docker-compose.yml` + recreate. docker-compose 1.29 → docker 26 incompat (`KeyError: 'ContainerConfig'`) → manual `docker rm` renamed-stuck → `docker-compose up -d` clean → ES ready in 30s. Snapshot repo `kreknin` registered + test snap SUCCESS → deleted.
- **09:14:** Wrote `scripts/books-vds-backup-daily-kreknin/{run.sh,.env.example,README.md}` локально. Built `.env` from pass entries (books-vds + vds-kzntsv NTFY + snolla-smtp SMTP). Deployed `/opt/stacks/backup/{run.sh,.env}` на VDS via scp. Generated `kreknin-key` ed25519 на VDS. Authorized pubkey on kreknin via workstation `id_ed25519_kreknin`. Pre-populated kreknin host key в `/root/.ssh/known_hosts` (CentOS 7 no accept-new).
- **09:15:** Smoke run start. DB dumps OK (mariadb 270MB, mongo 170KB, scheduler-mongo 2.8MB). ES snapshot SUCCESS. rsync started.
- **09:29:** rsync done, 4.87 GB in **13m28s** (7.6 MB/s). ntfy push отправлен. Email **timeout 30s** — bug в email_send.
- **09:30:** Diagnose. SMTP_PORT=465 + `smtp://` + `--ssl-reqd` = curl попытался STARTTLS на implicit-TLS-only порту. Fix: `smtps://` schema. Plus CRLF headers + Date header. Re-tested: SMTP send OK.
- **09:32:** Updated run.sh re-deployed. Cron installed `/etc/cron.d/books-vds-backup` (0 6 * * * root /opt/stacks/backup/run.sh). crond active.
## Pipeline state
- **Live:** cron `books-vds-backup`, daily 06:00 MSK, root user.
- **Verified:** initial sync 4.87 GB end-to-end + ntfy push отправлен.
- **Phone-side ntfy verify:** **pending user confirmation** (был ли push `BOOKS-VDS backup OK 2026-05-25` на ntfy app в общем канале `vds-backup`).
- **Email verify:** sent после fix (см. 09:32 fix). User должен подтвердить получение `[BOOKS-VDS] backup OK <date>` или standalone fix-test message в inbox.
- **Day-2 verify:** automatic at 06:00 MSK 2026-05-26 — first cron-trigger.
- **Retention prune verify:** automatic at day-8 (когда snapshot count > 7).
## Bugs fixed during smoke
- **SMTP timeout 30s** — `smtp://...:465` + `--ssl-reqd` = curl STARTTLS на implicit-TLS-only порту. Server ждёт TLS handshake, curl ждёт plain EHLO, deadlock. Fix: `smtps://` scheme.
- **email RFC 5322** — initial body had no Date/MIME headers, only From/To/Subject. Yandex отвергает без Date. Fix: добавлены Date, MIME-Version, Content-Type CRLF headers.
- **CentOS 7 OpenSSH 7.4 no accept-new** — `StrictHostKeyChecking=accept-new` parse-error. Fix: `StrictHostKeyChecking=yes` + pre-populated known_hosts.
- **docker-compose 1.29 ↔ docker 26 KeyError ContainerConfig** — recreate failed mid-way, container renamed-stuck. Manual `docker rm -f` + clean `up -d` rescued.
## Open follow-ups (не блокер)
- **Migrate SSH-compose стеки под Portainer** (отдельная таска, user-requested после backup live). Targets: elasticsearch, mongo, minio, books-db, traefik, proxy-chain.
- **SSH-аудит books VDS** — отдельный аудит по pattern [[vds-kzntsv-ssh-access]]. На сейчас знаем только root key `id_ed25519_books_ops` под root — других пользователей не enum'ил. Pre-existing keys unknown.
- **kreknin space audit** — quarterly check `du -sh /volume1/NetBackup/*` чтобы не упереться в 5.7T limit.
- **wd40 backup**: рассмотреть второй backup target (B2 / Glacier) для resilience — см. [[../concepts/future-resilient-architecture-goals]].
## Атомарный revert
```bash
# На books VDS:
ssh root@89.253.255.133 'rm /etc/cron.d/books-vds-backup; rm -rf /opt/stacks/backup /var/log/books-vds-backup'
# Revert ES path.repo edit:
ssh root@89.253.255.133 'cd /usr/docker/elasticsearch && \
cp docker-compose.yml.bak-pre-snapshots-2026-05-25 docker-compose.yml && \
docker rm -f elasticsearch && docker-compose up -d'
# На kreknin (через SSH):
ssh vitya@195.19.90.188 'rm -rf /volume1/NetBackup/books-vds'
# Remove the books-vds-backup-20260525 pubkey line from authorized_keys.
```
## Cross-refs
- Source host: [[../entities/books-vds]]
- Backup target: [[../entities/kreknin-synology]]
- Pattern parent: [[vds-kzntsv-bootstrap-2026-05-20]] §backup (msmtp + rsync --link-dest)
- Sibling pipeline: [[ruvds-backup-daily-kreknin-2026-05-24]] (rclone+SFTP)
- Notification: общий ntfy topic `vds-backup` на `ntfy.vds.kzntsv.site`.
- Misnaming legacy: [[../concepts/vds-kzntsv-ssh-access]] (бывший `books-ssh-access` до 2026-05-25).

34
scripts/books-vds-backup-daily-kreknin/.env.example Normal file
View File

@@ -0,0 +1,34 @@
# /opt/stacks/backup/.env on books VDS — sourced by run.sh
# Live values are NOT in git; sourced from pass entries:
# pass show books-vds/full-env → BOOKS_DB_* + BOOKS_MONGO_*
# pass show vds-kzntsv/full-env → NTFY_*
# pass show snolla-smtp/full-env → SMTP_* + OPS_NOTIFY_EMAIL
# pass show kreknin/full-env → DEST_USER (vitya), DEST_HOST (195.19.90.188)
# === Backup destination ===
DEST_USER=vitya
DEST_HOST=195.19.90.188
DEST_BASE=/volume1/NetBackup/books-vds
RETENTION_DAYS=7
SSH_KEY=/opt/stacks/backup/kreknin-key
# === DB creds (books VDS containers) ===
BOOKS_DB_ROOT_PASSWORD=...
BOOKS_MONGO_USERNAME=root
BOOKS_MONGO_PASSWORD=...
# === Elasticsearch snapshot repo (file-based, registered 2026-05-25) ===
ES_SNAPSHOT_REPO=kreknin
# === NTFY (shared channel — vds-backup topic; ntfy.vds.kzntsv.site) ===
NTFY_URL=https://ntfy.vds.kzntsv.site
NTFY_USER=...
NTFY_PASS=...
# === SMTP (Yandex relay, noreply@snolla.com) ===
SMTP_HOST=smtp.yandex.ru
SMTP_PORT=587
SMTP_USER=noreply@snolla.com
SMTP_PASS=...
SMTP_FROM=noreply@snolla.com
OPS_NOTIFY_EMAIL=...

115
scripts/books-vds-backup-daily-kreknin/README.md Normal file
View File

@@ -0,0 +1,115 @@
# books-vds-backup-daily-kreknin
Daily backup pipeline: **books VDS** (`89.253.255.133` / host4g.ru CentOS 7) → **kreknin Synology** (`195.19.90.188:/volume1/NetBackup/books-vds/<date>/`) via SSH + rsync.
Schedule: `0 6 * * *` Europe/Moscow (after VDS-infra at 05:00, after windows-host at 05:30). Pattern mirrors `scripts/vds-backup-rsync-kreknin/run.sh`.
## What's backed up
| Component | Method | Source |
|---|---|---|
| MariaDB (books-db) | `mariadb-dump --all-databases --single-transaction` | container `books-db` |
| Mongo (shared) | `mongodump --archive` w/ root auth | container `mongo` |
| Mongo (job-scheduler) | `mongodump --archive` no-auth | container `books-job-scheduler-mongo` |
| Elasticsearch | REST snapshot API → fs repo `kreknin` → rsync | container `elasticsearch` (path.repo=/snapshots) |
| MinIO blobs | rsync direct (immutable content-addressed) | `/usr/docker/minio/data` |
| App configs | rsync | `/opt/books/{api,job-scheduler,task-runner,ntfy,books-ops-mcp}` |
| Traefik state | rsync | `/usr/docker/traefik/{letsencrypt,data}` (acme.json + config) |
| System config | rsync | `/etc/{ssh,hosts,cron.d}`, `/root/.ssh` |
Retention: **7 daily snapshots** on kreknin (+ on ES repo). Older — pruned at end of each run.
## Notifications
Dual-channel after every run:
- **ntfy** → topic `vds-backup` on `ntfy.vds.kzntsv.site` (shared с VDS + RUVDS + windows-host backups).
- Success: title `BOOKS-VDS backup OK <date>`, tag `green_circle`.
- Fail: title `BOOKS-VDS backup FAILED <date>`, tag `red_circle`, priority `high`.
- **Email** via Yandex SMTP (`noreply@snolla.com` → `OPS_NOTIFY_EMAIL`).
- Sent via `curl --ssl-reqd --url smtp://...:587` (no msmtp dep on CentOS 7).
- Subject: `[BOOKS-VDS] backup OK <date>` / `[BOOKS-VDS] backup FAILED <date>`.
If notify-channels fail — backup itself does NOT fail (`|| true`).
## Files
- `run.sh` — backup logic. Synced to `/opt/stacks/backup/run.sh` on books VDS.
- `.env.example` — placeholder. Live `.env` built from `pass show` entries during install.
## Initial install (rebuild from scratch)
```bash
# From workstation, with id_ed25519_books_ops authorized:
ssh root@89.253.255.133 'mkdir -p /opt/stacks/backup'
scp scripts/books-vds-backup-daily-kreknin/run.sh root@89.253.255.133:/opt/stacks/backup/
# Build .env locally from pass entries:
{
echo "DEST_USER=vitya"
echo "DEST_HOST=195.19.90.188"
echo "DEST_BASE=/volume1/NetBackup/books-vds"
echo "RETENTION_DAYS=7"
echo "SSH_KEY=/opt/stacks/backup/kreknin-key"
echo "ES_SNAPSHOT_REPO=kreknin"
pass show books-vds/full-env | grep -E '^(BOOKS_DB_ROOT_PASSWORD|BOOKS_MONGO_)'
pass show vds-kzntsv/full-env | grep -E '^NTFY_'
pass show snolla-smtp/full-env | grep -E '^(SMTP_|OPS_NOTIFY_EMAIL)'
} | scp /dev/stdin root@89.253.255.133:/opt/stacks/backup/.env
ssh root@89.253.255.133 'chmod 600 /opt/stacks/backup/.env && chmod 755 /opt/stacks/backup/run.sh'
# Generate kreknin SSH key on books VDS:
ssh root@89.253.255.133 'ssh-keygen -t ed25519 -f /opt/stacks/backup/kreknin-key -N "" -C "books-vds-backup"'
# Get pubkey and authorize it on kreknin (workstation pivots — books VDS has no kreknin SSH yet):
PUBKEY=$(ssh root@89.253.255.133 cat /opt/stacks/backup/kreknin-key.pub)
ssh -i ~/.ssh/id_ed25519_kreknin vitya@195.19.90.188 \
"echo '$PUBKEY' >> ~/.ssh/authorized_keys && mkdir -p /volume1/NetBackup/books-vds"
# Verify SSH books → kreknin
ssh root@89.253.255.133 'ssh -i /opt/stacks/backup/kreknin-key -o StrictHostKeyChecking=accept-new vitya@195.19.90.188 hostname'
# Smoke run (manual)
ssh root@89.253.255.133 '/opt/stacks/backup/run.sh'
# Install cron
ssh root@89.253.255.133 'echo "0 6 * * * root /opt/stacks/backup/run.sh" > /etc/cron.d/books-vds-backup && chmod 644 /etc/cron.d/books-vds-backup'
```
## Decisions log
- **ES snapshot via REST API**, not data-dir rsync. Filesystem snapshot repo (`type=fs`, `location=/snapshots`, `compress=true`) registered as `kreknin` on 2026-05-25. Required one-time stack edit (add `path.repo=/snapshots` env + `./snapshots:/snapshots` bind in `/usr/docker/elasticsearch/docker-compose.yml`).
- **`curl --ssl-reqd` SMTP** instead of msmtp — avoid CentOS 7 msmtp install (EOL repo issues). Native `curl` works on STARTTLS port 587.
- **No `tasks.yml`-style auth on `books-job-scheduler-mongo`** — container has no `MONGO_INITDB_ROOT_*` env. `mongodump --archive` без `--uri` достаточен.
- **rsync /opt/books + /usr/docker/{minio,elasticsearch/snapshots,traefik}** — bind paths из inspect 2026-05-25; не `/var/lib/docker/volumes/` (named volumes для контейнеров пустые, кроме portainer/scheduler-mongo).
- **`books-vds-portainer/full-env` consolidated** into `books-vds/full-env` for one-file-per-host pattern parity with `vds-kzntsv/full-env`.
- **No `BackupConfig` snapshot/git pre-commit-like step** — VDS-infra `run.sh` тоже не делает, и нет аналогичной системы на books VDS.
## Smoke run (manual)
```bash
ssh root@89.253.255.133 '/opt/stacks/backup/run.sh'
# Watch log:
ssh root@89.253.255.133 'tail -F /var/log/books-vds-backup/$(date +%Y-%m-%d).log'
```
Verify on kreknin:
```bash
ssh vitya@195.19.90.188 'du -sh /volume1/NetBackup/books-vds/*/'
```
## Atomic revert (uninstall)
```bash
# On books VDS:
ssh root@89.253.255.133 'rm /etc/cron.d/books-vds-backup; rm -rf /opt/stacks/backup /var/log/books-vds-backup'
# On kreknin (via SSH):
ssh vitya@195.19.90.188 'rm -rf /volume1/NetBackup/books-vds'
# Remove the books-vds pubkey line from authorized_keys (one line with 'books-vds-backup' comment).
# On books VDS — revert ES path.repo change:
ssh root@89.253.255.133 'cd /usr/docker/elasticsearch && \
cp docker-compose.yml.bak-pre-snapshots-2026-05-25 docker-compose.yml && \
docker rm -f elasticsearch && docker-compose up -d'
```

216
scripts/books-vds-backup-daily-kreknin/run.sh Normal file
View File

@@ -0,0 +1,216 @@
#!/bin/bash
# Daily books VDS → kreknin backup. Triggered by /etc/cron.d/books-vds-backup at 06:00 MSK.
# Pattern: mirror VDS scripts/vds-backup-rsync-kreknin/run.sh, adapted for books VDS hybrid stack.
set -euo pipefail
ENV_FILE=/opt/stacks/backup/.env
# shellcheck disable=SC1090
source "$ENV_FILE"
TODAY=$(date +%Y-%m-%d)
LOG_DIR=/var/log/books-vds-backup
LOG="$LOG_DIR/$TODAY.log"
LOCK=/tmp/books-vds-backup.lock
DUMP_DIR=/tmp/books-vds-backup-dumps/$TODAY
DEST_PATH="$DEST_BASE/$TODAY"
LATEST="$DEST_BASE/latest"
mkdir -p "$LOG_DIR" "$DUMP_DIR"
exec >> "$LOG" 2>&1
echo "===== BOOKS-VDS backup start $TODAY $(date -Is) ====="
# single-instance lock
exec 9>"$LOCK"
if ! flock -n 9; then echo "already running, exiting"; exit 0; fi
START=$(date +%s)
SSH_OPTS="-i $SSH_KEY -o StrictHostKeyChecking=yes -o ConnectTimeout=30 -o LogLevel=ERROR"
# Note: kreknin host key must be pre-populated in /root/.ssh/known_hosts (CentOS 7 OpenSSH 7.4 has no accept-new option)
HOST_LABEL=BOOKS-VDS
SOURCE_DISPLAY="89.253.255.133 (books VDS)"
notify() {
local title="$1" msg="$2" tags="${3:-}" prio="${4:-default}"
curl -sS --max-time 10 -u "$NTFY_USER:$NTFY_PASS" \
-H "Title: $title" -H "Tags: $tags" -H "Priority: $prio" \
-d "$msg" "$NTFY_URL/vds-backup" > /dev/null || true
}
email_send() {
local subj="$1" body="$2"
local tmpf
tmpf=$(mktemp)
# RFC 5322: CRLF headers + Date required by Yandex
{
printf 'From: %s\r\n' "$SMTP_FROM"
printf 'To: %s\r\n' "$OPS_NOTIFY_EMAIL"
printf 'Subject: %s\r\n' "$subj"
printf 'Date: %s\r\n' "$(date -R)"
printf 'MIME-Version: 1.0\r\n'
printf 'Content-Type: text/plain; charset=UTF-8\r\n'
printf '\r\n'
echo "$body" | sed 's/$/\r/'
} > "$tmpf"
# smtps:// (implicit TLS) for Yandex port 465; smtp:// (STARTTLS) would be port 587.
# SMTP_PORT in pass is 465 — use smtps:// scheme.
curl -sS --max-time 30 \
--ssl-reqd \
--url "smtps://$SMTP_HOST:$SMTP_PORT" \
--user "$SMTP_USER:$SMTP_PASS" \
--mail-from "$SMTP_FROM" \
--mail-rcpt "$OPS_NOTIFY_EMAIL" \
--upload-file "$tmpf" || true
rm -f "$tmpf"
}
fmt_duration() {
local s=$1
printf "%dm%02ds" $((s/60)) $((s%60))
}
on_error() {
local line=$1 rc=$2
local err="line $line, exit $rc"
local now
now=$(date +%s)
local dur=$((now - ${START:-now}))
local dur_h
dur_h=$(fmt_duration "$dur")
local tail_log
tail_log=$(tail -40 "$LOG" 2>/dev/null || true)
echo "FATAL: $HOST_LABEL backup FAILED — $err"
notify "$HOST_LABEL backup FAILED $TODAY" \
"After $dur_h: $err. See $LOG" \
red_circle high
email_send "[$HOST_LABEL] backup FAILED $TODAY" \
"$HOST_LABEL daily backup FAILED.
Date: $TODAY
Duration: $dur_h
Error: $err
Source: $SOURCE_DISPLAY
Log: $LOG
Tail (last 40 lines):
$tail_log"
exit "$rc"
}
trap 'on_error $LINENO $?' ERR
# === Step 1: DB dumps ===
echo "--- DB dumps to $DUMP_DIR ---"
docker exec books-db mariadb-dump -uroot -p"$BOOKS_DB_ROOT_PASSWORD" \
--all-databases --single-transaction --quick 2>/dev/null \
| gzip > "$DUMP_DIR/mariadb.sql.gz"
echo "mariadb: $(stat -c %s "$DUMP_DIR/mariadb.sql.gz") bytes"
docker exec mongo mongodump --archive --quiet \
--uri="mongodb://$BOOKS_MONGO_USERNAME:$BOOKS_MONGO_PASSWORD@localhost:27017/?authSource=admin" 2>/dev/null \
| gzip > "$DUMP_DIR/mongo.archive.gz"
echo "mongo (shared): $(stat -c %s "$DUMP_DIR/mongo.archive.gz") bytes"
docker exec books-job-scheduler-mongo mongodump --archive --quiet 2>/dev/null \
| gzip > "$DUMP_DIR/job-scheduler-mongo.archive.gz"
echo "job-scheduler-mongo: $(stat -c %s "$DUMP_DIR/job-scheduler-mongo.archive.gz") bytes"
# === Step 2: ES snapshot via REST ===
echo "--- ES snapshot daily-$TODAY in repo $ES_SNAPSHOT_REPO ---"
ES_SNAP="daily-$TODAY"
SNAP_RESULT=$(docker exec elasticsearch curl -sS -X PUT \
"http://localhost:9200/_snapshot/$ES_SNAPSHOT_REPO/$ES_SNAP?wait_for_completion=true" \
-H 'Content-Type: application/json')
echo "ES snapshot result: $SNAP_RESULT"
if ! echo "$SNAP_RESULT" | grep -q '"state":"SUCCESS"'; then
echo "ES snapshot did NOT report SUCCESS state — failing"
exit 3
fi
echo "--- prune ES snapshots older than $RETENTION_DAYS ---"
# Pure bash parse — no jq/python dependency
ALL_SNAPS=$(docker exec elasticsearch curl -sS "http://localhost:9200/_snapshot/$ES_SNAPSHOT_REPO/_all" \
| grep -oE '"snapshot":"daily-[0-9-]+"' \
| sed -E 's/.*"(daily-[0-9-]+)".*/\1/' \
| sort)
SNAP_COUNT=$(echo "$ALL_SNAPS" | wc -l)
PRUNE_COUNT=$((SNAP_COUNT - RETENTION_DAYS))
if [ $PRUNE_COUNT -gt 0 ]; then
echo "$ALL_SNAPS" | head -n "$PRUNE_COUNT" | while read -r snap; do
[ -z "$snap" ] && continue
echo "deleting ES snapshot: $snap"
docker exec elasticsearch curl -sS -X DELETE \
"http://localhost:9200/_snapshot/$ES_SNAPSHOT_REPO/$snap" || true
echo
done
fi
# === Step 3: rsync to kreknin ===
echo "--- rsync to $DEST_USER@$DEST_HOST:$DEST_PATH ---"
ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "mkdir -p '$DEST_PATH'"
LINK_DEST_ARG=""
if ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "test -L '$LATEST' || test -d '$LATEST'" 2>/dev/null; then
LINK_DEST_ARG="--link-dest=$LATEST"
fi
rsync -aHh --info=stats2 --delete $LINK_DEST_ARG \
-e "ssh $SSH_OPTS" \
/opt/books \
/usr/docker/elasticsearch/snapshots \
/usr/docker/minio/data \
/usr/docker/traefik/letsencrypt \
/usr/docker/traefik/data \
/etc/ssh \
/etc/hosts \
/etc/cron.d \
/root/.ssh \
"$DUMP_DIR" \
"$DEST_USER@$DEST_HOST:$DEST_PATH/" || true
ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "rm -f '$LATEST' && ln -s '$DEST_PATH' '$LATEST'"
# === Step 4: retention prune on kreknin ===
echo "--- prune snapshots older than $RETENTION_DAYS days on kreknin ---"
ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" \
"cd '$DEST_BASE' && ls -1d 20*-*-* 2>/dev/null | sort | head -n -$RETENTION_DAYS | xargs -r rm -rf"
# === Step 5: local cleanup ===
rm -rf "$DUMP_DIR"
# === Step 6: report ===
END=$(date +%s)
DURATION=$((END - START))
SIZE=$(ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "du -sh '$DEST_PATH' 2>/dev/null | cut -f1" || echo "?")
SNAPSHOT_COUNT=$(ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "ls -1d $DEST_BASE/20*-*-* 2>/dev/null | wc -l" || echo "?")
DURATION_HUMAN=$(fmt_duration "$DURATION")
NTFY_BODY="$DURATION_HUMAN, size=$SIZE, snapshots=$SNAPSHOT_COUNT, dest=kreknin:$DEST_PATH"
notify "$HOST_LABEL backup OK $TODAY" "$NTFY_BODY" green_circle
EMAIL_BODY="$HOST_LABEL daily backup completed successfully.
Date: $TODAY
Duration: $DURATION_HUMAN
Size: $SIZE
Snapshots: $SNAPSHOT_COUNT
Source: $SOURCE_DISPLAY
Dest: kreknin:$DEST_PATH
Components:
- /opt/books/{api,job-scheduler,task-runner,ntfy,books-ops-mcp}
- /usr/docker/{minio/data, elasticsearch/snapshots, traefik/{letsencrypt,data}}
- /etc/{ssh,hosts,cron.d}
- /root/.ssh
- DB dumps: books-db (mariadb), mongo (shared), books-job-scheduler-mongo
- ES snapshot: $ES_SNAP
Log: $LOG"
email_send "[$HOST_LABEL] backup OK $TODAY" "$EMAIL_BODY"
echo "$HOST_LABEL backup OK: $NTFY_BODY"
echo "===== $HOST_LABEL backup done $(date -Is) ====="