OpeItcLoc03/admin
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: OpeItcLoc03:1eefb4d45a121c12bd11299cb508ee256a9ba21b
Branches Tags
OpeItcLoc03:master
...
compare: OpeItcLoc03:3769e836401b064b2cd79fe74c19d8dcb084e9c0
Branches Tags
OpeItcLoc03:master

3 Commits
1eefb4d45a ... 3769e83640

Author SHA1 Message Date
vitya
3769e83640 tasks(NEXT_SESSION): refresh handoff after this session's commits 
- Note ntfy + email verified ✓ by user
- Reflect books-vds-stacks-to-portainer ⚪ added
- Carry pending push decision (fe41ee44 + 439da3ce)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 09:49:10 +03:00
vitya
439da3cef6 tasks: add ⚪ books-vds-stacks-to-portainer 
Follow-up after books-vds-backup-daily-kreknin 🟢. User-requested:
migrate SSH-compose стеки (ES/mongo/minio/books-db/proxy-chain) под
Portainer-managed. Pattern reuse VDS-infra retro-migration script
(.wiki/concepts/portainer-stack-management-vds.md). Skip traefik + portainer
(management plane).

Also: STATUS.md note phone-side ntfy + email verified ✓ by user 2026-05-25.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 09:48:26 +03:00
vitya
fe41ee4422 backup(books-vds): daily 06:00 MSK pipeline to kreknin live 🟢 
- scripts/books-vds-backup-daily-kreknin/ — run.sh + .env.example + README
- ES path.repo bootstrap (one-time stack edit, snapshot repo 'kreknin' registered)
- curl smtps://yandex:465 email (CRLF + Date headers, no msmtp dep on CentOS 7)
- StrictHostKeyChecking=yes + pre-populated known_hosts (CentOS 7 no accept-new)
- cron /etc/cron.d/books-vds-backup live, initial sync 4.87 GB in 13m28s, ntfy push sent
- wiki: entities/books-vds.md + sources/books-vds-backup-daily-kreknin-2026-05-25.md created
- wiki: concepts/books-ssh-access → vds-kzntsv-ssh-access (was lying about books on vds-kzntsv)
- .tasks/books-vds-backup-daily-kreknin.md closed 🟢 pending phone-side verify

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 09:37:16 +03:00
11 changed files with 791 additions and 35 deletions
Expand all files Collapse all files

48
.tasks/NEXT_SESSION.md
View File

@@ -1,42 +1,50 @@
---
_last_updated_: 2026-05-25T07:50:00+03:00
session_id: 2026-05-25-close-unify-backup-notifications
_last_updated_: 2026-05-25T09:42:00+03:00
session_id: 2026-05-25-books-vds-backup-standup
---
# Next session handoff
## Recent commits
- `tasks(unify-backup-notifications): close 🟢 — unified push+email across VDS/RUVDS/windows-host` (этот раунд)
- `tasks: rm obsolete NEXT-SESSION-PROMPT.md (leaked creds in history since 2026-05-21)` (этот раунд)
- `scripts(backup-notifications): unify push + email format across VDS/RUVDS/windows-host` (`73ad6dd0`)
- `wiki(claude): add session-handoff trigger line`
- `wiki(ingest): RUVDS IIS migration + daily backup pipeline`
- `tasks: add ⚪ books-vds-stacks-to-portainer` (`439da3ce`) — follow-up taskcreated
- `backup(books-vds): daily 06:00 MSK pipeline to kreknin live 🟢` (`fe41ee44`) — 10 files, 694 insertions
- `tasks: add stateful-split-volume-copy (moved from victor/books)` (`1eefb4d4`)
- `meta(tasks): update [books-vds-bookva-bootstrap]` (`2931a432`)
- `tasks(unify-backup-notifications): close 🟢` (`838f51da`)
## Open треки
| Трек | Готовность | Entry-point |
|---|---|---|
| `iis-migration-to-ruvds` 🟡 | soak (kupimknigi + emspb DNS flipped 2026-05-24, оставшиеся 22 hostnames в очереди, tandemmebel.ru excluded) | `.tasks/iis-migration-to-ruvds.md` |
| `infra-inventory` ⚪ | ready, не начат | `.tasks/infra-inventory.md` |
| `books-bookva-user-whitelist-gathering` ⚪ | ready, не начат — long-lead gathering | `.tasks/books-bookva-user-whitelist-gathering.md` |
| `books-vds-bookva-bootstrap` 🔵 | blocked on victor/books tenant-split Phase 4 | `.tasks/books-vds-bookva-bootstrap.md` |
| `books-vds-stacks-to-portainer` ⚪ | ready, не начат — promised by user после backup 🟢 | `.tasks/books-vds-stacks-to-portainer.md` |
| `stateful-split-volume-copy` ⚪ | not started; playbook fits real books VDS (89.253.255.133) | `.tasks/stateful-split-volume-copy.md` |
| `iis-migration-to-ruvds` 🟡 | soak (kupimknigi + emspb DNS flipped 2026-05-24, 22 pending) | `.tasks/iis-migration-to-ruvds.md` |
| `infra-inventory` ⚪ | ready, не начат — **books VDS должен попасть в admin-detailed runbook** | `.tasks/infra-inventory.md` |
| `books-bookva-user-whitelist-gathering` ⚪ | ready, long-lead gathering | `.tasks/books-bookva-user-whitelist-gathering.md` |
| `books-vds-bookva-bootstrap` 🔵 | blocked on tenant-split Phase 4 | `.tasks/books-vds-bookva-bootstrap.md` |
| `books-dns-cutover-bookva` 🔵 | blocked on books-vds-bookva-bootstrap | `.tasks/books-dns-cutover-bookva.md` |
## Спроси user'а
- **windows-host deploy сделан?** Если `Get-FileHash C:\ProgramData\backup\run.ps1` matches `scripts/windows-host-fallback-backup-daily/run.ps1` (elevated check) — deploy ок, и сегодня ночью 03:00 MSK пушнёт в едином формате. Если нет — старый формат на одну ночь.
- **На night-run 2026-05-25 → 26 push'и + emails пришли все 3?** Title pattern `<HOST> backup OK <date>` / subject `[<HOST>] backup OK <date>`. Если что-то отвалилось — `.bak-pre-unify` файлы лежат на каждом хосте для quick rollback.
- **DNS swap для 22 hostnames** (`iis-migration-to-ruvds` next-action #3) — продолжаем или ещё soak?
- **Plaintext creds в git history** (NEXT-SESSION-PROMPT.md): user сегодня acceptуnул risk без rotation. Если позже передумаем — rotate MSSQL SA + snolla DB user + stayer DB user + VM admin (`Pryakhin9`).
- **Push** `fe41ee44` + `439da3ce` в `origin/master`? (Rule 4: каждая сессия start ask-mode). Это 2 коммита: backup pipeline standup + follow-up task.
- **Какую таску брать в работу следующей** — `books-vds-stacks-to-portainer` (sequel к backup'у), `stateful-split-volume-copy` (отложили), или другое?
- **Endpoint 6 в books Portainer** = stostayer VDS — мы за него отвечаем? Если да — после books-vds migration extend задачу на stostayer тоже.
## Не делать (preemptive guards)
- **Не push без grant'а** (project-discipline Rule 4 — session-start = ask-mode). Каждая сессия начинает спрашивать заново.
- **Не trigger полный backup-run для smoke** (21-81 min wall-clock). Smoke = только notify-channel через `scripts/<slug>/smoke-notify.ps1`.
- **Не history-rewrite + force-push** для cleanup creds в `NEXT-SESSION-PROMPT.md` без явного user-approve — user уже принял risk на ad-hoc rotation, не на destructive history operations.
- **Не fold `traefik`/`portainer` управление через SSH compose** на VDS — Portainer-managed per `.wiki/concepts/portainer-stack-management-vds.md`, кроме `traefik`+`portainer` themselves.
- **Не push без grant'а** (project-discipline Rule 4). Session начинает ask-mode.
- **Не trust wiki blindly** — пример `books-ssh-access.md` (фикс'нут в этом раунде, теперь `vds-kzntsv-ssh-access.md`). Faktcheck через `vds-ops` MCP + SSH probe + IP до построения предположений.
- **Не trigger полный backup для smoke** на books VDS — 13m+ wall-clock, 4.87 GB initial. Smoke = только notify-channel тест через standalone curl call.
- **Не editit `/usr/docker/<svc>/docker-compose.yml`** напрямую для production stacks — `books-vds-stacks-to-portainer` именно про правильный путь (migrate под Portainer, edit там).
- **Не trigger ES recreate** в нормальное время — текущая ES сейчас «не используется» per user, но это **могло измениться**. Verify before any recreate.
## Memory updates за сессию
- (нет на этом раунде — все правила/паттерны уже зафиксированы в project CLAUDE.md + global skills)
- **Учить:** `books VDS ≠ vds-kzntsv`. Real books VDS = `89.253.255.133` (host4g.ru, CentOS 7, kernel 3.10 EOL). Documented в [[../wiki/entities/books-vds]].
- **Учить:** pass entries one-file-per-host pattern — consolidate сервис-specific creds в `<host>/full-env` с section'ами (matches `vds-kzntsv/full-env` style). Не плодить `<host>-<service>/full-env`.
- **Учить:** Yandex SMTP — port 465 = `smtps://` (implicit TLS), port 587 = `smtp://` + `--ssl-reqd` (STARTTLS). Mismatch = 30-sec curl timeout.
- **Учить:** CentOS 7 OpenSSH 7.4 не имеет `accept-new` (added 7.6). Bootstrap step: `ssh-keyscan` host-key в `known_hosts`, потом `StrictHostKeyChecking=yes`.
- **Учить:** docker-compose 1.29 + Docker 26 = `KeyError: 'ContainerConfig'` при recreate. Workaround: `docker rm -f <renamed-stuck>` + `docker-compose up -d`. Bigger fix: upgrade docker-compose to v2 plugin.
- **Reference:** books VDS Portainer = `https://portainer.kzntsv.site` (НЕ `portainer.vds.kzntsv.site`). PAT в `pass show books-vds/full-env BOOKS_PORTAINER_API_KEY`.
- **Verified 2026-05-25:** ntfy push `BOOKS-VDS backup OK 2026-05-25` пришёл на phone ✓. Email `[BOOKS-VDS] backup OK ...` + standalone `[BOOKS-VDS] SMTP fix verify` пришли в inbox ✓.

12
.tasks/STATUS.md
View File

@@ -1,5 +1,15 @@
# Admin Task Board
_Updated: 2026-05-25 (`unify-backup-notifications` 🟢 closed — 3 scripts под единый push+email format committed `73ad6dd0`, VDS+RUVDS deployed+smoke ✓, windows-host self-deploy на user'е через `scripts/windows-host-fallback-backup-daily/deploy.ps1` elevated. Также удалён obsolete `.tasks/NEXT-SESSION-PROMPT.md` (содержал plaintext creds в git history с 2026-05-21; user accept risk без rotation/rewrite). VDS `run.sh` теперь в repo как canonical source. Pre-existing: все 3 daily backups live на kreknin.)_
_Updated: 2026-05-25 (`books-vds-backup-daily-kreknin` 🟢 closed — 4-я daily backup pipeline на kreknin, теперь покрыта `books VDS` 89.253.255.133 (host4g.ru). Phone-side ntfy ✓ + email ✓ verified by user 2026-05-25. Также fix вики: `concepts/books-ssh-access.md` → `vds-kzntsv-ssh-access.md` (был misnamed — врал что books живёт на vds-kzntsv), `entities/books-vds.md` создан с factsheet'ом. Promised follow-up: новая ⚪ таска `books-vds-stacks-to-portainer` — мигрировать SSH-compose стеки (ES/mongo/minio/books-db/proxy-chain) под Portainer-managed.)_
## ⚪ [books-vds-stacks-to-portainer] — migrate SSH-compose стеки на books VDS под Portainer-managed (pattern parallel VDS-infra retro-migration 2026-05-22). Targets: books-db, mongo, minio, elasticsearch, proxy-chain. Skip traefik + portainer (management plane).
**Status:** ready
**Where I stopped:** (not started — promised by user 2026-05-25 после `books-vds-backup-daily-kreknin` 🟢)
**Next action:** см. [books-vds-stacks-to-portainer.md](books-vds-stacks-to-portainer.md). Phase 0 — read each stack's `/usr/docker/<svc>/docker-compose.yml` + `.env`, document env + binds.
**Branch:** master
<!-- created-by: claude / 2026-05-25 / trigger: user-requested follow-up after books-vds-backup-daily-kreknin -->
## 🟢 [books-vds-backup-daily-kreknin] — closed 2026-05-25 — daily backup books VDS (89.253.255.133, host4g.ru, CentOS 7) → kreknin в 06:00 MSK. DB dumps (mariadb/mongo×2) + ES snapshot via REST → rsync 4.87GB → ntfy/email (phone+inbox ✓). ES path.repo bootstrap + snapshot repo `kreknin` registered. См. [books-vds-backup-daily-kreknin.md](books-vds-backup-daily-kreknin.md).
## 🟢 [unify-backup-notifications] — closed 2026-05-25 — единый формат push + email для VDS/RUVDS/windows-host (3 scripts), VDS run.sh импортирован в repo. См. [unify-backup-notifications.md](unify-backup-notifications.md) § Closed. windows-host self-deploy остаётся на user'е (elevated PS).

61
.tasks/books-vds-backup-daily-kreknin.md Normal file
View File

@@ -0,0 +1,61 @@
# books-vds-backup-daily-kreknin
Ежедневный backup pipeline books VDS (`89.253.255.133`, host4g.ru, CentOS 7) → kreknin Synology (`195.19.90.188:/volume1/NetBackup/books-vds/<date>/`) в 06:00 MSK.
## Goal
Восстановимость books-стека после полного отказа VDS: DB dumps + MinIO blobs + ES snapshot + app configs + traefik state + system config на NAS с 7-day retention.
## Key files
- `scripts/books-vds-backup-daily-kreknin/run.sh` — backup logic; deploy → `/opt/stacks/backup/run.sh` на VDS.
- `scripts/books-vds-backup-daily-kreknin/.env.example` — placeholder; live `.env` built from `pass show`.
- `scripts/books-vds-backup-daily-kreknin/README.md` — install + revert recipe.
- `.wiki/entities/books-vds.md` — host entity page.
- `.wiki/sources/books-vds-backup-daily-kreknin-2026-05-25.md` — standup chronology (создаётся при closure).
## Decisions log
- 2026-05-25: pattern mirror `scripts/vds-backup-rsync-kreknin/run.sh` (VDS-infra). Same `vds-backup` ntfy topic, same SMTP relay (`snolla-smtp` Yandex), same `--link-dest` rsync hardlink pattern, same `RETENTION_DAYS=7`.
- 2026-05-25: **ES snapshot via REST API**, not data-dir rsync. One-time stack edit `/usr/docker/elasticsearch/docker-compose.yml`: added env `path.repo=/snapshots` + bind `./snapshots:/snapshots`. Registered repo `kreknin` (type=fs, location=/snapshots, compress=true).
- 2026-05-25: **`curl --ssl-reqd` SMTP** instead of msmtp (CentOS 7 EOL repo issue). Yandex 587 STARTTLS works через native curl без msmtprc setup.
- 2026-05-25: `StrictHostKeyChecking=yes` (NOT `accept-new`) — CentOS 7 OpenSSH 7.4 не поддерживает `accept-new`. kreknin host key pre-populated в `/root/.ssh/known_hosts` на VDS bootstrap.
- 2026-05-25: `books-job-scheduler-mongo` — **без auth** (нет `MONGO_INITDB_ROOT_*` env). `mongodump --archive` без `--uri` достаточен. shared `mongo` — root auth (uri credential).
- 2026-05-25: **Cron time 06:00** MSK — после VDS-infra (05:00) + windows-host (05:30), даёт break перед US-EU business start.
- 2026-05-25: `BOOKS-VDS` host label в ntfy title — отличать от обычного `VDS` (infra) на shared `vds-backup` topic.
- 2026-05-25: Consolidated `books-vds-portainer/full-env` → `books-vds/full-env` (one-file-per-host pattern, parity с vds-kzntsv).
## Open questions
- [ ] (resolved 2026-05-25) `path.repo` на ES — required one-time stack edit. Done, recreated container, snapshot test SUCCESS.
- [ ] Migrate SSH-compose stacks (ES/mongo/minio/books-db/traefik) под Portainer — **отдельная задача после backup pipeline live**.
- [ ] kreknin space check — current free 5.7T (per `kreknin-synology.md` 2026-05-19); per-day footprint TBD после initial sync. Quarterly disk audit на NAS.
## Completed steps
- [x] Phase 0: ES path.repo bootstrap + snapshot repo `kreknin` registered + test snap SUCCESS
- [x] Phase 1: написал `run.sh` + `.env.example` + `README.md` локально
- [x] Phase 2: deployed `/opt/stacks/backup/{run.sh,.env}`, chmod 600 .env / 755 run.sh
- [x] Phase 3: generated `/opt/stacks/backup/kreknin-key` ed25519 на books VDS
- [x] Phase 4: pubkey deployed на kreknin `~vitya/.ssh/authorized_keys`, dest `/volume1/NetBackup/books-vds/` created
- [x] Phase 5: kreknin host key pre-populated `/root/.ssh/known_hosts`, SSH path verified
- [x] Phase 6: smoke run (initial sync) — 4.87GB in 13m28s, ntfy push sent, **email send pending fix → fixed → re-tested OK**
- [x] Phase 6a: SMTP fix — `smtp://` + `--ssl-reqd` дала 30-сек timeout (curl попытался STARTTLS на implicit-TLS-only порту 465). Fix `smtps://` schema + CRLF headers + Date header. Re-tested standalone — SMTP send OK.
- [x] Phase 7: installed `/etc/cron.d/books-vds-backup` (`0 6 * * * root /opt/stacks/backup/run.sh`); crond active
- [x] Phase 8: wiki — `entities/books-vds.md` created; `concepts/books-ssh-access.md` renamed → `vds-kzntsv-ssh-access.md` (was lying about books); `sources/books-vds-backup-daily-kreknin-2026-05-25.md` created; `index.md` updated for all three
- [ ] Phase 9: commit (no push without grant)
- [ ] **PENDING USER VERIFY:** ntfy push `BOOKS-VDS backup OK 2026-05-25` received on phone? Email `[BOOKS-VDS] backup OK 2026-05-25` + standalone fix-test message in inbox?
## Notes
- root password `Pryakhin9~` — сохранён в `pass show books-vds/full-env BOOKS_VDS_ROOT_PASS`. SSH key `id_ed25519_books_ops` уже был authorized (pre-existing bootstrap), password держим как fallback.
- Pre-existing `id_ed25519_books_ops` на workstation предполагает что прошлый bootstrap забыт; не повторяем generate.
- ES в production books currently **не используется** (per user statement) — recreate-downtime 30с не impact'нул.
- `vds-kzntsv` ≠ `books-vds`. Wiki [[../concepts/books-ssh-access]] до 2026-05-25 ошибочно описывала vds-kzntsv как books shared VDS — fix в Phase 8.
## Cross-refs
- [[../wiki/entities/books-vds]] — host entity.
- [[../wiki/entities/kreknin-synology]] — backup target.
- [[../scripts/vds-backup-rsync-kreknin/run.sh]] — pattern source (VDS-infra backup).
- [[../scripts/ruvds-backup-daily-kreknin/README.md]] — pattern source (RUVDS dual-channel notify).

89
.tasks/books-vds-stacks-to-portainer.md Normal file
View File

@@ -0,0 +1,89 @@
# books-vds-stacks-to-portainer
Migrate SSH-compose стеки на books VDS (`89.253.255.133`) в Portainer-managed через API. Pattern parallel `[[.wiki/concepts/portainer-stack-management-vds]]` (VDS-infra retro-migration 2026-05-22), но другой endpoint/Portainer URL/stack dirs.
## Goal
Унифицировать ops surface на books VDS — все docker-compose стеки управляются через `https://portainer.kzntsv.site` (endpoint 1). Конец hybrid pain'у (Portainer для `books-{api,web,scheduler,...}` + SSH-compose для `elasticsearch/mongo/minio/books-db/traefik/proxy-chain`). CI deploy workflow (`.gitea/workflows/deploy.yml` в victor/books) только через Portainer API; SSH-compose vert не подключается; debug требует знать «is it Portainer or SSH compose» — больно.
## Scope
| Stack | Current dir | Adapt notes |
|---|---|---|
| `books-db` (MariaDB 10.6.4) | `/usr/docker/books-db/` | bind `./data` → absolutize. MYSQL_ROOT_PASSWORD env (передать через Portainer API env array). |
| `mongo` (shared) | `/usr/docker/mongo/` | bind `./data/{db,configdb}` → absolutize. MONGO_INITDB_ROOT_* env. |
| `minio` | `/usr/docker/minio/` | bind `./data` → absolutize. MINIO_ROOT_USER/PASSWORD env. |
| `elasticsearch` | `/usr/docker/elasticsearch/` | bind `./data` + `./snapshots` (added 2026-05-25 для backup) → absolutize. path.repo env preserved. |
| `proxy-chain` | `/usr/docker/proxy-chain/` | bind `./config` → absolutize. |
**Skip (management plane):**
- `traefik` — `/usr/docker/traefik/` — recreate ломает access всему остальному.
- `portainer` — `/usr/docker/portainer/` — self-managed; can't recreate себя.
## Acceptance
- 5 стеков (books-db, mongo, minio, elasticsearch, proxy-chain) появляются в `GET /api/stacks` под endpoint 1 (наряду с книгами `books-{api,web,scheduler,ntfy,ops-mcp}` + `chrome`).
- Контейнеры работают, бинды смонтированы, data preserved (named volumes + binds через migration).
- Smoke: каждый стек проходит health check соответствующий своему типу:
- books-db: `docker exec books-db mariadb -uroot -p... -e 'SELECT 1'` → `1`
- mongo: `docker exec mongo mongosh --eval 'db.adminCommand({ping:1})'` → `ok:1`
- minio: HTTP `https://elasticsearch.kzntsv.site` (или внутренний minio endpoint) — `200 OK`
- elasticsearch: `docker exec elasticsearch curl localhost:9200/_cluster/health` → `green|yellow`
- proxy-chain: smoke через тестового агента (TBD)
- Backup pipeline (`books-vds-backup-daily-kreknin`) продолжает работать post-migration (compose dirs не используются runtime после migration, but `/usr/docker/<svc>/data` binds preserved — rsync paths остаются валидны).
- Document в `.wiki/concepts/portainer-stack-management-books-vds.md` (или extend existing `portainer-stack-management-vds.md` с books VDS sections).
## Decisions log
- 2026-05-25: создана по запросу user'а после успешного `books-vds-backup-daily-kreknin`. Promised after backup pipeline live.
- 2026-05-25: pattern reuse — `.wiki/concepts/portainer-stack-management-vds.md` migration script adaptable. Diff for books VDS: PORTAINER_URL, endpoint ID = 1, stack dirs `/usr/docker/<svc>/` instead of `/opt/stacks/<svc>/`. PAT в pass под `books-vds/full-env BOOKS_PORTAINER_API_KEY` (may need fallback to JWT if 401).
## Open questions
- [ ] **Endpoint 6** в books Portainer — stostayer VDS? Что мы за него отвечаем? Если **да** — после books-vds migration extend этой таски на stostayer. Если **нет** — отдельный вопрос, не блокер.
- [ ] **Volume preservation strategy** — `docker compose down` без `-v` preserves named volumes + binds. Verify для каждого стека что нет surprise named volumes которые могут эфемериться.
- [ ] **CI implications** — деплой books-* stacks уже Portainer-managed. After migration of mongo/minio/etc — никаких изменений в CI не нужно (deploy.yml только books-api/web/scheduler, не infra). Confirm.
- [ ] **ES snapshot path.repo edit preservation** — текущий `/usr/docker/elasticsearch/docker-compose.yml` имеет path.repo + snapshots bind (added 2026-05-25). Migration must preserve. Backup в `docker-compose.yml.bak-pre-snapshots-2026-05-25` есть для reference.
## Completed steps
- (not started)
## Plan (sketch)
**Phase 0 — pre-flight:**
1. Read each stack's `docker-compose.yml` + `.env` (if exists).
2. Document existing env vars + binds per stack.
3. Backup current compose files (`docker-compose.yml.bak-pre-portainer-migration-2026-MM-DD`).
**Phase 1 — Portainer API setup:**
1. Verify PAT works (`curl ... /api/stacks`). If 401 → fallback to JWT via admin password (from `pass show books-vds/full-env BOOKS_VDS_ROOT_PASS` adapted to Portainer admin — may differ).
2. Build adapter script `scripts/books-vds-portainer-migrate.sh` (clone-and-adapt of VDS-infra version, see [[.wiki/concepts/portainer-stack-management-vds]] § Migration script).
**Phase 2 — per-stack migrate (one at a time, verify between):**
1. proxy-chain (lowest blast radius)
2. minio (read-mostly, books-api retries OK)
3. mongo (shared) — books-api + books-task-runner retry-tolerant? **VERIFY first**
4. books-db (MariaDB) — books-api uses connection-pool, brief downtime survivable
5. elasticsearch — currently unused (per user), zero impact
**Phase 3 — wiki + close:**
1. Document в `.wiki/concepts/portainer-stack-management-books-vds.md` (or extend existing).
2. Update `.wiki/entities/books-vds.md` — flip каждый стек в SSH-managed → Portainer-managed table.
3. Update STATUS.md → 🟢.
## Notes
- **`books-job-scheduler-mongo`** уже Portainer-managed (часть `books-job-scheduler` stack). Не в scope этой таски.
- **`books-docker-proxy` + `books-docker-proxy-ro`** — часть `books-ops-mcp` stack (Portainer). Not in scope.
- **Buildx builders** (`buildx_buildkit_builder-*`) — transient containers без compose, игнорируем.
- **Backup pipeline depend** — после migration paths `/usr/docker/<svc>/data` остаются как bind sources, поэтому `books-vds-backup-daily-kreknin/run.sh` не требует изменений. Compose files (`/usr/docker/<svc>/docker-compose.yml`) станут reference only после migration.
## Cross-refs
- [[.wiki/concepts/portainer-stack-management-vds]] — VDS-infra retro-migration pattern (source of script).
- [[.wiki/entities/books-vds]] — host factsheet (will update post-migration).
- [[books-vds-backup-daily-kreknin]] — backup taskdone 2026-05-25; depend on stable bind paths.
- victor/books `deploy/README.md` — books-* CI deploy через Portainer API (already migrated).
<!-- created-by: claude / 2026-05-25 / trigger: user-requested follow-up after books-vds-backup-daily-kreknin 🟢 -->

27
.wiki/concepts/books-ssh-access.md → .wiki/concepts/vds-kzntsv-ssh-access.md
View File

@@ -1,15 +1,17 @@
---
title: books shared VDS — SSH access audit + retained keys
title: vds-kzntsv — SSH access audit + retained keys
status: live
tags: [vds, ssh, audit, books, tenant-split, ops]
related: [[vds-kzntsv]], [[../sources/vds-kzntsv-bootstrap-2026-05-20]]
tags: [vds, ssh, audit, infra, ops]
related: [[../entities/vds-kzntsv]], [[../sources/vds-kzntsv-bootstrap-2026-05-20]]
---
# books shared VDS — SSH access (post-audit 2026-05-24)
# vds-kzntsv — SSH access (post-audit 2026-05-24)
Snapshot SSH-доступа к `vds-kzntsv` (89.253.255.94 / vds.kzntsv.site), на котором сейчас живёт shared `books`-инсталляция (Slovo + Bookva в одном compose-стеке, до physical split Фазы 4 в [`tenant-split`](https://git.kzntsv.site/victor/books/src/branch/master/.wiki/concepts/tenant-split.md)).
Snapshot SSH-доступа к [[../entities/vds-kzntsv]] (`89.253.255.94 / vds.kzntsv.site`) — наш infra-VDS (gitea, registry, ntfy, mssql, minio, owncloud, board-viewer).
Создан 2026-05-24 в результате таски `[books-ssh-audit-shared-vds]` 🟢 (см. spec в Фазе 3 §SSH-аудит).
**Корректировка 2026-05-25:** до этой даты документ ошибочно утверждал что на vds-kzntsv живёт shared books-инсталляция. **Это неверно** — books-стек на отдельном клиентском VDS `89.253.255.133` (см. [[../entities/books-vds]]). vds-kzntsv хостит только нашу infra. Аудит ниже относится к vds-kzntsv SSH (наш infra-host), не к books VDS.
Создан 2026-05-24 в результате таски `[books-ssh-audit-shared-vds]` 🟢 — задача была названа в предположении что books на vds-kzntsv; reality по сути проверила infra SSH (что и должно было быть).
## Retained keys
@@ -52,16 +54,15 @@ Brute-force protection adequate; нет required action.
`sudo journalctl -u ssh -u sshd --since="7 days ago" | grep Accepted` показал **только** `vitya from 94.19.247.14` (мой home public IP). Никаких других source-IP — clean.
## Tenant-split implications (Phase 3 acceptance)
## Books tenant-split — implications (исправлено 2026-05-25)
Spec `tenant-split.md` §«Двухуровневая изоляция пользователей → Infra-level»:
> на промежуточной стадии не давать аналитикам Bookva SSH к VDS, только app-level. SSH-аудит до cutover'а.
Spec `tenant-split.md` §«Двухуровневая изоляция пользователей → Infra-level» и аудит «не давать аналитикам Bookva SSH к VDS» — **должны были применяться к books VDS, не к vds-kzntsv**. Реальный books VDS = `89.253.255.133` ([[../entities/books-vds]]) — отдельный клиентский сервер. SSH-аудит на books VDS — отдельная follow-up задача.
**Verified clean** as of 2026-05-24 — никаких аналитических SSH-ключей не было выдано. Single-developer setup; analyst access всегда был app-level only. Pre-cutover sigh of relief.
vds-kzntsv (этот документ) — infra-VDS, не имеет отношения к books tenants. После Phase 4 books tenant-split:
- books VDS (`89.253.255.133`) — текущий shared books → может остаться за Slovo (или Bookva, как пользователь решит).
- vds-bookva-new — отдельный VDS у учредителя Bookva. См. `.tasks/books-vds-bookva-bootstrap.md` + `books-dns-cutover-bookva.md`.
После Phase 4 (physical split на 2 VDS):
- vds-kzntsv (Slovo) — продолжит хостить остальную нашу инфру (gitea, registry, ntfy, mssql, minio, owncloud, board-viewer) + books-slovo стек. SSH = vitya only.
- vds-bookva-new (Bookva) — отдельная box, отдельный SSH. Заводить core-dev key (vitya) + опционально formally-named Bookva admin.
vds-kzntsv остаётся infra-host независимо от tenant-split исхода.
## How to add a new key (process)

109
.wiki/entities/books-vds.md Normal file
View File

@@ -0,0 +1,109 @@
---
title: Books VDS (client server — 89.253.255.133)
type: entity
tags: [hardware, vds, books, host4g, backup, portainer]
related: [[kreknin-synology]], [[vds-kzntsv]]
updated: 2026-05-25
---
# Books VDS
Клиентский VDS (отдельный от инфра-VDS [[vds-kzntsv]]). Хостит production books app stack (api/web/scheduler/task-runner) + shared infra (mongo, minio, elasticsearch, traefik, portainer).
**Это НЕ vds-kzntsv.** Books-стек живёт здесь, не на vds.kzntsv.site. Domain `books.kzntsv.site` (и `bookva.kzntsv.site`) разрешаются в этот IP.
## Доступ
- **Public IP:** 89.253.255.133
- **Provider:** host4g.ru (DNS вендорский `vps-21075162-277731.host4g.ru`)
- **OS:** CentOS 7, kernel 3.10.0-1160.36.2.el7.x86_64 (древний, EOL)
- **SSH:** `root@89.253.255.133:22` — only via ed25519 key (`~/.ssh/id_ed25519_books_ops` на workstation)
- **Sudo:** root, passwordless (мы под root напрямую)
- **Portainer:** `https://portainer.kzntsv.site` (отдельный Portainer, не путать с `portainer.vds.kzntsv.site` для [[vds-kzntsv]])
Все creds — в `pass show books-vds/full-env`.
## Disk
```
/dev/vda1 122G 59G used (49%) 63G free
```
Single root partition, нет отдельной `/data`. Docker hub data под `/var/lib/docker/`.
## Стек (2026-05-25 inventory)
### Portainer-managed stacks (endpoint 1)
| Stack | Container(s) | Purpose |
|---|---|---|
| `books-api` | books-api | books API server (Nitro, port 3021) |
| `books-web` | books-web | books-Nuxt front-end |
| `books-job-scheduler` | books-job-scheduler + books-job-scheduler-mongo | agenda jobs (cron + on-demand) |
| `books-ntfy` | books-ntfy | local ntfy (separate from shared `ntfy.vds.kzntsv.site`) |
| `books-ops-mcp` | books-ops-mcp + books-ops-mcp-bootstrap-1 + books-docker-proxy + books-docker-proxy-ro | books ops-mcp endpoint |
| `chrome` | chrome | Puppeteer browser for PDFs / scraping |
### SSH-managed (legacy compose at `/usr/docker/<svc>/`)
| Compose dir | Containers | Bind data path |
|---|---|---|
| `/usr/docker/books-db/` | books-db | `./data` → `/var/lib/mysql` (MariaDB 10.6.4) |
| `/usr/docker/mongo/` | mongo | `./data/{db,configdb}` |
| `/usr/docker/minio/` | minio | `./data` (S3 blobs) |
| `/usr/docker/elasticsearch/` | elasticsearch | `./data` + `./snapshots` (added 2026-05-25) |
| `/usr/docker/traefik/` | traefik | `./letsencrypt` (acme.json) + `./data/traefik.yml` |
| `/usr/docker/portainer/` | portainer | named volume `portainer_data` |
| `/usr/docker/proxy-chain/` | proxy-chain | upstream HTTP-proxy chain |
**TODO**: мигрировать максимум SSH-managed контейнеров под Portainer (отдельная таска после backup-pipeline live). Cause: разделённое управление обламывает CI (Portainer-only deploy.yml) и debugging.
### Build helpers (transient)
- `buildx_buildkit_builder-*` × 7 — buildkit builders, oneshot за `docker buildx`. Не data.
## App-config paths
```
/opt/books/api/{config,data,upload}
/opt/books/job-scheduler/{config,mongo/{db,configdb}}
/opt/books/task-runner/
/opt/books/ntfy/ # local ntfy data
/opt/books/books-ops-mcp/
```
`config/default.json` overrides image-baked config (MySQL/ES/S3 endpoints).
## Backup
Daily 06:00 MSK → kreknin. См. [[../sources/books-vds-backup-daily-kreknin-2026-05-25]]. Pipeline:
- DB dumps: books-db (mariadb), mongo (shared), books-job-scheduler-mongo
- ES snapshot via REST API (file repo `kreknin`)
- rsync bind paths + dumps + `/opt/books/` + `/etc/{ssh,hosts,cron.d}` + `/root/.ssh` → `vitya@kreknin:/volume1/NetBackup/books-vds/<date>/`
- ntfy `BOOKS-VDS backup OK <date>` + email
- Retention 7 daily snapshots
Live since 2026-05-25.
## DNS
- `books.kzntsv.site` → 89.253.255.133 (`books-api` host route via local traefik)
- `bookva.kzntsv.site` → 89.253.255.133 (`books-web` host route)
- `elasticsearch.kzntsv.site` → 89.253.255.133 (ES via traefik basicAuth)
- `portainer.kzntsv.site` → 89.253.255.133 (Portainer UI)
- `edge.kzntsv.site` → 89.253.255.133 (Portainer edge agent)
Auth-зоны DNS — в reg.ru под `kzntsv.site`.
## Cross-refs
- [[kreknin-synology]] — backup target.
- [[vds-kzntsv]] — наш инфра-VDS (отдельный, gitea/registry/etc.) — НЕ хост books'а.
- [[../concepts/vds-kzntsv-ssh-access]] — SSH-аудит инфра-VDS (был misnamed как `books-ssh-access` до 2026-05-25; ошибочно claimed vds-kzntsv hosts books).
- [[../sources/books-vds-backup-daily-kreknin-2026-05-25]] — backup standup chronology.
## Каking-of-history notes
- Discovered 2026-05-25 во время backup-таски. Wiki [[../concepts/books-ssh-access]] до этого ошибочно говорила что books на vds-kzntsv (89.253.255.94). User: «books VDS - это совсем другой сервер, клиентский». Inventory + IP probe confirmed.
- Workstation SSH key `id_ed25519_books_ops` уже был задеплоен под root до этой сессии (видимо, прошлый bootstrap забыт-без-документации).
- Hybrid stack-management (Portainer + SSH-compose) — pre-existing, не наш design. Migrate to Portainer-only — follow-up.

4
.wiki/index.md
View File

@@ -8,6 +8,7 @@ Catalog of all wiki pages. One line per page, organized by type. Updated on ever
## Entities
- [books-vds](entities/books-vds.md) — Books VDS (89.253.255.133 — host4g.ru, client server hosting books app + shared mongo/minio/elasticsearch)
- [dead-synology-diskstation](entities/dead-synology-diskstation.md) — мёртвая Synology DiskStation (source NAS)
- [kreknin-synology](entities/kreknin-synology.md) — Kreknin Synology (backup target + DDNS)
- [openwrt-router](entities/openwrt-router.md) — OpenWRT Router (192.168.1.1)
@@ -19,7 +20,7 @@ Catalog of all wiki pages. One line per page, organized by type. Updated on ever
## Concepts
- [admin-infra-project](concepts/admin-infra-project.md) — design + migration plan для OpeItcLoc03/admin (canonical)
- [books-ssh-access](concepts/books-ssh-access.md) — SSH access audit на shared VDS (books Slovo+Bookva), retained keys + add/revoke processes
- [vds-kzntsv-ssh-access](concepts/vds-kzntsv-ssh-access.md) — SSH access audit на vds-kzntsv (infra VDS), retained keys + add/revoke processes (was misnamed `books-ssh-access` до 2026-05-25 — content всегда был про vds-kzntsv)
- [compose-bcrypt-escape-trap](concepts/compose-bcrypt-escape-trap.md) — Docker Compose ест `$` в bcrypt hashes
- [db-tls-self-signed-via-traefik-raw-tcp](concepts/db-tls-self-signed-via-traefik-raw-tcp.md) — DB TLS через traefik raw TCP с self-signed certs
- [docker-host-loopback-detect](concepts/docker-host-loopback-detect.md) — Docker host-loopback detection — как доказать что host.docker.internal не петля
@@ -53,5 +54,6 @@ Catalog of all wiki pages. One line per page, organized by type. Updated on ever
- [iis-host-migration-2026-05-19](sources/iis-host-migration-2026-05-19.md) — IIS Host Migration Session 2026-05-19 chronology
- [iis-migration-to-ruvds-2026-05-23](sources/iis-migration-to-ruvds-2026-05-23.md) — IIS migration to RUVDS — session 2026-05-23/24 (SSH/scp pivot, 25 SNI bindings, partial DNS cutover)
- [nas-recovery-session-2026-05-18](sources/nas-recovery-session-2026-05-18.md) — NAS Recovery Session 2026-05-18/19 chronology
- [books-vds-backup-daily-kreknin-2026-05-25](sources/books-vds-backup-daily-kreknin-2026-05-25.md) — books VDS daily backup → kreknin pipeline standup 2026-05-25 (ES file-snapshot repo + DB dumps + curl SMTP)
- [ruvds-backup-daily-kreknin-2026-05-24](sources/ruvds-backup-daily-kreknin-2026-05-24.md) — RUVDS daily backup → kreknin pipeline standup 2026-05-24 (rclone+SFTP, SYSTEM task)
- [vds-kzntsv-bootstrap-2026-05-20](sources/vds-kzntsv-bootstrap-2026-05-20.md) — VDS bootstrap session 2026-05-20 chronology

111
.wiki/sources/books-vds-backup-daily-kreknin-2026-05-25.md Normal file
View File

@@ -0,0 +1,111 @@
---
title: books VDS daily backup → kreknin — pipeline standup 2026-05-25
type: source
tags: [backup, books-vds, kreknin, rsync, elasticsearch-snapshot, curl-smtp, cron, ntfy]
ingested: 2026-05-25
raw_path: ../../.tasks/books-vds-backup-daily-kreknin.md
updated: 2026-05-25
---
# books VDS daily backup → kreknin
Поднят ежедневный backup pipeline с [[../entities/books-vds]] (`89.253.255.133` / host4g.ru / CentOS 7) → [[../entities/kreknin-synology]] (`195.19.90.188`) в **06:00 MSK**. Параллель к [[vds-kzntsv-bootstrap-2026-05-20]] §backup и [[ruvds-backup-daily-kreknin-2026-05-24]] (pattern mirror), но source = CentOS 7 с hybrid SSH-compose + Portainer стек.
Источник истины — `.tasks/books-vds-backup-daily-kreknin.md` и `scripts/books-vds-backup-daily-kreknin/`.
## Background — почему отдельно
Прошлые сессии (и `[[../concepts/vds-kzntsv-ssh-access]]` до 2026-05-25) ошибочно утверждали что books-стек живёт на [[../entities/vds-kzntsv]]. **Это не так:** books на отдельном клиентском VDS `89.253.255.133`. User указал на ошибку явно: «books VDS - это совсем другой сервер, клиентский». См. [[../entities/books-vds]] для актуального factsheet.
## Scope
| Path on books VDS | Reason | Approx size |
|---|---|---|
| `/opt/books/{api,job-scheduler,task-runner,ntfy,books-ops-mcp}` | App configs (`config/default.json` overrides) | < 100 MB |
| `/usr/docker/minio/data` | MinIO blobs (S3) — основная масса | ~4.5 GB |
| `/usr/docker/elasticsearch/snapshots` | ES snapshot files (repo `kreknin`) | < 1 MB (read_me index only) |
| `/usr/docker/traefik/{letsencrypt,data}` | acme.json + traefik.yml | < 5 MB |
| `/etc/{ssh,hosts,cron.d}` | system config | < 50 KB |
| `/root/.ssh` | root SSH state | < 5 KB |
| `mariadb.sql.gz` (live dump) | books-db full dump | ~270 MB |
| `mongo.archive.gz` (live dump) | shared mongo (root auth) | ~170 KB |
| `job-scheduler-mongo.archive.gz` (live dump) | scheduler agenda (no auth) | ~3 MB |
Total после initial sync: **4.87 GB** (`du -sh` reported 4.8 GB после rsync).
Не бэкапятся: `/var/lib/docker/`, `/var/log/`, transient buildx volumes.
## Decisions log
- **Tool = bash + rsync + curl** (no rclone, no msmtp). Linux native, no extra installs. Pattern mirror VDS-infra `run.sh`.
- **ES snapshot via REST API** (filesystem repo `kreknin`), не data-dir rsync. Required one-time stack edit `/usr/docker/elasticsearch/docker-compose.yml`: добавлены env `path.repo=/snapshots` + bind `./snapshots:/snapshots`, ES recreated. Снимки атомарны и Lucene-consistent.
- **Email via `curl --ssl-reqd --url smtps://...:465`** (implicit TLS), не msmtp. CentOS 7 + EOL repos = msmtp install painful; curl native + Yandex SMTP works. Initial attempt с `smtp://...:465` + `--ssl-reqd` дала 30-sec timeout (curl попытался STARTTLS на TLS-only-from-start порту) — исправлено переходом на `smtps://`.
- **RFC 5322 CRLF headers + Date + MIME-Version + Content-Type** в email-body. Yandex отвергает без Date. Initial body без них принимался при STARTTLS теста, но for production safety — full headers.
- **`StrictHostKeyChecking=yes`** (НЕ `accept-new`) — CentOS 7 OpenSSH 7.4 не поддерживает `accept-new` (added in 7.6). kreknin host key pre-populated в `/root/.ssh/known_hosts` на bootstrap step.
- **Retention 7 daily snapshots** (matches VDS + RUVDS pattern). ES snapshot pruning отдельно — via REST DELETE.
- **`ntfy.vds.kzntsv.site/vds-backup` topic** (shared канал с VDS + RUVDS + windows-host). Title `BOOKS-VDS backup OK <date>` отличает от других хостов на phone-side.
- **books-job-scheduler-mongo без auth** — `MONGO_INITDB_ROOT_*` env пустой, `mongodump --archive` без `--uri` достаточен. Shared `mongo` — root auth (urircreds).
- **Cron 06:00 MSK** — после VDS-infra (05:00), RUVDS (04:30), windows-host (05:30). Break перед US-EU business start.
- **`books-vds-portainer/full-env` consolidated** → `books-vds/full-env` (parity с `vds-kzntsv/full-env` one-file-per-host pattern).
## Хронология standup'а (2026-05-25)
- **08:30:** user-указание начать `stateful-split-volume-copy` под supervision. Probe `vds-ops` MCP показал что books-db нет на vds-kzntsv (только generic mariadb/mongo/minio). Wiki `[[../concepts/vds-kzntsv-ssh-access]]` (тогда `books-ssh-access.md`) врала что books живёт на vds-kzntsv.
- **08:45:** user: «89.253.255.133 - books VDS, клиентский. Сделаем backup на kreknin». Pivot фокус.
- **08:48:** SSH probe `id_ed25519_books_ops` → ready, root access OK. Inventory: hybrid Portainer + SSH-compose, hosting books-api/web/scheduler/task-runner/db + shared infra.
- **08:55:** Portainer API token request — user создал `claude-code-automation` PAT в Portainer UI (`portainer.kzntsv.site`). Saved в pass `books-vds-portainer/full-env`. API tested OK. But `elasticsearch`/`mongo`/`minio`/`books-db`/`traefik` нет среди Portainer-managed stacks — they're SSH-compose. User: «после backup'а мигрируем максимум под Portainer».
- **08:58:** ES probe — `path.repo` не настроен, snapshot API не работает без edit'а compose.
- **09:00:** User approval на ES stack edit «сейчас, ES никто не использует». Sed edit `docker-compose.yml` + recreate. docker-compose 1.29 → docker 26 incompat (`KeyError: 'ContainerConfig'`) → manual `docker rm` renamed-stuck → `docker-compose up -d` clean → ES ready in 30s. Snapshot repo `kreknin` registered + test snap SUCCESS → deleted.
- **09:14:** Wrote `scripts/books-vds-backup-daily-kreknin/{run.sh,.env.example,README.md}` локально. Built `.env` from pass entries (books-vds + vds-kzntsv NTFY + snolla-smtp SMTP). Deployed `/opt/stacks/backup/{run.sh,.env}` на VDS via scp. Generated `kreknin-key` ed25519 на VDS. Authorized pubkey on kreknin via workstation `id_ed25519_kreknin`. Pre-populated kreknin host key в `/root/.ssh/known_hosts` (CentOS 7 no accept-new).
- **09:15:** Smoke run start. DB dumps OK (mariadb 270MB, mongo 170KB, scheduler-mongo 2.8MB). ES snapshot SUCCESS. rsync started.
- **09:29:** rsync done, 4.87 GB in **13m28s** (7.6 MB/s). ntfy push отправлен. Email **timeout 30s** — bug в email_send.
- **09:30:** Diagnose. SMTP_PORT=465 + `smtp://` + `--ssl-reqd` = curl попытался STARTTLS на implicit-TLS-only порту. Fix: `smtps://` schema. Plus CRLF headers + Date header. Re-tested: SMTP send OK.
- **09:32:** Updated run.sh re-deployed. Cron installed `/etc/cron.d/books-vds-backup` (0 6 * * * root /opt/stacks/backup/run.sh). crond active.
## Pipeline state
- **Live:** cron `books-vds-backup`, daily 06:00 MSK, root user.
- **Verified:** initial sync 4.87 GB end-to-end + ntfy push отправлен.
- **Phone-side ntfy verify:** **pending user confirmation** (был ли push `BOOKS-VDS backup OK 2026-05-25` на ntfy app в общем канале `vds-backup`).
- **Email verify:** sent после fix (см. 09:32 fix). User должен подтвердить получение `[BOOKS-VDS] backup OK <date>` или standalone fix-test message в inbox.
- **Day-2 verify:** automatic at 06:00 MSK 2026-05-26 — first cron-trigger.
- **Retention prune verify:** automatic at day-8 (когда snapshot count > 7).
## Bugs fixed during smoke
- **SMTP timeout 30s** — `smtp://...:465` + `--ssl-reqd` = curl STARTTLS на implicit-TLS-only порту. Server ждёт TLS handshake, curl ждёт plain EHLO, deadlock. Fix: `smtps://` scheme.
- **email RFC 5322** — initial body had no Date/MIME headers, only From/To/Subject. Yandex отвергает без Date. Fix: добавлены Date, MIME-Version, Content-Type CRLF headers.
- **CentOS 7 OpenSSH 7.4 no accept-new** — `StrictHostKeyChecking=accept-new` parse-error. Fix: `StrictHostKeyChecking=yes` + pre-populated known_hosts.
- **docker-compose 1.29 ↔ docker 26 KeyError ContainerConfig** — recreate failed mid-way, container renamed-stuck. Manual `docker rm -f` + clean `up -d` rescued.
## Open follow-ups (не блокер)
- **Migrate SSH-compose стеки под Portainer** (отдельная таска, user-requested после backup live). Targets: elasticsearch, mongo, minio, books-db, traefik, proxy-chain.
- **SSH-аудит books VDS** — отдельный аудит по pattern [[vds-kzntsv-ssh-access]]. На сейчас знаем только root key `id_ed25519_books_ops` под root — других пользователей не enum'ил. Pre-existing keys unknown.
- **kreknin space audit** — quarterly check `du -sh /volume1/NetBackup/*` чтобы не упереться в 5.7T limit.
- **wd40 backup**: рассмотреть второй backup target (B2 / Glacier) для resilience — см. [[../concepts/future-resilient-architecture-goals]].
## Атомарный revert
```bash
# На books VDS:
ssh root@89.253.255.133 'rm /etc/cron.d/books-vds-backup; rm -rf /opt/stacks/backup /var/log/books-vds-backup'
# Revert ES path.repo edit:
ssh root@89.253.255.133 'cd /usr/docker/elasticsearch && \
cp docker-compose.yml.bak-pre-snapshots-2026-05-25 docker-compose.yml && \
docker rm -f elasticsearch && docker-compose up -d'
# На kreknin (через SSH):
ssh vitya@195.19.90.188 'rm -rf /volume1/NetBackup/books-vds'
# Remove the books-vds-backup-20260525 pubkey line from authorized_keys.
```
## Cross-refs
- Source host: [[../entities/books-vds]]
- Backup target: [[../entities/kreknin-synology]]
- Pattern parent: [[vds-kzntsv-bootstrap-2026-05-20]] §backup (msmtp + rsync --link-dest)
- Sibling pipeline: [[ruvds-backup-daily-kreknin-2026-05-24]] (rclone+SFTP)
- Notification: общий ntfy topic `vds-backup` на `ntfy.vds.kzntsv.site`.
- Misnaming legacy: [[../concepts/vds-kzntsv-ssh-access]] (бывший `books-ssh-access` до 2026-05-25).

34
scripts/books-vds-backup-daily-kreknin/.env.example Normal file
View File

@@ -0,0 +1,34 @@
# /opt/stacks/backup/.env on books VDS — sourced by run.sh
# Live values are NOT in git; sourced from pass entries:
# pass show books-vds/full-env → BOOKS_DB_* + BOOKS_MONGO_*
# pass show vds-kzntsv/full-env → NTFY_*
# pass show snolla-smtp/full-env → SMTP_* + OPS_NOTIFY_EMAIL
# pass show kreknin/full-env → DEST_USER (vitya), DEST_HOST (195.19.90.188)
# === Backup destination ===
DEST_USER=vitya
DEST_HOST=195.19.90.188
DEST_BASE=/volume1/NetBackup/books-vds
RETENTION_DAYS=7
SSH_KEY=/opt/stacks/backup/kreknin-key
# === DB creds (books VDS containers) ===
BOOKS_DB_ROOT_PASSWORD=...
BOOKS_MONGO_USERNAME=root
BOOKS_MONGO_PASSWORD=...
# === Elasticsearch snapshot repo (file-based, registered 2026-05-25) ===
ES_SNAPSHOT_REPO=kreknin
# === NTFY (shared channel — vds-backup topic; ntfy.vds.kzntsv.site) ===
NTFY_URL=https://ntfy.vds.kzntsv.site
NTFY_USER=...
NTFY_PASS=...
# === SMTP (Yandex relay, noreply@snolla.com) ===
SMTP_HOST=smtp.yandex.ru
SMTP_PORT=587
SMTP_USER=noreply@snolla.com
SMTP_PASS=...
SMTP_FROM=noreply@snolla.com
OPS_NOTIFY_EMAIL=...

115
scripts/books-vds-backup-daily-kreknin/README.md Normal file
View File

@@ -0,0 +1,115 @@
# books-vds-backup-daily-kreknin
Daily backup pipeline: **books VDS** (`89.253.255.133` / host4g.ru CentOS 7) → **kreknin Synology** (`195.19.90.188:/volume1/NetBackup/books-vds/<date>/`) via SSH + rsync.
Schedule: `0 6 * * *` Europe/Moscow (after VDS-infra at 05:00, after windows-host at 05:30). Pattern mirrors `scripts/vds-backup-rsync-kreknin/run.sh`.
## What's backed up
| Component | Method | Source |
|---|---|---|
| MariaDB (books-db) | `mariadb-dump --all-databases --single-transaction` | container `books-db` |
| Mongo (shared) | `mongodump --archive` w/ root auth | container `mongo` |
| Mongo (job-scheduler) | `mongodump --archive` no-auth | container `books-job-scheduler-mongo` |
| Elasticsearch | REST snapshot API → fs repo `kreknin` → rsync | container `elasticsearch` (path.repo=/snapshots) |
| MinIO blobs | rsync direct (immutable content-addressed) | `/usr/docker/minio/data` |
| App configs | rsync | `/opt/books/{api,job-scheduler,task-runner,ntfy,books-ops-mcp}` |
| Traefik state | rsync | `/usr/docker/traefik/{letsencrypt,data}` (acme.json + config) |
| System config | rsync | `/etc/{ssh,hosts,cron.d}`, `/root/.ssh` |
Retention: **7 daily snapshots** on kreknin (+ on ES repo). Older — pruned at end of each run.
## Notifications
Dual-channel after every run:
- **ntfy** → topic `vds-backup` on `ntfy.vds.kzntsv.site` (shared с VDS + RUVDS + windows-host backups).
- Success: title `BOOKS-VDS backup OK <date>`, tag `green_circle`.
- Fail: title `BOOKS-VDS backup FAILED <date>`, tag `red_circle`, priority `high`.
- **Email** via Yandex SMTP (`noreply@snolla.com` → `OPS_NOTIFY_EMAIL`).
- Sent via `curl --ssl-reqd --url smtp://...:587` (no msmtp dep on CentOS 7).
- Subject: `[BOOKS-VDS] backup OK <date>` / `[BOOKS-VDS] backup FAILED <date>`.
If notify-channels fail — backup itself does NOT fail (`|| true`).
## Files
- `run.sh` — backup logic. Synced to `/opt/stacks/backup/run.sh` on books VDS.
- `.env.example` — placeholder. Live `.env` built from `pass show` entries during install.
## Initial install (rebuild from scratch)
```bash
# From workstation, with id_ed25519_books_ops authorized:
ssh root@89.253.255.133 'mkdir -p /opt/stacks/backup'
scp scripts/books-vds-backup-daily-kreknin/run.sh root@89.253.255.133:/opt/stacks/backup/
# Build .env locally from pass entries:
{
echo "DEST_USER=vitya"
echo "DEST_HOST=195.19.90.188"
echo "DEST_BASE=/volume1/NetBackup/books-vds"
echo "RETENTION_DAYS=7"
echo "SSH_KEY=/opt/stacks/backup/kreknin-key"
echo "ES_SNAPSHOT_REPO=kreknin"
pass show books-vds/full-env | grep -E '^(BOOKS_DB_ROOT_PASSWORD|BOOKS_MONGO_)'
pass show vds-kzntsv/full-env | grep -E '^NTFY_'
pass show snolla-smtp/full-env | grep -E '^(SMTP_|OPS_NOTIFY_EMAIL)'
} | scp /dev/stdin root@89.253.255.133:/opt/stacks/backup/.env
ssh root@89.253.255.133 'chmod 600 /opt/stacks/backup/.env && chmod 755 /opt/stacks/backup/run.sh'
# Generate kreknin SSH key on books VDS:
ssh root@89.253.255.133 'ssh-keygen -t ed25519 -f /opt/stacks/backup/kreknin-key -N "" -C "books-vds-backup"'
# Get pubkey and authorize it on kreknin (workstation pivots — books VDS has no kreknin SSH yet):
PUBKEY=$(ssh root@89.253.255.133 cat /opt/stacks/backup/kreknin-key.pub)
ssh -i ~/.ssh/id_ed25519_kreknin vitya@195.19.90.188 \
"echo '$PUBKEY' >> ~/.ssh/authorized_keys && mkdir -p /volume1/NetBackup/books-vds"
# Verify SSH books → kreknin
ssh root@89.253.255.133 'ssh -i /opt/stacks/backup/kreknin-key -o StrictHostKeyChecking=accept-new vitya@195.19.90.188 hostname'
# Smoke run (manual)
ssh root@89.253.255.133 '/opt/stacks/backup/run.sh'
# Install cron
ssh root@89.253.255.133 'echo "0 6 * * * root /opt/stacks/backup/run.sh" > /etc/cron.d/books-vds-backup && chmod 644 /etc/cron.d/books-vds-backup'
```
## Decisions log
- **ES snapshot via REST API**, not data-dir rsync. Filesystem snapshot repo (`type=fs`, `location=/snapshots`, `compress=true`) registered as `kreknin` on 2026-05-25. Required one-time stack edit (add `path.repo=/snapshots` env + `./snapshots:/snapshots` bind in `/usr/docker/elasticsearch/docker-compose.yml`).
- **`curl --ssl-reqd` SMTP** instead of msmtp — avoid CentOS 7 msmtp install (EOL repo issues). Native `curl` works on STARTTLS port 587.
- **No `tasks.yml`-style auth on `books-job-scheduler-mongo`** — container has no `MONGO_INITDB_ROOT_*` env. `mongodump --archive` без `--uri` достаточен.
- **rsync /opt/books + /usr/docker/{minio,elasticsearch/snapshots,traefik}** — bind paths из inspect 2026-05-25; не `/var/lib/docker/volumes/` (named volumes для контейнеров пустые, кроме portainer/scheduler-mongo).
- **`books-vds-portainer/full-env` consolidated** into `books-vds/full-env` for one-file-per-host pattern parity with `vds-kzntsv/full-env`.
- **No `BackupConfig` snapshot/git pre-commit-like step** — VDS-infra `run.sh` тоже не делает, и нет аналогичной системы на books VDS.
## Smoke run (manual)
```bash
ssh root@89.253.255.133 '/opt/stacks/backup/run.sh'
# Watch log:
ssh root@89.253.255.133 'tail -F /var/log/books-vds-backup/$(date +%Y-%m-%d).log'
```
Verify on kreknin:
```bash
ssh vitya@195.19.90.188 'du -sh /volume1/NetBackup/books-vds/*/'
```
## Atomic revert (uninstall)
```bash
# On books VDS:
ssh root@89.253.255.133 'rm /etc/cron.d/books-vds-backup; rm -rf /opt/stacks/backup /var/log/books-vds-backup'
# On kreknin (via SSH):
ssh vitya@195.19.90.188 'rm -rf /volume1/NetBackup/books-vds'
# Remove the books-vds pubkey line from authorized_keys (one line with 'books-vds-backup' comment).
# On books VDS — revert ES path.repo change:
ssh root@89.253.255.133 'cd /usr/docker/elasticsearch && \
cp docker-compose.yml.bak-pre-snapshots-2026-05-25 docker-compose.yml && \
docker rm -f elasticsearch && docker-compose up -d'
```

216
scripts/books-vds-backup-daily-kreknin/run.sh Normal file
View File

@@ -0,0 +1,216 @@
#!/bin/bash
# Daily books VDS → kreknin backup. Triggered by /etc/cron.d/books-vds-backup at 06:00 MSK.
# Pattern: mirror VDS scripts/vds-backup-rsync-kreknin/run.sh, adapted for books VDS hybrid stack.
set -euo pipefail
ENV_FILE=/opt/stacks/backup/.env
# shellcheck disable=SC1090
source "$ENV_FILE"
TODAY=$(date +%Y-%m-%d)
LOG_DIR=/var/log/books-vds-backup
LOG="$LOG_DIR/$TODAY.log"
LOCK=/tmp/books-vds-backup.lock
DUMP_DIR=/tmp/books-vds-backup-dumps/$TODAY
DEST_PATH="$DEST_BASE/$TODAY"
LATEST="$DEST_BASE/latest"
mkdir -p "$LOG_DIR" "$DUMP_DIR"
exec >> "$LOG" 2>&1
echo "===== BOOKS-VDS backup start $TODAY $(date -Is) ====="
# single-instance lock
exec 9>"$LOCK"
if ! flock -n 9; then echo "already running, exiting"; exit 0; fi
START=$(date +%s)
SSH_OPTS="-i $SSH_KEY -o StrictHostKeyChecking=yes -o ConnectTimeout=30 -o LogLevel=ERROR"
# Note: kreknin host key must be pre-populated in /root/.ssh/known_hosts (CentOS 7 OpenSSH 7.4 has no accept-new option)
HOST_LABEL=BOOKS-VDS
SOURCE_DISPLAY="89.253.255.133 (books VDS)"
notify() {
local title="$1" msg="$2" tags="${3:-}" prio="${4:-default}"
curl -sS --max-time 10 -u "$NTFY_USER:$NTFY_PASS" \
-H "Title: $title" -H "Tags: $tags" -H "Priority: $prio" \
-d "$msg" "$NTFY_URL/vds-backup" > /dev/null || true
}
email_send() {
local subj="$1" body="$2"
local tmpf
tmpf=$(mktemp)
# RFC 5322: CRLF headers + Date required by Yandex
{
printf 'From: %s\r\n' "$SMTP_FROM"
printf 'To: %s\r\n' "$OPS_NOTIFY_EMAIL"
printf 'Subject: %s\r\n' "$subj"
printf 'Date: %s\r\n' "$(date -R)"
printf 'MIME-Version: 1.0\r\n'
printf 'Content-Type: text/plain; charset=UTF-8\r\n'
printf '\r\n'
echo "$body" | sed 's/$/\r/'
} > "$tmpf"
# smtps:// (implicit TLS) for Yandex port 465; smtp:// (STARTTLS) would be port 587.
# SMTP_PORT in pass is 465 — use smtps:// scheme.
curl -sS --max-time 30 \
--ssl-reqd \
--url "smtps://$SMTP_HOST:$SMTP_PORT" \
--user "$SMTP_USER:$SMTP_PASS" \
--mail-from "$SMTP_FROM" \
--mail-rcpt "$OPS_NOTIFY_EMAIL" \
--upload-file "$tmpf" || true
rm -f "$tmpf"
}
fmt_duration() {
local s=$1
printf "%dm%02ds" $((s/60)) $((s%60))
}
on_error() {
local line=$1 rc=$2
local err="line $line, exit $rc"
local now
now=$(date +%s)
local dur=$((now - ${START:-now}))
local dur_h
dur_h=$(fmt_duration "$dur")
local tail_log
tail_log=$(tail -40 "$LOG" 2>/dev/null || true)
echo "FATAL: $HOST_LABEL backup FAILED — $err"
notify "$HOST_LABEL backup FAILED $TODAY" \
"After $dur_h: $err. See $LOG" \
red_circle high
email_send "[$HOST_LABEL] backup FAILED $TODAY" \
"$HOST_LABEL daily backup FAILED.
Date: $TODAY
Duration: $dur_h
Error: $err
Source: $SOURCE_DISPLAY
Log: $LOG
Tail (last 40 lines):
$tail_log"
exit "$rc"
}
trap 'on_error $LINENO $?' ERR
# === Step 1: DB dumps ===
echo "--- DB dumps to $DUMP_DIR ---"
docker exec books-db mariadb-dump -uroot -p"$BOOKS_DB_ROOT_PASSWORD" \
--all-databases --single-transaction --quick 2>/dev/null \
| gzip > "$DUMP_DIR/mariadb.sql.gz"
echo "mariadb: $(stat -c %s "$DUMP_DIR/mariadb.sql.gz") bytes"
docker exec mongo mongodump --archive --quiet \
--uri="mongodb://$BOOKS_MONGO_USERNAME:$BOOKS_MONGO_PASSWORD@localhost:27017/?authSource=admin" 2>/dev/null \
| gzip > "$DUMP_DIR/mongo.archive.gz"
echo "mongo (shared): $(stat -c %s "$DUMP_DIR/mongo.archive.gz") bytes"
docker exec books-job-scheduler-mongo mongodump --archive --quiet 2>/dev/null \
| gzip > "$DUMP_DIR/job-scheduler-mongo.archive.gz"
echo "job-scheduler-mongo: $(stat -c %s "$DUMP_DIR/job-scheduler-mongo.archive.gz") bytes"
# === Step 2: ES snapshot via REST ===
echo "--- ES snapshot daily-$TODAY in repo $ES_SNAPSHOT_REPO ---"
ES_SNAP="daily-$TODAY"
SNAP_RESULT=$(docker exec elasticsearch curl -sS -X PUT \
"http://localhost:9200/_snapshot/$ES_SNAPSHOT_REPO/$ES_SNAP?wait_for_completion=true" \
-H 'Content-Type: application/json')
echo "ES snapshot result: $SNAP_RESULT"
if ! echo "$SNAP_RESULT" | grep -q '"state":"SUCCESS"'; then
echo "ES snapshot did NOT report SUCCESS state — failing"
exit 3
fi
echo "--- prune ES snapshots older than $RETENTION_DAYS ---"
# Pure bash parse — no jq/python dependency
ALL_SNAPS=$(docker exec elasticsearch curl -sS "http://localhost:9200/_snapshot/$ES_SNAPSHOT_REPO/_all" \
| grep -oE '"snapshot":"daily-[0-9-]+"' \
| sed -E 's/.*"(daily-[0-9-]+)".*/\1/' \
| sort)
SNAP_COUNT=$(echo "$ALL_SNAPS" | wc -l)
PRUNE_COUNT=$((SNAP_COUNT - RETENTION_DAYS))
if [ $PRUNE_COUNT -gt 0 ]; then
echo "$ALL_SNAPS" | head -n "$PRUNE_COUNT" | while read -r snap; do
[ -z "$snap" ] && continue
echo "deleting ES snapshot: $snap"
docker exec elasticsearch curl -sS -X DELETE \
"http://localhost:9200/_snapshot/$ES_SNAPSHOT_REPO/$snap" || true
echo
done
fi
# === Step 3: rsync to kreknin ===
echo "--- rsync to $DEST_USER@$DEST_HOST:$DEST_PATH ---"
ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "mkdir -p '$DEST_PATH'"
LINK_DEST_ARG=""
if ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "test -L '$LATEST' || test -d '$LATEST'" 2>/dev/null; then
LINK_DEST_ARG="--link-dest=$LATEST"
fi
rsync -aHh --info=stats2 --delete $LINK_DEST_ARG \
-e "ssh $SSH_OPTS" \
/opt/books \
/usr/docker/elasticsearch/snapshots \
/usr/docker/minio/data \
/usr/docker/traefik/letsencrypt \
/usr/docker/traefik/data \
/etc/ssh \
/etc/hosts \
/etc/cron.d \
/root/.ssh \
"$DUMP_DIR" \
"$DEST_USER@$DEST_HOST:$DEST_PATH/" || true
ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "rm -f '$LATEST' && ln -s '$DEST_PATH' '$LATEST'"
# === Step 4: retention prune on kreknin ===
echo "--- prune snapshots older than $RETENTION_DAYS days on kreknin ---"
ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" \
"cd '$DEST_BASE' && ls -1d 20*-*-* 2>/dev/null | sort | head -n -$RETENTION_DAYS | xargs -r rm -rf"
# === Step 5: local cleanup ===
rm -rf "$DUMP_DIR"
# === Step 6: report ===
END=$(date +%s)
DURATION=$((END - START))
SIZE=$(ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "du -sh '$DEST_PATH' 2>/dev/null | cut -f1" || echo "?")
SNAPSHOT_COUNT=$(ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "ls -1d $DEST_BASE/20*-*-* 2>/dev/null | wc -l" || echo "?")
DURATION_HUMAN=$(fmt_duration "$DURATION")
NTFY_BODY="$DURATION_HUMAN, size=$SIZE, snapshots=$SNAPSHOT_COUNT, dest=kreknin:$DEST_PATH"
notify "$HOST_LABEL backup OK $TODAY" "$NTFY_BODY" green_circle
EMAIL_BODY="$HOST_LABEL daily backup completed successfully.
Date: $TODAY
Duration: $DURATION_HUMAN
Size: $SIZE
Snapshots: $SNAPSHOT_COUNT
Source: $SOURCE_DISPLAY
Dest: kreknin:$DEST_PATH
Components:
- /opt/books/{api,job-scheduler,task-runner,ntfy,books-ops-mcp}
- /usr/docker/{minio/data, elasticsearch/snapshots, traefik/{letsencrypt,data}}
- /etc/{ssh,hosts,cron.d}
- /root/.ssh
- DB dumps: books-db (mariadb), mongo (shared), books-job-scheduler-mongo
- ES snapshot: $ES_SNAP
Log: $LOG"
email_send "[$HOST_LABEL] backup OK $TODAY" "$EMAIL_BODY"
echo "$HOST_LABEL backup OK: $NTFY_BODY"
echo "===== $HOST_LABEL backup done $(date -Is) ====="