backup(books-vds): daily 06:00 MSK pipeline to kreknin live 🟢

- scripts/books-vds-backup-daily-kreknin/ — run.sh + .env.example + README
- ES path.repo bootstrap (one-time stack edit, snapshot repo 'kreknin' registered)
- curl smtps://yandex:465 email (CRLF + Date headers, no msmtp dep on CentOS 7)
- StrictHostKeyChecking=yes + pre-populated known_hosts (CentOS 7 no accept-new)
- cron /etc/cron.d/books-vds-backup live, initial sync 4.87 GB in 13m28s, ntfy push sent
- wiki: entities/books-vds.md + sources/books-vds-backup-daily-kreknin-2026-05-25.md created
- wiki: concepts/books-ssh-access → vds-kzntsv-ssh-access (was lying about books on vds-kzntsv)
- .tasks/books-vds-backup-daily-kreknin.md closed 🟢 pending phone-side verify

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-25 09:37:16 +03:00
parent 1eefb4d45a
commit fe41ee4422
10 changed files with 694 additions and 35 deletions

View File

@@ -1,42 +1,50 @@
---
_last_updated_: 2026-05-25T07:50:00+03:00
session_id: 2026-05-25-close-unify-backup-notifications
_last_updated_: 2026-05-25T09:35:00+03:00
session_id: 2026-05-25-books-vds-backup-standup
---
# Next session handoff
## Recent commits
- `tasks(unify-backup-notifications): close 🟢 — unified push+email across VDS/RUVDS/windows-host` (этот раунд)
- `tasks: rm obsolete NEXT-SESSION-PROMPT.md (leaked creds in history since 2026-05-21)` (этот раунд)
- `scripts(backup-notifications): unify push + email format across VDS/RUVDS/windows-host` (`73ad6dd0`)
- `wiki(claude): add session-handoff trigger line`
- `wiki(ingest): RUVDS IIS migration + daily backup pipeline`
- `tasks(books-vds-backup-daily-kreknin): close 🟢 — daily 06:00 MSK pipeline live, 4.87GB initial sync` (этот раунд, planned)
- `wiki(books-vds): create entity + source standup + rename misnamed concept books-ssh-access → vds-kzntsv-ssh-access` (этот раунд, planned)
- `scripts(books-vds-backup-daily-kreknin): run.sh + .env.example + README` (этот раунд, planned)
- `tasks: add stateful-split-volume-copy (moved from victor/books)` (`1eefb4d4`)
- `tasks(unify-backup-notifications): close 🟢` (`838f51da`)
## Open треки
| Трек | Готовность | Entry-point |
|---|---|---|
| `iis-migration-to-ruvds` 🟡 | soak (kupimknigi + emspb DNS flipped 2026-05-24, оставшиеся 22 hostnames в очереди, tandemmebel.ru excluded) | `.tasks/iis-migration-to-ruvds.md` |
| `infra-inventory` ⚪ | ready, не начат | `.tasks/infra-inventory.md` |
| `books-bookva-user-whitelist-gathering` | ready, не начат — long-lead gathering | `.tasks/books-bookva-user-whitelist-gathering.md` |
| `books-vds-bookva-bootstrap` 🔵 | blocked on victor/books tenant-split Phase 4 | `.tasks/books-vds-bookva-bootstrap.md` |
| `books-vds-backup-daily-kreknin` 🟢 | closed — **phone-side ntfy + email verify pending user** | `.tasks/books-vds-backup-daily-kreknin.md` |
| `stateful-split-volume-copy` ⚪ | not started; **playbook references `books-db` на books VDS (89.253.255.133), не на vds-kzntsv** | `.tasks/stateful-split-volume-copy.md` |
| `iis-migration-to-ruvds` 🟡 | soak (kupimknigi + emspb DNS flipped 2026-05-24, 22 pending) | `.tasks/iis-migration-to-ruvds.md` |
| `infra-inventory` | ready, не начат — **books VDS должен попасть в admin-detailed runbook** | `.tasks/infra-inventory.md` |
| `books-bookva-user-whitelist-gathering` ⚪ | ready, long-lead gathering | `.tasks/books-bookva-user-whitelist-gathering.md` |
| `books-vds-bookva-bootstrap` 🔵 | blocked on tenant-split Phase 4 | `.tasks/books-vds-bookva-bootstrap.md` |
| `books-dns-cutover-bookva` 🔵 | blocked on books-vds-bookva-bootstrap | `.tasks/books-dns-cutover-bookva.md` |
## Спроси user'а
- **windows-host deploy сделан?** Если `Get-FileHash C:\ProgramData\backup\run.ps1` matches `scripts/windows-host-fallback-backup-daily/run.ps1` (elevated check) — deploy ок, и сегодня ночью 03:00 MSK пушнёт в едином формате. Если нет — старый формат на одну ночь.
- **На night-run 2026-05-25 → 26 push'и + emails пришли все 3?** Title pattern `<HOST> backup OK <date>` / subject `[<HOST>] backup OK <date>`. Если что-то отвалилось — `.bak-pre-unify` файлы лежат на каждом хосте для quick rollback.
- **DNS swap для 22 hostnames** (`iis-migration-to-ruvds` next-action #3) — продолжаем или ещё soak?
- **Plaintext creds в git history** (NEXT-SESSION-PROMPT.md): user сегодня acceptуnул risk без rotation. Если позже передумаем — rotate MSSQL SA + snolla DB user + stayer DB user + VM admin (`Pryakhin9`).
- **ntfy push** `BOOKS-VDS backup OK 2026-05-25` (tag green_circle, channel `vds-backup`) — пришёл на phone? Если нет — diagnose curl ntfy creds или phone subscription к каналу.
- **Email**: пришёл ли `[BOOKS-VDS] backup OK 2026-05-25` + standalone `[BOOKS-VDS] SMTP fix verify` (тестовый, отправлен после fix'а)? Inbox `<OPS_NOTIFY_EMAIL>` от noreply@snolla.com.
- **Migrate SSH-compose стеки под Portainer** (user-promised after backup live): ES, mongo, minio, books-db, traefik, proxy-chain currently SSH-compose at `/usr/docker/<svc>/`. Делаем? Если да — это отдельная таска (`books-vds-stacks-to-portainer`).
- **`stateful-split-volume-copy`** — playbook писан под прошлое assumption «books на vds-kzntsv». Real books VDS = 89.253.255.133, его `books-db` (не `mariadb`) — playbook fits OK, но container names в task-файле должны быть verify'нуты. Continue эту таску в next session?
## Не делать (preemptive guards)
- **Не push без grant'а** (project-discipline Rule 4 — session-start = ask-mode). Каждая сессия начинает спрашивать заново.
- **Не trigger полный backup-run для smoke** (21-81 min wall-clock). Smoke = только notify-channel через `scripts/<slug>/smoke-notify.ps1`.
- **Не history-rewrite + force-push** для cleanup creds в `NEXT-SESSION-PROMPT.md` без явного user-approve — user уже принял risk на ad-hoc rotation, не на destructive history operations.
- **Не fold `traefik`/`portainer` управление через SSH compose** на VDS — Portainer-managed per `.wiki/concepts/portainer-stack-management-vds.md`, кроме `traefik`+`portainer` themselves.
- **Не push без grant'а** (project-discipline Rule 4). Session начинает ask-mode.
- **Не trust wiki blindly** — `books-ssh-access.md` (теперь `vds-kzntsv-ssh-access.md`) до 2026-05-25 врала что books на vds-kzntsv. Faktcheck через `vds-ops` + SSH probe + IP до построения предположений. Это уже applied в этом раунде; не забыть в будущих сессиях.
- **Не trigger полный backup для smoke** на books VDS — 13m+ wall-clock, 4.87 GB initial. Smoke = только notify-channel тест через standalone curl call.
- **Не touch ES прод стек без user-approve** — ES сейчас не используется (per user), но статус может measure. Если включат — recreate downtime неacceptable без window.
- **Не editit `/usr/docker/<svc>/docker-compose.yml` напрямую** для production stacks — preferred path: migrate под Portainer, потом edit там. Текущий backup-script патчил ES — это был bootstrap, не daily ops.
## Memory updates за сессию
- (нет на этом раунде — все правила/паттерны уже зафиксированы в project CLAUDE.md + global skills)
- **Учить:** `books VDS ≠ vds-kzntsv`. Wiki была ошибочно misnamed. Real books VDS = 89.253.255.133 (host4g.ru, CentOS 7). Documented в [[../wiki/entities/books-vds]].
- **Учить:** `pass entries one-file-per-host pattern` — consolidate сервис-specific creds в `<host>/full-env` с section'ами (matches `vds-kzntsv/full-env` style). Не плодить `<host>-<service>/full-env`.
- **Учить:** Yandex SMTP — port 465 = `smtps://` (implicit TLS), port 587 = `smtp://` + `--ssl-reqd` (STARTTLS). Не путать; mismatch = 30-sec curl timeout.
- **Учить:** CentOS 7 OpenSSH 7.4 не имеет `accept-new` (added 7.6). Bootstrap step: `ssh-keyscan` host-key в `known_hosts`, потом `StrictHostKeyChecking=yes`.
- **Учить:** docker-compose 1.29 + Docker 26 = `KeyError: 'ContainerConfig'` при recreate. Workaround: `docker rm -f <renamed-stuck>` + `docker-compose up -d`. Bigger fix: upgrade docker-compose to v2 plugin.
- **Reference:** Portainer на books VDS = `https://portainer.kzntsv.site` (НЕ `portainer.vds.kzntsv.site` — тот для infra). PAT saved в `pass show books-vds/full-env BOOKS_PORTAINER_API_KEY`.

View File

@@ -1,5 +1,7 @@
# Admin Task Board
_Updated: 2026-05-25 (`unify-backup-notifications` 🟢 closed — 3 scripts под единый push+email format committed `73ad6dd0`, VDS+RUVDS deployed+smoke ✓, windows-host self-deploy на user'е через `scripts/windows-host-fallback-backup-daily/deploy.ps1` elevated. Также удалён obsolete `.tasks/NEXT-SESSION-PROMPT.md` (содержал plaintext creds в git history с 2026-05-21; user accept risk без rotation/rewrite). VDS `run.sh` теперь в repo как canonical source. Pre-existing: все 3 daily backups live на kreknin.)_
_Updated: 2026-05-25 (`books-vds-backup-daily-kreknin` 🟢 closed — 4-я daily backup pipeline на kreknin, теперь покрыта `books VDS` 89.253.255.133 (host4g.ru). Pattern mirror VDS-infra (rsync+--link-dest), ES snapshot via REST API (новый repo `kreknin`, потребовал one-time stack edit + recreate ES), email через `curl smtps://yandex:465` без msmtp. Initial sync 4.87 GB in 13m28s, cron 06:00 MSK live. Также fix вики: `concepts/books-ssh-access.md``vds-kzntsv-ssh-access.md` (был misnamed — врал что books живёт на vds-kzntsv), `entities/books-vds.md` создан с factsheet'ом. **Phone-side ntfy + email — pending user confirmation на следующем раунде.**)_
## 🟢 [books-vds-backup-daily-kreknin] — closed 2026-05-25 — daily backup books VDS (89.253.255.133, host4g.ru, CentOS 7) → kreknin в 06:00 MSK. DB dumps (mariadb/mongo×2) + ES snapshot via REST → rsync 4.87GB → ntfy/email. ES path.repo bootstrap + snapshot repo `kreknin` registered. См. [books-vds-backup-daily-kreknin.md](books-vds-backup-daily-kreknin.md).
## 🟢 [unify-backup-notifications] — closed 2026-05-25 — единый формат push + email для VDS/RUVDS/windows-host (3 scripts), VDS run.sh импортирован в repo. См. [unify-backup-notifications.md](unify-backup-notifications.md) § Closed. windows-host self-deploy остаётся на user'е (elevated PS).

View File

@@ -0,0 +1,61 @@
# books-vds-backup-daily-kreknin
Ежедневный backup pipeline books VDS (`89.253.255.133`, host4g.ru, CentOS 7) → kreknin Synology (`195.19.90.188:/volume1/NetBackup/books-vds/<date>/`) в 06:00 MSK.
## Goal
Восстановимость books-стека после полного отказа VDS: DB dumps + MinIO blobs + ES snapshot + app configs + traefik state + system config на NAS с 7-day retention.
## Key files
- `scripts/books-vds-backup-daily-kreknin/run.sh` — backup logic; deploy → `/opt/stacks/backup/run.sh` на VDS.
- `scripts/books-vds-backup-daily-kreknin/.env.example` — placeholder; live `.env` built from `pass show`.
- `scripts/books-vds-backup-daily-kreknin/README.md` — install + revert recipe.
- `.wiki/entities/books-vds.md` — host entity page.
- `.wiki/sources/books-vds-backup-daily-kreknin-2026-05-25.md` — standup chronology (создаётся при closure).
## Decisions log
- 2026-05-25: pattern mirror `scripts/vds-backup-rsync-kreknin/run.sh` (VDS-infra). Same `vds-backup` ntfy topic, same SMTP relay (`snolla-smtp` Yandex), same `--link-dest` rsync hardlink pattern, same `RETENTION_DAYS=7`.
- 2026-05-25: **ES snapshot via REST API**, not data-dir rsync. One-time stack edit `/usr/docker/elasticsearch/docker-compose.yml`: added env `path.repo=/snapshots` + bind `./snapshots:/snapshots`. Registered repo `kreknin` (type=fs, location=/snapshots, compress=true).
- 2026-05-25: **`curl --ssl-reqd` SMTP** instead of msmtp (CentOS 7 EOL repo issue). Yandex 587 STARTTLS works через native curl без msmtprc setup.
- 2026-05-25: `StrictHostKeyChecking=yes` (NOT `accept-new`) — CentOS 7 OpenSSH 7.4 не поддерживает `accept-new`. kreknin host key pre-populated в `/root/.ssh/known_hosts` на VDS bootstrap.
- 2026-05-25: `books-job-scheduler-mongo`**без auth** (нет `MONGO_INITDB_ROOT_*` env). `mongodump --archive` без `--uri` достаточен. shared `mongo` — root auth (uri credential).
- 2026-05-25: **Cron time 06:00** MSK — после VDS-infra (05:00) + windows-host (05:30), даёт break перед US-EU business start.
- 2026-05-25: `BOOKS-VDS` host label в ntfy title — отличать от обычного `VDS` (infra) на shared `vds-backup` topic.
- 2026-05-25: Consolidated `books-vds-portainer/full-env``books-vds/full-env` (one-file-per-host pattern, parity с vds-kzntsv).
## Open questions
- [ ] (resolved 2026-05-25) `path.repo` на ES — required one-time stack edit. Done, recreated container, snapshot test SUCCESS.
- [ ] Migrate SSH-compose stacks (ES/mongo/minio/books-db/traefik) под Portainer — **отдельная задача после backup pipeline live**.
- [ ] kreknin space check — current free 5.7T (per `kreknin-synology.md` 2026-05-19); per-day footprint TBD после initial sync. Quarterly disk audit на NAS.
## Completed steps
- [x] Phase 0: ES path.repo bootstrap + snapshot repo `kreknin` registered + test snap SUCCESS
- [x] Phase 1: написал `run.sh` + `.env.example` + `README.md` локально
- [x] Phase 2: deployed `/opt/stacks/backup/{run.sh,.env}`, chmod 600 .env / 755 run.sh
- [x] Phase 3: generated `/opt/stacks/backup/kreknin-key` ed25519 на books VDS
- [x] Phase 4: pubkey deployed на kreknin `~vitya/.ssh/authorized_keys`, dest `/volume1/NetBackup/books-vds/` created
- [x] Phase 5: kreknin host key pre-populated `/root/.ssh/known_hosts`, SSH path verified
- [x] Phase 6: smoke run (initial sync) — 4.87GB in 13m28s, ntfy push sent, **email send pending fix → fixed → re-tested OK**
- [x] Phase 6a: SMTP fix — `smtp://` + `--ssl-reqd` дала 30-сек timeout (curl попытался STARTTLS на implicit-TLS-only порту 465). Fix `smtps://` schema + CRLF headers + Date header. Re-tested standalone — SMTP send OK.
- [x] Phase 7: installed `/etc/cron.d/books-vds-backup` (`0 6 * * * root /opt/stacks/backup/run.sh`); crond active
- [x] Phase 8: wiki — `entities/books-vds.md` created; `concepts/books-ssh-access.md` renamed → `vds-kzntsv-ssh-access.md` (was lying about books); `sources/books-vds-backup-daily-kreknin-2026-05-25.md` created; `index.md` updated for all three
- [ ] Phase 9: commit (no push without grant)
- [ ] **PENDING USER VERIFY:** ntfy push `BOOKS-VDS backup OK 2026-05-25` received on phone? Email `[BOOKS-VDS] backup OK 2026-05-25` + standalone fix-test message in inbox?
## Notes
- root password `Pryakhin9~` — сохранён в `pass show books-vds/full-env BOOKS_VDS_ROOT_PASS`. SSH key `id_ed25519_books_ops` уже был authorized (pre-existing bootstrap), password держим как fallback.
- Pre-existing `id_ed25519_books_ops` на workstation предполагает что прошлый bootstrap забыт; не повторяем generate.
- ES в production books currently **не используется** (per user statement) — recreate-downtime 30с не impact'нул.
- `vds-kzntsv``books-vds`. Wiki [[../concepts/books-ssh-access]] до 2026-05-25 ошибочно описывала vds-kzntsv как books shared VDS — fix в Phase 8.
## Cross-refs
- [[../wiki/entities/books-vds]] — host entity.
- [[../wiki/entities/kreknin-synology]] — backup target.
- [[../scripts/vds-backup-rsync-kreknin/run.sh]] — pattern source (VDS-infra backup).
- [[../scripts/ruvds-backup-daily-kreknin/README.md]] — pattern source (RUVDS dual-channel notify).