scripts(books-vds-portainer-migration): Phase 1 adapter ready
Clone-adapted from .wiki/concepts/portainer-stack-management-vds.md § Migration script. Diffs from VDS-infra version: - PORTAINER_URL = portainer.kzntsv.site - Auth via X-API-Key (PAT works, no JWT fallback needed) - DIR prefix /usr/docker/<stack> (not /opt/stacks/<stack>) - Down step via `docker rm -f` by compose-project label (bypasses docker-compose v1 ContainerConfig bug entirely; no compose binary on path required) - Refuses traefik/portainer migration (management plane) Idempotent: delete-then-create against existing stack by name on endpoint 1. Re-runs are safe. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,50 +1,53 @@
|
||||
---
|
||||
_last_updated_: 2026-05-25T09:42:00+03:00
|
||||
session_id: 2026-05-25-books-vds-backup-standup
|
||||
_last_updated_: 2026-05-25T11:15:00+03:00
|
||||
session_id: 2026-05-25-books-vds-stacks-to-portainer
|
||||
---
|
||||
|
||||
# Next session handoff
|
||||
|
||||
## Recent commits
|
||||
|
||||
- `tasks: add ⚪ books-vds-stacks-to-portainer` (`439da3ce`) — follow-up taskcreated
|
||||
- `backup(books-vds): daily 06:00 MSK pipeline to kreknin live 🟢` (`fe41ee44`) — 10 files, 694 insertions
|
||||
- `tasks: add stateful-split-volume-copy (moved from victor/books)` (`1eefb4d4`)
|
||||
- `meta(tasks): update [books-vds-bookva-bootstrap]` (`2931a432`)
|
||||
- `tasks(unify-backup-notifications): close 🟢` (`838f51da`)
|
||||
- `tasks(books-vds-stacks-to-portainer): Phase 0 probe complete, +imgproxy scope` (`bfb942cd`) — Phase 0 done, scope expanded 5→6 stacks.
|
||||
- `tasks(NEXT_SESSION): refresh handoff after this session's commits` (`3769e836`) — prior session handoff.
|
||||
- `tasks: add ⚪ books-vds-stacks-to-portainer` (`439da3ce`)
|
||||
- `backup(books-vds): daily 06:00 MSK pipeline to kreknin live 🟢` (`fe41ee44`)
|
||||
|
||||
## Open треки
|
||||
|
||||
| Трек | Готовность | Entry-point |
|
||||
|---|---|---|
|
||||
| `books-vds-stacks-to-portainer` ⚪ | ready, не начат — promised by user после backup 🟢 | `.tasks/books-vds-stacks-to-portainer.md` |
|
||||
| `books-vds-stacks-to-portainer` 🔴 | **Phase 0 done, Phase 1 next** — write adapter script `scripts/books-vds-portainer-migrate.sh` | `.tasks/books-vds-stacks-to-portainer.md` |
|
||||
| `stateful-split-volume-copy` ⚪ | not started; playbook fits real books VDS (89.253.255.133) | `.tasks/stateful-split-volume-copy.md` |
|
||||
| `iis-migration-to-ruvds` 🟡 | soak (kupimknigi + emspb DNS flipped 2026-05-24, 22 pending) | `.tasks/iis-migration-to-ruvds.md` |
|
||||
| `infra-inventory` ⚪ | ready, не начат — **books VDS должен попасть в admin-detailed runbook** | `.tasks/infra-inventory.md` |
|
||||
| `infra-inventory` ⚪ | ready, не начат — books VDS должен попасть в admin-detailed runbook | `.tasks/infra-inventory.md` |
|
||||
| `books-bookva-user-whitelist-gathering` ⚪ | ready, long-lead gathering | `.tasks/books-bookva-user-whitelist-gathering.md` |
|
||||
| `books-vds-bookva-bootstrap` 🔵 | blocked on tenant-split Phase 4 | `.tasks/books-vds-bookva-bootstrap.md` |
|
||||
| `books-dns-cutover-bookva` 🔵 | blocked on books-vds-bookva-bootstrap | `.tasks/books-dns-cutover-bookva.md` |
|
||||
|
||||
## Mid-task capture
|
||||
|
||||
**left mid-task:** `books-vds-stacks-to-portainer` (Phase 1 entry)
|
||||
|
||||
**where_stopped:** Phase 0 closed — все 6 compose probed (`books-db`, `mongo`, `minio`, `elasticsearch`, `proxy-chain`, `imgproxy`), Portainer PAT verified, endpoint 6 stostayer = not ours.
|
||||
|
||||
**next concrete:** Write `scripts/books-vds-portainer-migrate.sh` (clone-adapt из `.wiki/concepts/portainer-stack-management-vds.md` § Migration script). Diffs: `PORTAINER_URL=https://portainer.kzntsv.site`, endpointId=1, dir prefix `/usr/docker/`, auth `X-API-Key` (PAT works, no JWT fallback). Then Phase 2 ladder: proxy-chain → imgproxy → minio → mongo → books-db → elasticsearch.
|
||||
|
||||
## Спроси user'а
|
||||
|
||||
- **Push** `fe41ee44` + `439da3ce` в `origin/master`? (Rule 4: каждая сессия start ask-mode). Это 2 коммита: backup pipeline standup + follow-up task.
|
||||
- **Какую таску брать в работу следующей** — `books-vds-stacks-to-portainer` (sequel к backup'у), `stateful-split-volume-copy` (отложили), или другое?
|
||||
- **Endpoint 6 в books Portainer** = stostayer VDS — мы за него отвечаем? Если да — после books-vds migration extend задачу на stostayer тоже.
|
||||
- **(нет на этом раунде — auto-push grant active, scope decisions made, момент migrate-by-migrate ладдером).**
|
||||
|
||||
## Не делать (preemptive guards)
|
||||
|
||||
- **Не push без grant'а** (project-discipline Rule 4). Session начинает ask-mode.
|
||||
- **Не trust wiki blindly** — пример `books-ssh-access.md` (фикс'нут в этом раунде, теперь `vds-kzntsv-ssh-access.md`). Faktcheck через `vds-ops` MCP + SSH probe + IP до построения предположений.
|
||||
- **Не trigger полный backup для smoke** на books VDS — 13m+ wall-clock, 4.87 GB initial. Smoke = только notify-channel тест через standalone curl call.
|
||||
- **Не editit `/usr/docker/<svc>/docker-compose.yml`** напрямую для production stacks — `books-vds-stacks-to-portainer` именно про правильный путь (migrate под Portainer, edit там).
|
||||
- **Не trigger ES recreate** в нормальное время — текущая ES сейчас «не используется» per user, но это **могло измениться**. Verify before any recreate.
|
||||
- **Не migrate traefik / portainer** — management plane, recreate ломает access (Skip-list).
|
||||
- **Не trigger ES recreate без user awareness** — текущая ES «не используется» per user 2026-05-25, но verify before recreate. ES в ладдере **последним** на покое.
|
||||
- **Не editit `/usr/docker/<svc>/docker-compose.yml`** напрямую после migration — Portainer-managed working_dir = `/data/compose/<id>/`. Compose files в `/usr/docker/<svc>/` становятся reference only.
|
||||
- **Не break backup pipeline** — `books-vds-backup-daily-kreknin` rsync paths = `/usr/docker/<svc>/data` (binds). Bind paths preserved через migration, paths остаются валидны. Verify post-migration rsync прошёл хотя бы 1 раз.
|
||||
- **Не push without grant in future sessions** — auto-push грант session-only (Rule 4). Next session start = ask-mode again.
|
||||
|
||||
## Memory updates за сессию
|
||||
|
||||
- **Учить:** `books VDS ≠ vds-kzntsv`. Real books VDS = `89.253.255.133` (host4g.ru, CentOS 7, kernel 3.10 EOL). Documented в [[../wiki/entities/books-vds]].
|
||||
- **Учить:** pass entries one-file-per-host pattern — consolidate сервис-specific creds в `<host>/full-env` с section'ами (matches `vds-kzntsv/full-env` style). Не плодить `<host>-<service>/full-env`.
|
||||
- **Учить:** Yandex SMTP — port 465 = `smtps://` (implicit TLS), port 587 = `smtp://` + `--ssl-reqd` (STARTTLS). Mismatch = 30-sec curl timeout.
|
||||
- **Учить:** CentOS 7 OpenSSH 7.4 не имеет `accept-new` (added 7.6). Bootstrap step: `ssh-keyscan` host-key в `known_hosts`, потом `StrictHostKeyChecking=yes`.
|
||||
- **Учить:** docker-compose 1.29 + Docker 26 = `KeyError: 'ContainerConfig'` при recreate. Workaround: `docker rm -f <renamed-stuck>` + `docker-compose up -d`. Bigger fix: upgrade docker-compose to v2 plugin.
|
||||
- **Reference:** books VDS Portainer = `https://portainer.kzntsv.site` (НЕ `portainer.vds.kzntsv.site`). PAT в `pass show books-vds/full-env BOOKS_PORTAINER_API_KEY`.
|
||||
- **Verified 2026-05-25:** ntfy push `BOOKS-VDS backup OK 2026-05-25` пришёл на phone ✓. Email `[BOOKS-VDS] backup OK ...` + standalone `[BOOKS-VDS] SMTP fix verify` пришли в inbox ✓.
|
||||
- **Учить:** Portainer single instance `portainer.kzntsv.site` управляет 2 endpoint'ами — `1` = books VDS local docker daemon, `6` = stostayer (другой клиент, **не наш**, scope не расширять). User-confirmed 2026-05-25.
|
||||
- **Учить:** PAT auth через `X-API-Key: ptr_...` header работает (без JWT fallback). Pass entry `books-vds/full-env BOOKS_PORTAINER_API_KEY`.
|
||||
- **Учить:** на books VDS все SSH-compose стеки **bind-only** (нет named volumes для них) и **no env_file** (env inline в compose). Migration упрощается: gotcha #1 (env_file strip) не релевантен, gotcha #2 (absolutize) — единственная требуемая transform.
|
||||
- **Учить:** `imgproxy` (2 контейнера, `imgproxy.kzntsv.site`, S3→minio) был missing из `entities/books-vds.md` SSH-managed table до 2026-05-25. Fix'нут в этом раунде. Lesson: при entity wiki бутстрапе на legacy host'ах — обязательно `ls /usr/docker/` cross-check vs running `docker ps`, не доверять предположениям.
|
||||
- **Reference:** migration pattern source — `.wiki/concepts/portainer-stack-management-vds.md` (VDS-infra 12-stack retro-migration 2026-05-22). Adaptable для books VDS с 3-4 diff'ами.
|
||||
|
||||
84
scripts/books-vds-portainer-migration/README.md
Normal file
84
scripts/books-vds-portainer-migration/README.md
Normal file
@@ -0,0 +1,84 @@
|
||||
# books-vds-portainer-migration
|
||||
|
||||
Migrates SSH-compose stacks at `/usr/docker/<stack>/` on books VDS (`89.253.255.133`) to Portainer-managed via API. Adapted from canonical pattern: [`.wiki/concepts/portainer-stack-management-vds.md`](../../.wiki/concepts/portainer-stack-management-vds.md).
|
||||
|
||||
Related task: [`books-vds-stacks-to-portainer`](../../.tasks/books-vds-stacks-to-portainer.md).
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# 1. Get PAT from pass
|
||||
export BOOKS_PORTAINER_API_KEY=$(pass show books-vds/full-env BOOKS_PORTAINER_API_KEY)
|
||||
|
||||
# 2. Copy script to VDS
|
||||
scp -i ~/.ssh/id_ed25519_books_ops migrate.sh root@89.253.255.133:/tmp/
|
||||
|
||||
# 3. Migrate one stack at a time, verify between
|
||||
ssh -i ~/.ssh/id_ed25519_books_ops root@89.253.255.133 \
|
||||
"BOOKS_PORTAINER_API_KEY='$BOOKS_PORTAINER_API_KEY' bash /tmp/migrate.sh proxy-chain"
|
||||
```
|
||||
|
||||
## Ladder (lowest → highest blast radius)
|
||||
|
||||
1. **proxy-chain** — internal-only, no traefik exposure → smoke for the script
|
||||
2. **imgproxy** — 2-container compose, serves `imgproxy.kzntsv.site`
|
||||
3. **minio** — read-mostly, books-api retries OK
|
||||
4. **mongo** — shared, used by books-api + books-task-runner. **VERIFY retry-tolerance first**
|
||||
5. **books-db** — MariaDB, books-api connection-pool reconnects
|
||||
6. **elasticsearch** — currently "not used" per user. Verify before recreate.
|
||||
|
||||
## Smoke per stack (after each migration)
|
||||
|
||||
```bash
|
||||
# proxy-chain — internal smoke (no public endpoint)
|
||||
docker exec proxy-chain wget -qO- localhost:8000 || echo "$?"
|
||||
|
||||
# imgproxy — serves imgproxy.kzntsv.site
|
||||
curl -sS -o /dev/null -w "%{http_code}\n" https://imgproxy.kzntsv.site/
|
||||
|
||||
# minio
|
||||
curl -sS -o /dev/null -w "%{http_code}\n" https://minio.kzntsv.site/minio/health/live
|
||||
|
||||
# mongo
|
||||
docker exec mongo mongo --quiet --eval 'db.adminCommand({ping:1}).ok' \
|
||||
-u root -p "$(pass show books-vds/full-env | grep MONGO_PWD)"
|
||||
|
||||
# books-db
|
||||
docker exec books-db mariadb -uroot -p"$(pass show books-vds/full-env | grep BOOKS_DB_PWD)" -e 'SELECT 1'
|
||||
|
||||
# elasticsearch
|
||||
docker exec elasticsearch curl -sS localhost:9200/_cluster/health | jq .status
|
||||
```
|
||||
|
||||
## What the script does
|
||||
|
||||
1. **Backup** `docker-compose.yml.bak-pre-portainer-migration-<date>` (idempotent — keeps first backup).
|
||||
2. **Transform** compose: strip `env_file:` (defensive — books VDS has none), absolutize `./` paths to `/usr/docker/<stack>/`.
|
||||
3. **Parse** `.env` into API env array (no-op for books VDS — no `.env` files).
|
||||
4. **Delete** existing Portainer stack with same name on endpoint 1 (idempotency for re-runs).
|
||||
5. **Down** ad-hoc containers via `docker rm -f` by `com.docker.compose.project` label (bypasses docker-compose v1 `ContainerConfig` bug).
|
||||
6. **POST** `/api/stacks/create/standalone/string?endpointId=1` — Portainer pulls images, creates network attachments, applies traefik labels through docker socket events.
|
||||
|
||||
## Diffs from VDS-infra script (`portainer-stack-management-vds.md`)
|
||||
|
||||
| Aspect | VDS-infra | books VDS |
|
||||
|---|---|---|
|
||||
| Portainer URL | `portainer.vds.kzntsv.site` | `portainer.kzntsv.site` |
|
||||
| Endpoint ID | 1 (same) | 1 |
|
||||
| Stack dir prefix | `/opt/stacks/<stack>/` | `/usr/docker/<stack>/` |
|
||||
| Auth | JWT (PAT 401'd) | **X-API-Key** (PAT works) |
|
||||
| Compose down | `cd $DIR && docker compose down` | **`docker rm -f` by label** (skips compose binary entirely) |
|
||||
| `env_file:` strip | required (most stacks had .env) | defensive (books VDS has none) |
|
||||
|
||||
## Safety
|
||||
|
||||
- **Re-runnable.** Delete-then-create idempotency means re-running for the same stack just rebuilds it (data binds preserved, names preserved, network re-attached).
|
||||
- **Compose file preserved on disk** (in `/usr/docker/<stack>/` + dated backup). Portainer manages lifecycle but doesn't move the file.
|
||||
- **Bind paths preserved** — `/usr/docker/<stack>/data` etc. stay at the same disk path. Backup pipeline (`scripts/books-vds-backup-daily-kreknin/`) doesn't need adjustment.
|
||||
- **Skips traefik / portainer** by name — refuses to migrate management plane.
|
||||
|
||||
## Post-migration
|
||||
|
||||
- Update [`.wiki/entities/books-vds.md`](../../.wiki/entities/books-vds.md) — flip each stack from "SSH-managed" to "Portainer-managed" table.
|
||||
- Extend or fork [`.wiki/concepts/portainer-stack-management-vds.md`](../../.wiki/concepts/portainer-stack-management-vds.md) with books VDS section + gotchas if new ones encountered.
|
||||
- Close task as 🟢.
|
||||
109
scripts/books-vds-portainer-migration/migrate.sh
Normal file
109
scripts/books-vds-portainer-migration/migrate.sh
Normal file
@@ -0,0 +1,109 @@
|
||||
#!/bin/bash
|
||||
# books-vds-portainer-migration/migrate.sh
|
||||
#
|
||||
# Migrates one SSH-compose stack at /usr/docker/<STACK>/ to Portainer-managed via API.
|
||||
# Pattern source: .wiki/concepts/portainer-stack-management-vds.md § Migration script.
|
||||
# Diffs from VDS-infra version:
|
||||
# PORTAINER_URL = https://portainer.kzntsv.site
|
||||
# ENDPOINT_ID = 1 (books VDS local docker daemon)
|
||||
# DIR prefix = /usr/docker/<stack> (not /opt/stacks/<stack>)
|
||||
# Auth = X-API-Key header (PAT works, no JWT fallback)
|
||||
# Down step = direct `docker rm -f` by compose-project label
|
||||
# (bypasses both v1 standalone quirks and Portainer recreate edge cases)
|
||||
#
|
||||
# Usage (run on books VDS):
|
||||
# scp migrate.sh root@89.253.255.133:/tmp/
|
||||
# ssh root@89.253.255.133
|
||||
# export BOOKS_PORTAINER_API_KEY="ptr_..." # from `pass show books-vds/full-env`
|
||||
# bash /tmp/migrate.sh <stack-name>
|
||||
#
|
||||
# Targets (recommended ladder, lowest → highest blast radius):
|
||||
# proxy-chain → imgproxy → minio → mongo → books-db → elasticsearch
|
||||
#
|
||||
# Skip (management plane — would lose access):
|
||||
# traefik, portainer
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
STACK="${1:?stack name required — e.g. proxy-chain | imgproxy | minio | mongo | books-db | elasticsearch}"
|
||||
DIR="/usr/docker/$STACK"
|
||||
PORTAINER_URL="https://portainer.kzntsv.site"
|
||||
ENDPOINT_ID=1
|
||||
API_KEY="${BOOKS_PORTAINER_API_KEY:?env var required — pass show books-vds/full-env BOOKS_PORTAINER_API_KEY}"
|
||||
|
||||
# Refuse to migrate management-plane stacks
|
||||
case "$STACK" in
|
||||
traefik|portainer)
|
||||
echo "✗ refusing to migrate '$STACK' — management plane (recreate breaks access)"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
[ -d "$DIR" ] || { echo "✗ missing $DIR"; exit 1; }
|
||||
[ -f "$DIR/docker-compose.yml" ] || { echo "✗ missing $DIR/docker-compose.yml"; exit 1; }
|
||||
|
||||
echo "=== migrate '$STACK' from $DIR → Portainer endpoint $ENDPOINT_ID ==="
|
||||
|
||||
# 1. Backup compose file (idempotent — keeps first backup if re-run)
|
||||
BACKUP="$DIR/docker-compose.yml.bak-pre-portainer-migration-$(date +%Y-%m-%d)"
|
||||
if [ ! -f "$BACKUP" ]; then
|
||||
cp "$DIR/docker-compose.yml" "$BACKUP"
|
||||
echo "→ backup: $BACKUP"
|
||||
else
|
||||
echo "→ backup exists: $BACKUP (keeping original)"
|
||||
fi
|
||||
|
||||
# 2. Transform compose:
|
||||
# - strip env_file directives (defensive — Portainer string-mode can't materialize .env)
|
||||
# - absolutize ./ paths (Portainer working_dir = /data/compose/<id>/, not stack dir)
|
||||
COMPOSE=$(sed \
|
||||
-e '/^\s*env_file:/d' \
|
||||
-e '/^\s*-\s*\.env\s*$/d' \
|
||||
-e "s|\./|$DIR/|g" \
|
||||
"$DIR/docker-compose.yml")
|
||||
|
||||
# 3. Parse .env into Portainer API env array (no-op if absent — books VDS has no .env files)
|
||||
ENV_ARRAY="[]"
|
||||
if [ -f "$DIR/.env" ]; then
|
||||
ENV_ARRAY=$(grep -vE '^\s*(#|$)' "$DIR/.env" | jq -Rs '
|
||||
split("\n") | map(select(length>0) | split("=") | {name: .[0], value: (.[1:] | join("="))})')
|
||||
fi
|
||||
|
||||
PAYLOAD=$(jq -n --arg name "$STACK" --arg compose "$COMPOSE" --argjson env "$ENV_ARRAY" \
|
||||
'{name:$name, stackFileContent:$compose, env:$env, fromAppTemplate:false}')
|
||||
|
||||
# 4. Idempotency: delete existing Portainer stack with same name on same endpoint
|
||||
EXISTING_ID=$(curl -ksS -H "X-API-Key: $API_KEY" "$PORTAINER_URL/api/stacks" | \
|
||||
jq -r ".[] | select(.Name==\"$STACK\" and .EndpointId==$ENDPOINT_ID) | .Id" | head -1)
|
||||
if [ -n "$EXISTING_ID" ]; then
|
||||
echo "→ existing Portainer stack id=$EXISTING_ID — deleting"
|
||||
curl -ksS -X DELETE "$PORTAINER_URL/api/stacks/$EXISTING_ID?endpointId=$ENDPOINT_ID" \
|
||||
-H "X-API-Key: $API_KEY" >/dev/null
|
||||
fi
|
||||
|
||||
# 5. Down ad-hoc containers (direct `docker rm -f` by compose-project label —
|
||||
# bypasses docker-compose v1 ContainerConfig bug + any v2 recreate quirks)
|
||||
CONTAINERS=$(docker ps -aq --filter "label=com.docker.compose.project=$STACK" || true)
|
||||
if [ -n "$CONTAINERS" ]; then
|
||||
COUNT=$(echo "$CONTAINERS" | wc -l)
|
||||
echo "→ removing $COUNT ad-hoc container(s) (by compose-project label)"
|
||||
docker rm -f $CONTAINERS
|
||||
fi
|
||||
|
||||
# 6. Create via Portainer API
|
||||
echo "→ POST /api/stacks/create/standalone/string"
|
||||
RESP=$(curl -ksS -X POST "$PORTAINER_URL/api/stacks/create/standalone/string?endpointId=$ENDPOINT_ID" \
|
||||
-H "X-API-Key: $API_KEY" -H "Content-Type: application/json" --data "$PAYLOAD")
|
||||
NEW_ID=$(echo "$RESP" | jq -r '.Id // empty')
|
||||
|
||||
if [ -n "$NEW_ID" ]; then
|
||||
echo "✓ stack '$STACK' created, Portainer Id=$NEW_ID"
|
||||
echo
|
||||
echo "→ containers now running:"
|
||||
docker ps --filter "label=com.docker.compose.project=$STACK" \
|
||||
--format 'table {{.Names}}\t{{.Image}}\t{{.Status}}'
|
||||
else
|
||||
echo "✗ FAILED:"
|
||||
echo "$RESP" | jq .
|
||||
exit 1
|
||||
fi
|
||||
Reference in New Issue
Block a user