admin/.tasks/migrate-elasticsearch-to-books-vds.md

migrate-elasticsearch-to-books-vds

Goal

Переехать ES индексы с Windows host (elasticold.kzntsv.site, ES 7.10.1) на books VDS (elasticsearch.kzntsv.site, ES 7.10.0, single-node behind traefik basicAuth). Переключить books-api / books-task-runner / books-job-scheduler через default.json на новый endpoint. Source ES оставить running как rollback (не выключать).

Approach

Reindex-from-remote (не snapshot/restore).

Why:

• Source = Windows docker, нет path.repo configured + нет bind для snapshot dir → snapshot подход требует restart source = consumer downtime.
• Target = books VDS Portainer stack 33, modify env через Portainer API = recreate target only, consumers unaffected (они пока на source).
• Same Lucene 8.7.0 → docs reindex transparently через _reindex API.
• 730mb total → ~5 min over public network.

Trade-off: reindex использует target's dynamic mapping. Pre-create target indices с source mappings/settings (snapshot уже сохранён в .scratch/).

Source/target inventory

Source (Windows, localhost:9200, behind elasticold.kzntsv.site traefik basicAuth):

• ES 7.10.1, Lucene 8.7.0, discovery.type=single-node
• 3 indices:
  • artmone 2621 docs, 1.4mb
  • epz 820604 docs, 699.7mb
  • products 105922 docs, 26.6mb
• Compose: C:\Users\vitya\projects\docker\diskstation\elasticsearch\docker-compose.yml
• Bind: ./data:/usr/share/elasticsearch/data. No snapshot path.
• basicAuth user books, hash $apr1$vyxr1l5z$fpxHmNBfAx8cHrQTje4Fx/

Target (books VDS 89.253.255.133, Portainer stack 33 elasticsearch, behind elasticsearch.kzntsv.site):

• ES 7.10.0, Lucene 8.7.0, single-node, yellow (replica unassigned — expected)
• 1 index: read_me (4.8kb placeholder)
• path.repo=/snapshots, kreknin fs repo registered
• Bind: /usr/docker/elasticsearch/{data,snapshots} → /usr/share/elasticsearch/{data,/snapshots}
• basicAuth user books, same hash как source → same password

Consumers (all on books VDS, configs bind-mounted /opt/books/<svc>/config/default.json):

• books-api (stack 23): port 3021, Nitro
• books-task-runner (stack 22, sibling of job-scheduler+mongo)
• books-job-scheduler (stack 22)

Все три имеют block:

"elasticsearch": {
    "node": "https://elasticold.kzntsv.site/",
    "auth": { "username": "books", "password": "<same>" }
}

Key files

• C:\Users\vitya\projects\docker\diskstation\elasticsearch\docker-compose.yml — source ES compose (no edit planned)
• C:\Users\vitya\projects\.admin\.scratch\source-mappings.json — frozen source mappings 2026-05-25
• C:\Users\vitya\projects\.admin\.scratch\source-settings.json — frozen source settings 2026-05-25
• books VDS /opt/books/api/config/default.json:27 — consumer config (idem task-runner, job-scheduler)
• books VDS Portainer stack 33 — target ES compose (modify env via Portainer API)
• books VDS /usr/docker/elasticsearch/snapshots/ — kreknin fs repo, pre-migration backup target

Execution plan

1. Pre-migration ES snapshot books VDS (kreknin repo, pre-migration-<date> snapshot name) — instant rollback for target.
2. Save source mappings/settings — done (.scratch/source-{mappings,settings}.json).
3. Modify target ES stack via Portainer API: add env reindex.remote.whitelist=elasticold.kzntsv.site:443. Recreate stack (~10s target downtime; consumers unaffected — они на source).
4. Pre-create target indices с source mappings/settings (number_of_replicas=0 для single-node clean green).
5. POST /_reindex per index, wait_for_completion=false, slices=auto. Poll task status.
6. Verify counts: target _cat/indices doc.counts == source. Verify random doc fetch by _id matches.
7. Update consumer configs: replace elasticold.kzntsv.site → elasticsearch.kzntsv.site в /opt/books/{api,task-runner,job-scheduler}/config/default.json (sed in-place).
8. Bounce consumers через docker restart books-api books-task-runner books-job-scheduler (mongo / db не трогаем — bind-mount уже видит новый config).
9. Smoke: tail docker logs --since 60s books-api books-task-runner books-job-scheduler → нет ES connection errors. Trigger known query → response from target.
10. Don't disable source (per user) — Windows ES + elasticold traefik route остаётся live для rollback.

Decisions log

• 2026-05-25: подход reindex-from-remote, не snapshot/restore. Reason: source compose не имеет path.repo configured + bind для snapshot — это потребовало бы restart source (= consumer downtime). Target compose в Portainer, restart target не задевает consumers (они пока на source). Same Lucene 8.7.0 → reindex transparent.
• 2026-05-25: pre-create target indices с frozen source mappings — иначе reindex использовал бы dynamic mapping, possibly losing exact analyzer config. Mappings/settings уже извлечены в .scratch/.
• 2026-05-25: consumer config update через sed -i (bind-mounted files), restart through docker restart (не Portainer stack recreate) — mongo / db в той же stack id 22 не должны bounce.

Open questions

• number_of_replicas: source = 1 (yellow на single-node). Target тоже single-node → set to 0 на pre-create для green. Source оставить как есть. Resolved: set 0 на target pre-create.

Completed steps

• Source ES probed: 3 indices, 730mb total, ES 7.10.1
• Target ES probed: ES 7.10.0, 1 placeholder index read_me, no collisions
• Source mappings/settings frozen в .scratch/
• basicAuth password identified из consumer config (same on both sides)
• Containers inventory: 3 consumer containers, all on books VDS
• Pre-migration ES snapshot pre-migration-2026-05-25 в kreknin repo (read_me only, SUCCESS — rollback point)
• Target ES compose updated через Portainer API: reindex.remote.whitelist=elasticold.kzntsv.site:443. Container restart 07:51:39 UTC. Yellow cluster post-restart (replica unassigned = expected).
• 3 target indices pre-created с frozen source mappings/settings, number_of_replicas=0 → green.
• Reindex from remote: artmone (2621/2621, 25s), epz (820604/820604, ~5m async), products (105922/105922, ~25m async).
• Doc count parity src=dst для всех 3 indices.
• Per-doc sample fetch: byte-match _id=29359 (products) + _id=22023500 (epz).
• Consumer configs sed in-place: elasticold → elasticsearch, .bak-pre-es-migration-2026-05-25 saved для rollback.
• Restart 3 containers: books-api running, books-task-runner healthy, books-job-scheduler healthy.
• Direct internal smoke (docker exec books-api node -e ...): API sees https://elasticsearch.kzntsv.site/, _cluster/health 200 OK.
• Post-migration source traefik access log = 0 elasticold hits за 5 min.

Closure note (2026-05-25)

Migration complete. 3 indices (artmone/epz/products, 820k+108k+2.6k docs, 850mb total target storage) on books VDS ES live. Consumers switched (books-api + books-task-runner + books-job-scheduler).

Source НЕ disabled (per user requirement) — Windows ES + elasticold.kzntsv.site traefik route остаются live для emergency rollback. Кешированный data на windows host остаётся в одном экземпляре до отдельного decommission решения.

Rollback: ssh root@89.253.255.133 'for f in /opt/books/{api,task-runner,job-scheduler}/config/default.json; do cp "$f.bak-pre-es-migration-2026-05-25" "$f"; done; docker restart books-api books-task-runner books-job-scheduler'.

Future ops: при decommission Windows ES — docker stop elasticsearch на windows + remove traefik label elasticold.kzntsv.site route. path.repo / kreknin snapshot pre-migration-2026-05-25 уже не несёт практически данных (один read_me index), может быть удалён вместе с next backup retention rotation.

Notes

• Source basicAuth password в /opt/books/api/config/default.json (consumer config). Не дублировать в эту таску.
• books-job-scheduler-mongo — другая БД, не задевается.
• Rollback path: sed -i 's/elasticsearch\.kzntsv\.site/elasticold.kzntsv.site/g' /opt/books/*/config/default.json + restart consumers.