vitya
3246dabbb5
928616 docs (artmone 2621 / epz 820604 / products 105922) migrated via reindex-from-remote. Pre-migration snapshot к kreknin repo, target stack 33 получил `reindex.remote.whitelist=elasticold.kzntsv.site:443` через Portainer API. Consumer configs (books-api + books-task-runner + books-job-scheduler) sed'd elasticold→elasticsearch.kzntsv.site, 3 containers restarted. Per-doc byte-match verified, internal smoke от books-api → новый ES 200 OK. Source ES оставлен running per user (rollback ready).
118 lines
8.6 KiB
Markdown
118 lines
8.6 KiB
Markdown
# migrate-elasticsearch-to-books-vds
|
||
|
||
## Goal
|
||
|
||
Переехать ES индексы с Windows host (`elasticold.kzntsv.site`, ES 7.10.1) на books VDS (`elasticsearch.kzntsv.site`, ES 7.10.0, single-node behind traefik basicAuth). Переключить books-api / books-task-runner / books-job-scheduler через `default.json` на новый endpoint. Source ES оставить running как rollback (не выключать).
|
||
|
||
## Approach
|
||
|
||
**Reindex-from-remote** (не snapshot/restore).
|
||
|
||
Why:
|
||
- Source = Windows docker, нет `path.repo` configured + нет bind для snapshot dir → snapshot подход требует restart source = consumer downtime.
|
||
- Target = books VDS Portainer stack 33, modify env через Portainer API = recreate target only, consumers unaffected (они пока на source).
|
||
- Same Lucene 8.7.0 → docs reindex transparently через `_reindex` API.
|
||
- 730mb total → ~5 min over public network.
|
||
|
||
Trade-off: reindex использует target's dynamic mapping. Pre-create target indices с source mappings/settings (snapshot уже сохранён в `.scratch/`).
|
||
|
||
## Source/target inventory
|
||
|
||
**Source (Windows, `localhost:9200`, behind `elasticold.kzntsv.site` traefik basicAuth):**
|
||
- ES 7.10.1, Lucene 8.7.0, `discovery.type=single-node`
|
||
- 3 indices:
|
||
- `artmone` 2621 docs, 1.4mb
|
||
- `epz` 820604 docs, 699.7mb
|
||
- `products` 105922 docs, 26.6mb
|
||
- Compose: `C:\Users\vitya\projects\docker\diskstation\elasticsearch\docker-compose.yml`
|
||
- Bind: `./data:/usr/share/elasticsearch/data`. **No snapshot path.**
|
||
- basicAuth user `books`, hash `$apr1$vyxr1l5z$fpxHmNBfAx8cHrQTje4Fx/`
|
||
|
||
**Target (books VDS 89.253.255.133, Portainer stack 33 `elasticsearch`, behind `elasticsearch.kzntsv.site`):**
|
||
- ES 7.10.0, Lucene 8.7.0, single-node, yellow (replica unassigned — expected)
|
||
- 1 index: `read_me` (4.8kb placeholder)
|
||
- `path.repo=/snapshots`, kreknin fs repo registered
|
||
- Bind: `/usr/docker/elasticsearch/{data,snapshots}` → `/usr/share/elasticsearch/{data,/snapshots}`
|
||
- basicAuth user `books`, **same hash как source** → same password
|
||
|
||
**Consumers (all on books VDS, configs bind-mounted `/opt/books/<svc>/config/default.json`):**
|
||
- `books-api` (stack 23): port 3021, Nitro
|
||
- `books-task-runner` (stack 22, sibling of job-scheduler+mongo)
|
||
- `books-job-scheduler` (stack 22)
|
||
|
||
Все три имеют block:
|
||
```json
|
||
"elasticsearch": {
|
||
"node": "https://elasticold.kzntsv.site/",
|
||
"auth": { "username": "books", "password": "<same>" }
|
||
}
|
||
```
|
||
|
||
## Key files
|
||
|
||
- `C:\Users\vitya\projects\docker\diskstation\elasticsearch\docker-compose.yml` — source ES compose (no edit planned)
|
||
- `C:\Users\vitya\projects\.admin\.scratch\source-mappings.json` — frozen source mappings 2026-05-25
|
||
- `C:\Users\vitya\projects\.admin\.scratch\source-settings.json` — frozen source settings 2026-05-25
|
||
- books VDS `/opt/books/api/config/default.json:27` — consumer config (idem task-runner, job-scheduler)
|
||
- books VDS Portainer stack 33 — target ES compose (modify env via Portainer API)
|
||
- books VDS `/usr/docker/elasticsearch/snapshots/` — kreknin fs repo, pre-migration backup target
|
||
|
||
## Execution plan
|
||
|
||
1. **Pre-migration ES snapshot books VDS** (kreknin repo, `pre-migration-<date>` snapshot name) — instant rollback for target.
|
||
2. **Save source mappings/settings** — done (`.scratch/source-{mappings,settings}.json`).
|
||
3. **Modify target ES stack** via Portainer API: add env `reindex.remote.whitelist=elasticold.kzntsv.site:443`. Recreate stack (~10s target downtime; consumers unaffected — они на source).
|
||
4. **Pre-create target indices** с source mappings/settings (number_of_replicas=0 для single-node clean green).
|
||
5. **POST `/_reindex`** per index, wait_for_completion=false, slices=auto. Poll task status.
|
||
6. **Verify counts**: target `_cat/indices` doc.counts == source. Verify random doc fetch by `_id` matches.
|
||
7. **Update consumer configs**: replace `elasticold.kzntsv.site` → `elasticsearch.kzntsv.site` в `/opt/books/{api,task-runner,job-scheduler}/config/default.json` (sed in-place).
|
||
8. **Bounce consumers** через `docker restart books-api books-task-runner books-job-scheduler` (mongo / db не трогаем — bind-mount уже видит новый config).
|
||
9. **Smoke**: tail `docker logs --since 60s books-api books-task-runner books-job-scheduler` → нет ES connection errors. Trigger known query → response from target.
|
||
10. **Don't disable source** (per user) — Windows ES + elasticold traefik route остаётся live для rollback.
|
||
|
||
## Decisions log
|
||
|
||
- 2026-05-25: подход reindex-from-remote, не snapshot/restore. Reason: source compose не имеет `path.repo` configured + bind для snapshot — это потребовало бы restart source (= consumer downtime). Target compose в Portainer, restart target не задевает consumers (они пока на source). Same Lucene 8.7.0 → reindex transparent.
|
||
- 2026-05-25: pre-create target indices с frozen source mappings — иначе reindex использовал бы dynamic mapping, possibly losing exact analyzer config. Mappings/settings уже извлечены в `.scratch/`.
|
||
- 2026-05-25: consumer config update через `sed -i` (bind-mounted files), restart through `docker restart` (не Portainer stack recreate) — mongo / db в той же stack id 22 не должны bounce.
|
||
|
||
## Open questions
|
||
|
||
- [x] `number_of_replicas`: source = 1 (yellow на single-node). Target тоже single-node → set to 0 на pre-create для green. Source оставить как есть. **Resolved: set 0 на target pre-create.**
|
||
|
||
## Completed steps
|
||
|
||
- [x] Source ES probed: 3 indices, 730mb total, ES 7.10.1
|
||
- [x] Target ES probed: ES 7.10.0, 1 placeholder index `read_me`, no collisions
|
||
- [x] Source mappings/settings frozen в `.scratch/`
|
||
- [x] basicAuth password identified из consumer config (same on both sides)
|
||
- [x] Containers inventory: 3 consumer containers, all on books VDS
|
||
- [x] Pre-migration ES snapshot `pre-migration-2026-05-25` в kreknin repo (read_me only, SUCCESS — rollback point)
|
||
- [x] Target ES compose updated через Portainer API: `reindex.remote.whitelist=elasticold.kzntsv.site:443`. Container restart 07:51:39 UTC. Yellow cluster post-restart (replica unassigned = expected).
|
||
- [x] 3 target indices pre-created с frozen source mappings/settings, `number_of_replicas=0` → green.
|
||
- [x] Reindex from remote: artmone (2621/2621, 25s), epz (820604/820604, ~5m async), products (105922/105922, ~25m async).
|
||
- [x] Doc count parity src=dst для всех 3 indices.
|
||
- [x] Per-doc sample fetch: byte-match _id=29359 (products) + _id=22023500 (epz).
|
||
- [x] Consumer configs sed in-place: `elasticold` → `elasticsearch`, .bak-pre-es-migration-2026-05-25 saved для rollback.
|
||
- [x] Restart 3 containers: books-api running, books-task-runner healthy, books-job-scheduler healthy.
|
||
- [x] Direct internal smoke (`docker exec books-api node -e ...`): API sees `https://elasticsearch.kzntsv.site/`, `_cluster/health` 200 OK.
|
||
- [x] Post-migration source traefik access log = 0 elasticold hits за 5 min.
|
||
|
||
## Closure note (2026-05-25)
|
||
|
||
Migration **complete**. 3 indices (artmone/epz/products, 820k+108k+2.6k docs, 850mb total target storage) on books VDS ES live. Consumers switched (books-api + books-task-runner + books-job-scheduler).
|
||
|
||
**Source НЕ disabled** (per user requirement) — Windows ES + elasticold.kzntsv.site traefik route остаются live для emergency rollback. Кешированный data на windows host остаётся в одном экземпляре до отдельного decommission решения.
|
||
|
||
**Rollback**: `ssh root@89.253.255.133 'for f in /opt/books/{api,task-runner,job-scheduler}/config/default.json; do cp "$f.bak-pre-es-migration-2026-05-25" "$f"; done; docker restart books-api books-task-runner books-job-scheduler'`.
|
||
|
||
**Future ops**: при decommission Windows ES — `docker stop elasticsearch` на windows + remove traefik label `elasticold.kzntsv.site` route. `path.repo` / kreknin snapshot pre-migration-2026-05-25 уже не несёт практически данных (один read_me index), может быть удалён вместе с next backup retention rotation.
|
||
|
||
## Notes
|
||
|
||
- Source basicAuth password в `/opt/books/api/config/default.json` (consumer config). Не дублировать в эту таску.
|
||
- `books-job-scheduler-mongo` — другая БД, не задевается.
|
||
- Rollback path: `sed -i 's/elasticsearch\.kzntsv\.site/elasticold.kzntsv.site/g' /opt/books/*/config/default.json` + restart consumers.
|
||
|
||
<!-- created-by: vitya / 2026-05-25 / trigger: ES consolidation на books VDS, source-host decommission preparation -->
|