# migrate-elasticsearch-to-books-vds

## Goal

Переехать ES индексы с Windows host (`elasticold.kzntsv.site`, ES 7.10.1) на books VDS (`elasticsearch.kzntsv.site`, ES 7.10.0, single-node behind traefik basicAuth). Переключить books-api / books-task-runner / books-job-scheduler через `default.json` на новый endpoint. Source ES оставить running как rollback (не выключать).

## Approach

**Reindex-from-remote** (не snapshot/restore).

Why:
- Source = Windows docker, нет `path.repo` configured + нет bind для snapshot dir → snapshot подход требует restart source = consumer downtime.
- Target = books VDS Portainer stack 33, modify env через Portainer API = recreate target only, consumers unaffected (они пока на source).
- Same Lucene 8.7.0 → docs reindex transparently через `_reindex` API.
- 730mb total → ~5 min over public network.

Trade-off: reindex использует target's dynamic mapping. Pre-create target indices с source mappings/settings (snapshot уже сохранён в `.scratch/`).

## Source/target inventory

**Source (Windows, `localhost:9200`, behind `elasticold.kzntsv.site` traefik basicAuth):**
- ES 7.10.1, Lucene 8.7.0, `discovery.type=single-node`
- 3 indices:
  - `artmone` 2621 docs, 1.4mb
  - `epz` 820604 docs, 699.7mb
  - `products` 105922 docs, 26.6mb
- Compose: `C:\Users\vitya\projects\docker\diskstation\elasticsearch\docker-compose.yml`
- Bind: `./data:/usr/share/elasticsearch/data`. **No snapshot path.**
- basicAuth user `books`, hash `$apr1$vyxr1l5z$fpxHmNBfAx8cHrQTje4Fx/`

**Target (books VDS 89.253.255.133, Portainer stack 33 `elasticsearch`, behind `elasticsearch.kzntsv.site`):**
- ES 7.10.0, Lucene 8.7.0, single-node, yellow (replica unassigned — expected)
- 1 index: `read_me` (4.8kb placeholder)
- `path.repo=/snapshots`, kreknin fs repo registered
- Bind: `/usr/docker/elasticsearch/{data,snapshots}` → `/usr/share/elasticsearch/{data,/snapshots}`
- basicAuth user `books`, **same hash как source** → same password

**Consumers (all on books VDS, configs bind-mounted `/opt/books/<svc>/config/default.json`):**
- `books-api` (stack 23): port 3021, Nitro
- `books-task-runner` (stack 22, sibling of job-scheduler+mongo)
- `books-job-scheduler` (stack 22)

Все три имеют block:
```json
"elasticsearch": {
    "node": "https://elasticold.kzntsv.site/",
    "auth": { "username": "books", "password": "<same>" }
}
```

## Key files

- `C:\Users\vitya\projects\docker\diskstation\elasticsearch\docker-compose.yml` — source ES compose (no edit planned)
- `C:\Users\vitya\projects\.admin\.scratch\source-mappings.json` — frozen source mappings 2026-05-25
- `C:\Users\vitya\projects\.admin\.scratch\source-settings.json` — frozen source settings 2026-05-25
- books VDS `/opt/books/api/config/default.json:27` — consumer config (idem task-runner, job-scheduler)
- books VDS Portainer stack 33 — target ES compose (modify env via Portainer API)
- books VDS `/usr/docker/elasticsearch/snapshots/` — kreknin fs repo, pre-migration backup target

## Execution plan

1. **Pre-migration ES snapshot books VDS** (kreknin repo, `pre-migration-<date>` snapshot name) — instant rollback for target.
2. **Save source mappings/settings** — done (`.scratch/source-{mappings,settings}.json`).
3. **Modify target ES stack** via Portainer API: add env `reindex.remote.whitelist=elasticold.kzntsv.site:443`. Recreate stack (~10s target downtime; consumers unaffected — они на source).
4. **Pre-create target indices** с source mappings/settings (number_of_replicas=0 для single-node clean green).
5. **POST `/_reindex`** per index, wait_for_completion=false, slices=auto. Poll task status.
6. **Verify counts**: target `_cat/indices` doc.counts == source. Verify random doc fetch by `_id` matches.
7. **Update consumer configs**: replace `elasticold.kzntsv.site` → `elasticsearch.kzntsv.site` в `/opt/books/{api,task-runner,job-scheduler}/config/default.json` (sed in-place).
8. **Bounce consumers** через `docker restart books-api books-task-runner books-job-scheduler` (mongo / db не трогаем — bind-mount уже видит новый config).
9. **Smoke**: tail `docker logs --since 60s books-api books-task-runner books-job-scheduler` → нет ES connection errors. Trigger known query → response from target.
10. **Don't disable source** (per user) — Windows ES + elasticold traefik route остаётся live для rollback.

## Decisions log

- 2026-05-25: подход reindex-from-remote, не snapshot/restore. Reason: source compose не имеет `path.repo` configured + bind для snapshot — это потребовало бы restart source (= consumer downtime). Target compose в Portainer, restart target не задевает consumers (они пока на source). Same Lucene 8.7.0 → reindex transparent.
- 2026-05-25: pre-create target indices с frozen source mappings — иначе reindex использовал бы dynamic mapping, possibly losing exact analyzer config. Mappings/settings уже извлечены в `.scratch/`.
- 2026-05-25: consumer config update через `sed -i` (bind-mounted files), restart through `docker restart` (не Portainer stack recreate) — mongo / db в той же stack id 22 не должны bounce.

## Open questions

- [x] `number_of_replicas`: source = 1 (yellow на single-node). Target тоже single-node → set to 0 на pre-create для green. Source оставить как есть. **Resolved: set 0 на target pre-create.**

## Completed steps

- [x] Source ES probed: 3 indices, 730mb total, ES 7.10.1
- [x] Target ES probed: ES 7.10.0, 1 placeholder index `read_me`, no collisions
- [x] Source mappings/settings frozen в `.scratch/`
- [x] basicAuth password identified из consumer config (same on both sides)
- [x] Containers inventory: 3 consumer containers, all on books VDS
- [x] Pre-migration ES snapshot `pre-migration-2026-05-25` в kreknin repo (read_me only, SUCCESS — rollback point)
- [x] Target ES compose updated через Portainer API: `reindex.remote.whitelist=elasticold.kzntsv.site:443`. Container restart 07:51:39 UTC. Yellow cluster post-restart (replica unassigned = expected).
- [x] 3 target indices pre-created с frozen source mappings/settings, `number_of_replicas=0` → green.
- [x] Reindex from remote: artmone (2621/2621, 25s), epz (820604/820604, ~5m async), products (105922/105922, ~25m async).
- [x] Doc count parity src=dst для всех 3 indices.
- [x] Per-doc sample fetch: byte-match _id=29359 (products) + _id=22023500 (epz).
- [x] Consumer configs sed in-place: `elasticold` → `elasticsearch`, .bak-pre-es-migration-2026-05-25 saved для rollback.
- [x] Restart 3 containers: books-api running, books-task-runner healthy, books-job-scheduler healthy.
- [x] Direct internal smoke (`docker exec books-api node -e ...`): API sees `https://elasticsearch.kzntsv.site/`, `_cluster/health` 200 OK.
- [x] Post-migration source traefik access log = 0 elasticold hits за 5 min.

## Closure note (2026-05-25)

Migration **complete**. 3 indices (artmone/epz/products, 820k+108k+2.6k docs, 850mb total target storage) on books VDS ES live. Consumers switched (books-api + books-task-runner + books-job-scheduler).

**Source НЕ disabled** (per user requirement) — Windows ES + elasticold.kzntsv.site traefik route остаются live для emergency rollback. Кешированный data на windows host остаётся в одном экземпляре до отдельного decommission решения.

**Rollback**: `ssh root@89.253.255.133 'for f in /opt/books/{api,task-runner,job-scheduler}/config/default.json; do cp "$f.bak-pre-es-migration-2026-05-25" "$f"; done; docker restart books-api books-task-runner books-job-scheduler'`.

**Future ops**: при decommission Windows ES — `docker stop elasticsearch` на windows + remove traefik label `elasticold.kzntsv.site` route. `path.repo` / kreknin snapshot pre-migration-2026-05-25 уже не несёт практически данных (один read_me index), может быть удалён вместе с next backup retention rotation.

## Notes

- Source basicAuth password в `/opt/books/api/config/default.json` (consumer config). Не дублировать в эту таску.
- `books-job-scheduler-mongo` — другая БД, не задевается.
- Rollback path: `sed -i 's/elasticsearch\.kzntsv\.site/elasticold.kzntsv.site/g' /opt/books/*/config/default.json` + restart consumers.

<!-- created-by: vitya / 2026-05-25 / trigger: ES consolidation на books VDS, source-host decommission preparation -->