OpeItcLoc03/admin
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
280 Commits 1 Branch 0 Tags
5be6662c60151f0162b5931430b0e59c7cbb4612
Commit Graph

77 Commits

Author SHA1 Message Date
vitya
11554d45ac wiki(ingest): vds-kzntsv network-stack mismatch RCA + ifupdown anti-pattern 
Incident 2026-05-28 ~05:45–13:52 MSK на vds-kzntsv. ~2.5ч активного outage
+ ~5.5ч на эфемерной статике до окончательного fix хостером.

Revised RCA: наш netplan+systemd-networkd конфликтовал с provider's expected
ifupdown stack. Их start/ipadd procedure ожидает чистый ifupdown и не может
auto-recover когда networkd «держит» eth0. 8 дней работало потому что
networkd сам тянул DHCP. Когда что-то на стороне Rusonyx разорвало
DHCP-binding — auto-recovery не сработала.

Fix: systemctl mask netplan + systemd-networkd* (на running system без stop —
IP и SSH сохранились), Rusonyx ребутнул VM и положил чистый
/etc/network/interfaces.d/ifcfg-eth0 через свой start/ipadd. Netmask /18,
gw 89.253.192.1, чистый ifupdown.

Wiki:
- NEW concepts/vds-kzntsv-dhcp-outage-2026-05-28 — full RCA + recovery
  runbook (эфемерная статика + permanent-fix via ifupdown) + diagnostic
  dot-graph + revised lessons-learned + anti-pattern
- NEW sources/vds-kzntsv-incident-2026-05-28 — timeline 05:25 backup OK →
  08:43 statics → 13:52 final reset; provider's ifcfg-eth0 content; ticket
  text reference
- UPDATE entities/vds-kzntsv — mask /18, ifupdown stack, kernel cmdline
  net.ifnames=0 объясняет eth0 naming, hypervisor hw80, pass-store путь,
  Known issues §
- UPDATE concepts/rusonyx-vps-onboarding-quirks — quirk #9 переписан про
  /18 layout + canonical ifcfg, quirk #10 NEW про ifupdown vs netplan
  stack choice + bootstrap mask commands
- UPDATE index.md + log.md

Tasks:
- STATUS header: incident RESOLVED summary
- NEXT_SESSION: следующая сессия — cleanup netplan-artifacts (optional),
  registry GC (~20G pending), board-viewer-build unhealthy разбор

Memory (out-of-repo): vds-kzntsv-rusonyx-network-recovery переписан с
revised RCA — canonical bootstrap step «mask netplan/networkd» для всех
Rusonyx Ubuntu VDS.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 14:12:14 +03:00
vitya
3246dabbb5 tasks(migrate-elasticsearch-to-books-vds): 🟢 closed — 3 indices Windows→books VDS 
928616 docs (artmone 2621 / epz 820604 / products 105922) migrated via
reindex-from-remote. Pre-migration snapshot к kreknin repo, target stack 33
получил `reindex.remote.whitelist=elasticold.kzntsv.site:443` через Portainer
API. Consumer configs (books-api + books-task-runner + books-job-scheduler)
sed'd elasticold→elasticsearch.kzntsv.site, 3 containers restarted.
Per-doc byte-match verified, internal smoke от books-api → новый ES 200 OK.
Source ES оставлен running per user (rollback ready).
 2026-05-25 11:33:17 +03:00
vitya
86c44c3db9 tasks(books-vds-stacks-to-portainer): close 🟢 — 6 stacks migrated SSH-compose → Portainer 
Phase 2 ladder executed 2026-05-25 (~10min end-to-end, lowest → highest blast radius):
  proxy-chain (id 28) → imgproxy (29) → minio (30) → mongo (31) → books-db (32) → elasticsearch (33)

All 6 smoked green. books-api + books-task-runner reconnected transparently through
mongo + books-db recreate windows. ES snapshot repo `kreknin` preserved through bind.

Phase 3:
- created .wiki/concepts/portainer-stack-management-books-vds.md (pattern application + diffs from vds-kzntsv)
- entity books-vds.md: 6 stacks flipped SSH-managed → Portainer-managed
- index.md: new concept entry

Pre-flight quirks documented (CentOS 7 yum dead, no jq, no docker login).

Acceptance: 4/5 verified, backup pipeline (#5) verified by inspection (bind paths
preserved) — full pipeline run pending next 06:00 MSK.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 10:27:23 +03:00
vitya
bfb942cd1d tasks(books-vds-stacks-to-portainer): Phase 0 probe complete, +imgproxy scope 
Phase 0 findings:
- 6 target stacks (5 original + imgproxy discovered on /usr/docker/imgproxy/)
- All bind-only, no env_file, no surprise named volumes
- Portainer PAT works via X-API-Key; endpoint 1 = books VDS, endpoint 6 = stostayer (not ours, user-confirmed)
- mongo container lost image tag (digest only) — fresh mongo:4.2 pull at recreate acceptable
- Q1 (endpoint 6) + Q2 (volume preservation) closed

Updated entity books-vds: imgproxy added to SSH-managed inventory (was missing).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 10:12:55 +03:00
vitya
fe41ee4422 backup(books-vds): daily 06:00 MSK pipeline to kreknin live 🟢 
- scripts/books-vds-backup-daily-kreknin/ — run.sh + .env.example + README
- ES path.repo bootstrap (one-time stack edit, snapshot repo 'kreknin' registered)
- curl smtps://yandex:465 email (CRLF + Date headers, no msmtp dep on CentOS 7)
- StrictHostKeyChecking=yes + pre-populated known_hosts (CentOS 7 no accept-new)
- cron /etc/cron.d/books-vds-backup live, initial sync 4.87 GB in 13m28s, ntfy push sent
- wiki: entities/books-vds.md + sources/books-vds-backup-daily-kreknin-2026-05-25.md created
- wiki: concepts/books-ssh-access → vds-kzntsv-ssh-access (was lying about books on vds-kzntsv)
- .tasks/books-vds-backup-daily-kreknin.md closed 🟢 pending phone-side verify

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 09:37:16 +03:00
vitya
22786e1865 wiki(ingest): RUVDS IIS migration + daily backup pipeline 
- entities/ruvds-iis-host (NEW) — 80.64.31.36, Win Server 2025 Core,
  25 SNI HTTPS bindings, 2/24 hostnames DNS-flipped
- sources/iis-migration-to-ruvds-2026-05-23 (NEW) — chronology,
  SSH/scp pivot после home-ISP outbound 445 block
- sources/ruvds-backup-daily-kreknin-2026-05-24 (NEW) — rclone+SFTP
  SYSTEM task daily 04:30, ntfy общий канал
- concepts/traefik-acme-json-to-iis-cert-import (NEW) — PFX + SNI
  recipe
- concepts/windows-server-2025-core-bootstrap — SMB deprecate,
  HTTP middlebox warning, HTTP/2 note; backup/cert open-Qs закрыты
- entities/windows-recovery-host — partial-cutover state,
  imgproxy SPOF carve-out, tandemmebel indefinitely здесь
- overview / index / log — catalog refresh

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 23:31:14 +03:00
vitya
b7b8e27a27 tasks(books-ssh-audit-shared-vds): close 🟢 — audit clean (1 retained, 0 revoked) 
SSH audit на shared VDS vds-kzntsv (89.253.255.94, hosts books Slovo +
Bookva в shared compose-стеке) перед Phase 3 cutover'ом tenant-split.

Findings:
- 1 retained key: vitya@DESKTOP-NSEF0UK (core dev, sole admin)
- 0 keys to revoke — никаких analyst / former employee / unknown keys
- 1 cosmetic cleanup: removed dead root authorized_keys entry (was
  duplicate of vitya's key, dead из-за `permitrootlogin no`)
- sshd hardening verified via `sshd -T` (effective config: root-no,
  password-no, kbd-no). **Gotcha** noted: raw grep of /etc/ssh/sshd_config
  shows defaults; sshd_config.d/ overrides делают effective. Future
  audits use `sshd -T`, not raw grep.
- fail2ban active, 2670 failed / 37 banned hist, currently 0
- last 7 days journalctl ssh: только vitya@94.19.247.14 (мой home IP)

Acceptance per spec (Фаза 3, шаг 2 tenant-split):
✅ authorized_keys reviewed
✅ Non-core keys revoked (N/A — none existed)
✅ Analyst keys revoked (N/A — never issued)
✅ Documented in .wiki/concepts/books-ssh-access.md

Ingest: .wiki/concepts/books-ssh-access.md — retained keys table +
sshd state + fail2ban + login history + add/revoke processes +
cross-refs. Logged + indexed.

STATUS.md ⚪ block → ARCHIVE.md (per-task file kept in .tasks/).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 15:59:50 +03:00
vitya
fdefe96d6f tasks(iis-migration-to-ruvds): pause после failed-robocopy + ingest SMB-default finding 
Temp admin (2026-05-23 09:37-18:08) начал импл iis-migration-to-ruvds,
не закончил bootstrap, упал на UNC robocopy (exit 16) и dropped tree dirty.

Session recovery:
- secrets leak fix: `.secrets/ruvds-iis.env` → `pass show ruvds-iis/full-env`
  (etap-2 discipline restored). `.secrets/` + `*.env` + `*-log.txt` + `*-size.txt`
  added to `.gitignore` чтобы не повторилось.
- root cause зафиксирован: TCP/445 closed по дефолту на fresh Win Server 2025
  Core + SMB share не создан → UNC robocopy не работает без RUVDS bootstrap.
- new concept `windows-server-2025-core-bootstrap.md` — default-blockers
  table + transfer-методов матрица (RDP-redirect / SMB / WinRM / SFTP) +
  bootstrap-чеклист 12 шагов. Recommendation = SMB inbound с source-IP
  whitelist.
- task ⚪ → 🟡 paused с concrete next-step (capacity audit, transfer-method
  confirm, RUVDS bootstrap, backup source, recreate IIS sites + conn-string
  swap, pilot kupimknigi + DNS swap).
- Open question raised: source 11 sites сумма vs RUVDS 30 GB HDD — capacity
  blocker possible (snolla одна 8.66 GB).

Не push'нуто — ждёт user grant per project-discipline Rule 4.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-23 22:07:34 +03:00
vitya
d99602894f ops(vds): Portainer-canonical rule + migrate 12 ad-hoc stacks 
Findings из сессии 2026-05-22 (user корректировка: «через портейнер!»):
deploy через ssh + docker compose up — anti-pattern, breaks ops surface
homogeneity. Все stacks на VDS должны быть Portainer-managed.

Retro-migrated 12 stacks (board-viewer, ntfy, registry, verdaccio, gitea,
owncloud, postgres, mongo, mariadb, redis, minio-imgproxy, mssql) через
JWT auth + /api/stacks/create/standalone/string API. Migration script
template в новой wiki концепции.

Skipped traefik+portainer (management plane recreate ломает access).

Gotchas закреплены в portainer-stack-management-vds.md §Gotchas:
- env_file: .env requires strip + env array в API payload
- ./ binds для sibling files (nginx.conf) absolutize → /opt/stacks/<>/
- docker login registry.kzntsv.site обязателен на VDS host
- container recreate downtime ~30-60s/stack (sequential)
- Portainer working_dir /data/compose/<id>/ не /opt/stacks/<name>/

CLAUDE.md новое правило: VDS docker stacks → Portainer canonical.
STATUS header updated; board-viewer-vds-deploy closed 🟢 как часть sweep.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 16:47:16 +03:00
vitya
adcd8bda1f tasks(minio+iis): close 🟢 — CMS image pipeline стоп-на-windows-host 
Finding: MoreThenCms.Modules.Imgproxy.dll (closed-source) содержит
hardcoded https://imgproxy.kzntsv.site URL (strings analysis из binary).
Архитектура: browser → traefik → IIS snolla → ImgproxyHandler →
server-side GET imgproxy.kzntsv.site → windows-host traefik → imgproxy-nginx
→ imgproxy → MinIO localhost:9000.

Sample URL decode: https://www.pilorama98.ru/imgproxy/<sig>/.../
czM6Ly9waWxvcmFtYTk4L3Byb2R1Y3RzL2wv... → base64 =
s3://pilorama98/products/l/<uuid>.jpg

DLL closed-source, исходника в репо нет. 4 options рассмотрены
(hosts+cert trick / nginx-relay / keep-as-is / decompile+recompile).
User decision: Option C — оставить как есть. Windows-host imgproxy/nginx
/MinIO живут indefinitely, VDS stack = standby.

minio-imgproxy-vds-migration 🟢 — VDS stack ready as standby
iis-cutover-to-vds-services 🟢 — MSSQL leg done, image pipeline stays
Decommission window-host MSSQL осtaется (48ч soak до 2026-05-24);
MinIO/imgproxy/nginx на windows — НЕ decommission.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 10:47:29 +03:00
vitya
8a30696a45 tasks(minio): clarify VDS imgproxy permanent home = imgproxy.vds.kzntsv.site 
User confirmed imgproxy.kzntsv.site занят другим сервисом на другом сервере
(не наш scope). VDS imgproxy уже на imgproxy.vds.kzntsv.site (live).

CMS audit 2026-05-22 не нашёл refs на imgproxy/kzntsv/minio в:
- snolla source code (recursive grep)
- Admin/ templates
- bin/MoreThenCms.Modules.Imgproxy.dll strings (только namespace literal)
- MoreThenCms DB SiteSettings + ModuleProperties
- Public HTML rendering (всё relative /images/ + /themes/)

DLL deployed но не wired в active flow — vestigial или invoked через
Azure storage SDK (AccountName=snolla;AccountKey=... connection string
в snolla/Web.config — Azure-compat доступ к MinIO с неустановленным
endpoint). Identification реального consumer'а — отдельная audit-задача
перед source decommission.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 10:07:43 +03:00
vitya
e17e8eb1c3 mssql+minio vds migration done — IIS snolla cutover live 
- mssql-vds-migration 🟢: Express 2022 Linux на mssql.kzntsv.site:1433,
  5 DBs restored (schema 2019→2022 auto-upgrade), 8 prod hosts 200 OK
- minio-imgproxy-vds-migration 🟡: stack live, 3 GiB / 19324 obj mirrored
  (2020→2025 upgrade via S3 API), DNS swap imgproxy.kzntsv.site pending
- iis-cutover-to-vds-services 🟡: 1 web.config edit catch-all для 11 hosts;
  MinIO leg pending DNS swap (URLs уже через imgproxy.kzntsv.site domain)
- 3 wiki concepts: mssql-on-vds, minio-imgproxy-on-vds, index updated

Findings: (1) server logins НЕ переезжают в .bak — orphan fix manual,
(2) traefik recreate-not-restart для port mapping change,
(3) PowerShell не вызывает pass bash-script (NULL pw → misleading auth fail),
(4) mc 2025 не auto-creates target buckets, (5) non-ASCII filename silent
skip в mc mirror — verify по count+size обязательно.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 09:55:17 +03:00
vitya
68706d0bee tasks(owncloud-vds-deploy): close 🟢 — 26 GB live + 60s timeout finding 
Import complete: 25 044 objs / 26 GB. 99.97% залито rclone'ом (22.2 GB
за ~2.5 ч), оставшиеся 6 файлов (4.04 GB) — 4× .pat ~221 MB + 2× .seospider
1.55+1.7 GB — упали с 502/500 на каждом rclone PUT из-за 60-секундного
HTTP timeout в reva v2.27 datagateway (hardcoded в Go http.Client.Timeout,
не env-configurable).

Workaround: VDS-side curl PUT loopback через throwaway sftp key.
Local network 142 MB/s → 4.04 GB за 28 секунд, все PROPFIND size match.

Wiki: §Gotcha 5 в concepts/ocis-on-vds-deploy-recipe.md документирует
finding + recipe для workaround. Traefik buffering middleware пробован,
не помог (likely vulcand/oxy buffer bug на больших телах) — откатил.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 07:26:07 +03:00
vitya
74cf219533 wiki(concepts): ingest ocis-on-vds-deploy-recipe + entity refresh 
New concept page documents the oCIS-on-VDS deploy pattern with the 4
non-obvious gotchas surfaced during owncloud-vds-deploy:

  1. UID mismatch — image ocis-user is 1000, host vitya is 1001 →
     compose `user: "1001:1001"` override (alt chown rejected as orphan-UID)
  2. PROXY_TLS=false — explicit per docs when reverse proxy terminates HTTPS
  3. PROXY_ENABLE_BASIC_AUTH=true — required for WebDAV/LibreGraph API
  4. LibreGraph POST /graph/v1.0/users — only way to create users (no CLI)

Plus decomposedfs storage layout, atomic-revert recipe, backup integration
note. Source: .tasks/owncloud-vds-deploy.md.

entities/vds-kzntsv.md updated: stack table + hostnames + file layout
include owncloud row.

index.md catalog + log.md ingest entry refreshed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 19:29:43 +03:00
vitya
fd8bb13e75 resilience(roadmap): workshop pass 1 — close design-task; file top-3 impl-tasks 
- close [resilience-roadmap-design] 🟢 — roadmap expanded via interactive workshop
  with user (per-service RTO/RPO matrix for 10 services, CMS backup 2-phase plan,
  IIS migration track, 4 secondary topics documented as recommendations)
- file 3 impl-tasks ⚪ ready (spawned by roadmap-design):
  - cms-stopgap-backup-daily — Phase 1, plug RPO=∞ via daily rsync to kreknin
  - mssql-minio-migration-to-vds — Phase 2 enabler, achieves RPO 1h target
  - windows-hosting-vendor-research — design-task for IIS migration off home host
- unblock [admin-infra-project-review] 🔵 → ⚪ (last blocker cleared)
- expand .wiki/concepts/future-resilient-architecture-goals.md with §Workshop pass 1
  (canonical) above original §Pre-workshop placeholder (historical)

Key user-decisions zafiksirovany in workshop:
- CMS RTO 4h / RPO 1h (not 1h/15min — overkill for one-person infra)
- MSSQL+MinIO migrate to VDS (not Windows-side managed hosting)
- MSSQL Always-On AG rejected — log shipping every 60min sufficient
- IIS long-term dies with snolla-on-node; transitional via managed Windows host
- Cloud off-site = Yandex Object Storage when needed (kreknin already off-site)
- Monitoring = Uptime Kuma on VDS, high-priority among not-filed

Acceptance per spec (user confirmed plan + concrete impl-tasks top-3 + doc committed) — met.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 16:25:04 +03:00
vitya
38f1d3358a wiki(index): refresh catalog after subtree import — 6 entities, 18 concepts, 3 sources 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 13:52:32 +03:00
vitya
3cdcdad048 cleanup: drop CMS files imported via subtree (they stay in MoreThenCms); drop stale bootstrap-manifest.md (per-project metadata from MoreThenCms's bootstrap, does not apply to admin) 2026-05-21 13:48:36 +03:00
vitya
c40418239e import: merge .wiki/concepts/ from temp prefix into existing dir (history preserved via merge+rename) 2026-05-21 13:47:59 +03:00
vitya
98bcc37d32 import: .wiki/sources/ from MoreThenCms via subtree-split 
git-subtree-dir: .wiki/sources
git-subtree-mainline: d25c0577c8
git-subtree-split: a5e96432bc
 2026-05-21 13:46:39 +03:00
vitya
d25c0577c8 import: .wiki/entities/ from MoreThenCms via subtree-split 
git-subtree-dir: .wiki/entities
git-subtree-mainline: 6d8d75474d
git-subtree-split: 24661c10fb
 2026-05-21 13:46:38 +03:00
vitya
6d8d75474d prep: clear gitkeep before subtree import 2026-05-21 13:46:33 +03:00
vitya
d558ccfaed chore: mark admin-infra-project-pointers done, unblock morecms-subtree-split 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 13:37:10 +03:00
vitya
67f7bbee74 wiki(claude): add design-context pointers for admin-infra-project 
Replaces seeded stub in Domain conventions with a 3-source pointer block:
1. concepts/admin-infra-project.md (canonical local design)
2. ~/projects/.workshop/.archive/2026-05-21-admin-infra-project.md (rationale)
3. .wiki/overview.md (orientation only)

Closes task admin-infra-project-pointers — bootstrap pointer for the
entire migration / agenda chain.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 13:37:06 +03:00
OpeItcLoc03
4c35e2a925 meta(wiki): log += ingest concepts/admin-infra-project 2026-05-21 10:21:29 +00:00
OpeItcLoc03
f1565eaff4 meta(wiki): index += concepts/admin-infra-project 2026-05-21 10:21:28 +00:00
OpeItcLoc03
531184111a meta(wiki): ingest concepts/admin-infra-project in OpeItcLoc03/admin 2026-05-21 10:21:28 +00:00
vitya
34e779e002 bootstrap: admin project skeleton (CLAUDE.md, .wiki/, .tasks/) 2026-05-21 13:17:45 +03:00