scripts(backup-notifications): unify push + email format across VDS/RUVDS/windows-host

3 host-pipelines (VDS bash, RUVDS+windows-host ps1) had drifted formats: ntfy title `VDS backup OK $D` vs `RUVDS backup OK ($D)`, tags `white_check_mark` vs `green_circle`, email subject `[VDS] backup OK` vs `RUVDS backup -- SUCCESS`. Phone-side фильтрация и desktop reading ломались за счёт inconsistency. Unified to: - ntfy push: title `<HOST> backup OK <date>`, body `<duration_human>, size=<>, snapshots=<>, dest=kreknin:<>`, tags `green_circle` (OK) / `red_circle` (FAILED). - email: subject `[<HOST>] backup <STATUS> <date>` (STATUS=OK|FAILED), body — structured Date/Duration/Size/Snapshots/Source/Dest/Components/Log. Failure body extends with `Tail (last 40 lines)`. Also imports VDS `run.sh` into repo as `scripts/vds-backup-rsync-kreknin/` — closes drift из общего `scripts/<slug>/` pattern (RUVDS+windows-host уже жили там; VDS жил только на /opt/stacks/backup/scripts/). Deploy status: - VDS: deployed via scp + sudo install, sha256=27b09ca272bb, smoke ntfy+email ✓ - RUVDS: deployed via scp + Move-Item, sha256=f3bb57a86af5, smoke ntfy+email ✓ - windows-host: deploy.ps1 + smoke-notify.ps1 готовы в scripts/, **pending elevated PS у user'а** (ACL=SYSTEM+Administrators, не пишется без UAC). Spec + decisions + completed: .tasks/unify-backup-notifications.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 07:19:15 +03:00
parent 29724d410f
commit 73ad6dd06a
9 changed files with 610 additions and 35 deletions
--- a/.tasks/NEXT_SESSION.md
+++ b/.tasks/NEXT_SESSION.md
@@ -0,0 +1,39 @@
+---
+_last_updated_: 2026-05-25T07:30:00+03:00
+session_id: 2026-05-25-unify-backup-notifications
+---
+
+# Next session handoff
+
+## Recent commits
+
+- `scripts(backup-notifications): unify push + email format across VDS/RUVDS/windows-host` (этот раунд)
+- `wiki(claude): add session-handoff trigger line`
+- `wiki(ingest): RUVDS IIS migration + daily backup pipeline`
+- `tasks(books-ssh-audit-shared-vds): close 🟢`
+- `tasks(windows-host-fallback-backup-daily): close 🟢`
+
+## Open треки
+
+| Трек | Готовность | Entry-point |
+|---|---|---|
+| `unify-backup-notifications` 🟡 | 90% — 3 scripts edited, VDS+RUVDS deployed+smoke ✓, **windows-host deploy pending elevated PS** | `.tasks/unify-backup-notifications.md` + `scripts/windows-host-fallback-backup-daily/deploy.ps1` |
+| `iis-migration-to-ruvds` 🟡 | soak (kupimknigi + emspb DNS flipped 2026-05-24) | `.tasks/iis-migration-to-ruvds.md` |
+| `infra-inventory` ⚪ | ready, not started | `.tasks/infra-inventory.md` |
+| `books-vds-bookva-bootstrap` 🔵 | blocked on victor/books tenant-split Phase 4 | `.tasks/books-vds-bookva-bootstrap.md` |
+
+## Спроси user'а
+
+- **Открывал ли elevated PowerShell и запустил `& .\scripts\windows-host-fallback-backup-daily\deploy.ps1`?** Это closes `unify-backup-notifications`.
+- **Пришли ли TEST ntfy + email от VDS + RUVDS** (2 push'а с title `[TEST] <HOST> backup OK 2026-05-25`)? Format читается одинаково?
+- **DNS swap для оставшихся 22 hostnames** (`iis-migration-to-ruvds` next-action #3) — продолжаем или ждём ещё soak'а?
+
+## Не делать (preemptive guards)
+
+- **Не push без grant'а** (project-discipline Rule 4 — session-start = ask-mode). Этот раунд commit'а — local-only пока user не разрешит push.
+- **Не trigger полный backup-run для smoke** (21-81 min wall-clock). Smoke = только notify-channel через `smoke-notify.ps1`, не full pipeline.
+- **Не оставлять windows-host deploy в подвешенном состоянии** — без deploy он сегодня ночью (03:00 MSK) пушнёт в STAROM формате, regression. После deploy → закрыть task `unify-backup-notifications`.
+
+## Memory updates за сессию
+
+- (нет на этом раунде)
--- a/.tasks/STATUS.md
+++ b/.tasks/STATUS.md
@@ -1,5 +1,20 @@
 # Admin Task Board
-_Updated: 2026-05-24 (`books-ssh-audit-shared-vds` 🟢 closed — audit clean: 1 retained vitya key, 0 revoked, removed cosmetic dead root key, sshd hardening verified via `sshd -T`. Doc в `.wiki/concepts/books-ssh-access.md`. Pre-cutover acceptance для tenant-split Phase 3 ✅. Earlier today: `windows-host-fallback-backup-daily` 🟢 closed (daily 03:00 MSK rclone SFTP windows-host → kreknin live, 23 GB / 4868 sec smoke #1). Все 3 host'а (windows + RUVDS + VDS) backup'ятся на kreknin sequentially за одну ночь. Current scope: IIS migration в soak (kupimknigi + emspb flipped, tandemmebel excluded), infra-inventory ready, остальные books-* awaits tenant-split Phase 3-4 design Direction.)_
+_Updated: 2026-05-25 (`unify-backup-notifications` 🟡 paused — 3 scripts edited под единый push+email format; VDS+RUVDS deployed + smoke notify ✓; windows-host deploy pending elevated PS у user'а. VDS `run.sh` импортирован в repo как canonical source. Pre-existing: `books-ssh-audit-shared-vds` 🟢, `windows-host-fallback-backup-daily` 🟢, все 3 daily backups live на kreknin.)_
+
+## 🟡 [unify-backup-notifications] — единый формат push + email для всех 3 backup-pipelines (VDS / RUVDS / windows-host)
+
+**Status:** paused (pending windows-host elevated deploy)
+**Where I stopped:** 2026-05-25 — 3 scripts edited + parser-check ✓. **VDS + RUVDS deployed + smoke notify ✓** (ntfy_ok + email_ok с TEST-prefix). **windows-host pending elevated deploy** — `scripts/windows-host-fallback-backup-daily/deploy.ps1` готов, user запустит из elevated PS позже. Smoke на windows-host skipped по user-decision (доверяем parser-проверке).
+**Next action:**
+1. User: open elevated PowerShell, `cd C:\Users\vitya\projects\.admin`, `& .\scripts\windows-host-fallback-backup-daily\deploy.ps1`.
+2. (Optional) `& .\scripts\windows-host-fallback-backup-daily\smoke-notify.ps1` — отправит TEST ntfy + email, verify format.
+3. Verify первой ночью (2026-05-25 03:00 MSK windows-host → 04:30 RUVDS → 05:00 VDS) что push'и + email'ы пришли с единым форматом, нет regression.
+4. Закрыть task.
+**Branch:** master
+<!-- created-by: vitya / 2026-05-25 / trigger: user-decision — unify backup notifications -->
+<!-- paused: vitya / 2026-05-25 / pending windows-host elevated deploy -->
+
+---

 <!--
  Status legend:
--- a/.tasks/unify-backup-notifications.md
+++ b/.tasks/unify-backup-notifications.md
@@ -0,0 +1,127 @@
+# unify-backup-notifications
+
+## Goal
+
+Привести notification-формат всех 3 backup-pipelines (VDS, RUVDS, windows-host) к единому виду: **один формат для ntfy push** + **один формат для email**. Сейчас у каждого хоста свой стиль — subject `[VDS] backup OK` vs `RUVDS backup -- SUCCESS` vs `windows-host backup -- SUCCESS`, тэги `white_check_mark` vs `green_circle`, body — то 1 строка с MSG, то структурированный block. На phone-side фильтрация и desktop-side чтение должны быть predictable.
+
+Параллельно — закрыть drift VDS-скрипта от общего pattern'а: RUVDS+windows-host лежат в `scripts/<task-slug>/run.ps1` локально, VDS жил только на remote (`/opt/stacks/backup/scripts/run.sh`). Импортирую как `scripts/vds-backup-rsync-kreknin/run.sh` (canonical source), правлю локально, deploy.
+
+## Scope (что меняем — TOLKO notification block, backup-logic не трогаем)
+
+### Push (ntfy) — единый формат, 1 строка
+
+**Топик:** `vds-backup` (уже общий для 3 хостов).
+**Tags:** `green_circle` (OK) / `red_circle` (FAILED).
+**Priority:** `default` / `high`.
+
+```
+Title: <HOST> backup OK <date>
+Body:  <duration_human>, size=<size>, snapshots=<count>, dest=kreknin:<path>
+
+Title: <HOST> backup FAILED <date>
+Body:  After <duration_human>: <error>. See <logpath>
+```
+
+Где `<HOST>` ∈ {`VDS`, `RUVDS`, `windows-host`}.
+`<duration_human>` = `Xm YYs` (e.g. `21m07s`, `0m45s`).
+
+### Email — единый формат, structured multi-line
+
+**Subject:** `[<HOST>] backup <STATUS> <date>` где `STATUS` ∈ {`OK`, `FAILED`}.
+
+**Body (success):**
+
+```
+<HOST> daily backup completed successfully.
+
+Date:      <date>
+Duration:  <duration_human>
+Size:      <size>
+Snapshots: <count>
+Source:    <hostname> (<ip>)
+Dest:      kreknin:<path>
+
+Components:
+- <component 1>
+- <component 2>
+...
+
+Log: <log-path>
+```
+
+**Body (failure):**
+
+```
+<HOST> daily backup FAILED.
+
+Date:     <date>
+Duration: <duration_human>
+Error:    <error>
+Source:   <hostname> (<ip>)
+Log:      <log-path>
+
+Tail (last 40 lines):
+<tail>
+```
+
+## Key files
+
+- `scripts/vds-backup-rsync-kreknin/run.sh` — bash, deployed to VDS `/opt/stacks/backup/scripts/run.sh`. Functions: `notify()` (ntfy), `email_send()` (msmtp). Callers — line 122-123 (success) + 49-53 (failure).
+- `scripts/ruvds-backup-daily-kreknin/run.ps1` — PS1, deployed to RUVDS `C:\ProgramData\backup\run.ps1`. Functions: `Notify-Ntfy` + `Notify-Email`. Callers — line 140-161 (success) + 169-170 (failure).
+- `scripts/windows-host-fallback-backup-daily/run.ps1` — PS1, deployed to windows-host `C:\ProgramData\backup\run.ps1`. Same function names. Callers — line 169-189 (success) + 197-198 (failure).
+
+## Implementation steps
+
+1. Edit VDS `run.sh`: добавить `[<HOST>] backup <STATUS> <date>` email-subject, structured body для success+failure, ntfy tags на `green_circle`/`red_circle`.
+2. Edit RUVDS `run.ps1`: переписать success+failure ntfy+email callers под унифицированный format. Поменять subject на `[RUVDS] backup OK <date>` etc.
+3. Edit windows-host `run.ps1`: same.
+4. Deploy:
+   - windows-host: `Copy-Item scripts/windows-host-.../run.ps1 → C:\ProgramData\backup\run.ps1` (elevated; user может потребоваться UAC).
+   - RUVDS: `scp` to `C:\ProgramData\backup\run.ps1` через ssh-key.
+   - VDS: `scp` + `sudo install` to `/opt/stacks/backup/scripts/run.sh`.
+5. Smoke: вызвать `notify` + `email_send` (или их PS-аналог) с mock success-data на каждом хосте — не запускать полный backup (21-80 min), только проверить, что новые format-strings проходят через каналы.
+6. Verify user-side: phone ntfy app получил, gmail получил, форматы единые.
+7. Commit + push.
+
+## Acceptance
+
+- [x] 3 scripts в `scripts/` имеют идентичный notification-block (modulo language: bash vs ps1).
+- [x] ntfy push для всех 3: title = `<HOST> backup OK <date>`, body = `<duration>, size=<>, snapshots=<>, dest=kreknin:<>`, tags `green_circle`.
+- [x] Email subject для всех 3: `[<HOST>] backup OK <date>`. Body — единый template.
+- [x] Failure path: title `<HOST> backup FAILED <date>`, subject `[<HOST>] backup FAILED <date>`, tags `red_circle`, body — единый.
+- [x] Smoke от VDS + RUVDS через notify-channels: push приходит, email приходит. (2026-05-25, см. completed steps)
+- [ ] Smoke от windows-host — **deferred** (user-decision 2026-05-25: пропустить smoke, доверяем parser-проверке).
+- [ ] **Deploy на windows-host pending** — user сделает elevated PS позже. До deploy'а windows-host остаётся на pre-unify формате.
+- [ ] User-verify: phone ntfy уведомления — TEST из VDS + RUVDS пришли, format читается?
+
+## Completed steps
+
+- [x] **2026-05-25:** spec написана + STATUS.md обновлён, task 🔴 active.
+- [x] **2026-05-25:** VDS `run.sh` импортирован из `vds.kzntsv.site:/opt/stacks/backup/scripts/run.sh` в `scripts/vds-backup-rsync-kreknin/run.sh` — closes drift из общего `scripts/<slug>/` pattern.
+- [x] **2026-05-25:** 3 scripts edited: VDS bash + RUVDS ps1 + WHOST ps1. Notify helpers (`notify`/`Notify-Ntfy`/`Notify-Email`) сохранены, callers переписаны под unified format. Parser-check ✓ всех 3 (bash -n / `[Parser]::ParseFile`).
+- [x] **2026-05-25:** VDS deployed: `scp /tmp/run.sh.new` + `sudo install -m 755 -o root -g root` → `/opt/stacks/backup/scripts/run.sh`. SHA256=`27b09ca272bb` match. Backup pre-unify в `.bak-pre-unify`.
+- [x] **2026-05-25:** RUVDS deployed: `scp` → `C:\ProgramData\backup\run.ps1.new` → `Move-Item -Force`. SHA256=`f3bb57a86af5` match. Backup в `.bak-pre-unify`.
+- [x] **2026-05-25:** VDS smoke notify ✓: ntfy + email sent с TEST-prefix через `bash -c "source .env; ..."` snippet.
+- [x] **2026-05-25:** RUVDS smoke notify ✓: SCP + run `smoke-notify.ps1` → `ntfy_ok` + `email_ok`.
+- [x] **2026-05-25:** windows-host `smoke-notify.ps1` + `deploy.ps1` подготовлены в `scripts/windows-host-fallback-backup-daily/`. **Pending elevated execution user'ом.**
+
+## Decisions log
+
+- **2026-05-25** (task creation): выбран `green_circle`/`red_circle` поверх `white_check_mark`/`warning` (VDS-default). Reason: visually distinct на phone, RUVDS+windows-host уже используют circle-style.
+- **2026-05-25**: `OK` вместо `SUCCESS` в subject — shorter, у VDS уже было `OK`.
+- **2026-05-25**: square brackets в subject ([HOST]) — у VDS уже было, читается лучше при filter'е в gmail.
+- **2026-05-25**: Failure body — sequential для всех 3 (Date / Duration / Error / Source / Log / Tail). У VDS уже был tail-of-log; экстендим на RUVDS+WHOST.
+- **2026-05-25**: VDS `run.sh` импортирован в repo как `scripts/vds-backup-rsync-kreknin/run.sh` — closes drift из общего pattern (`scripts/<slug>/`). Deploy = scp + sudo install обратно на VDS.
+
+## Open questions
+
+- [ ] **Snapshots count для RUVDS/WHOST.** VDS делает `ls -1d $DEST_BASE/20*-*-* | wc -l` через SSH. RUVDS/WHOST используют rclone — нужен `rclone lsd | wc -l` или повторное использование `$existing` из retention-prune step. Включим в edit.
+- [ ] **Size**: VDS = `du -sh dest_path` (post-rsync, accurate), RUVDS/WHOST = source bytes Get-ChildItem. Standardize? Слишком expensive делать `rclone size` на dest (extra round-trip). **Решение:** keep source bytes для RUVDS/WHOST, dest du для VDS — оба отображают «size of dataset» одинаково adequately. В email можно подписать как `Size:     <X> GB`, не различая источник.
+- [ ] **Source IP/hostname**: hardcode (VDS=`89.253.255.94`, RUVDS=`80.64.31.36`, windows-host=`94.19.247.14`) vs `$(hostname -I)` / `$env:COMPUTERNAME`. Hardcode — short-term, проще; для prod-grade нужен detect.
+
+## Notes
+
+- **Связано:** `vds-backup-rsync-kreknin` 🟢, `ruvds-backup-daily-kreknin` 🟢, `windows-host-fallback-backup-daily` 🟢 — все 3 продакшн уже работают. Эта таска — cosmetic + observability unification, не функциональная.
+- **Атомарный revert:** `git revert HEAD` + redeploy 3-х previous scripts. Pre-edit content закоммичен.
+
+<!-- created-by: vitya / 2026-05-25 / trigger: user-decision — unify push+email across VDS/RUVDS/windows-host -->
--- a/scripts/ruvds-backup-daily-kreknin/run.ps1
+++ b/scripts/ruvds-backup-daily-kreknin/run.ps1
@@ -14,6 +14,14 @@ $rcloneCfg = "$base\rclone.conf"
 $remoteBase = "kreknin:NetBackup/ruvds-iis"
 $remoteToday = "$remoteBase/$today"

+$HostLabel = 'RUVDS'
+$SourceDisplay = "RUVDS ($env:COMPUTERNAME / 80.64.31.36)"
+$DestDisplay = "kreknin:/volume1/NetBackup/ruvds-iis/$today/"
+
+function Format-Duration([int]$sec) {
+    '{0}m{1:D2}s' -f ([int]($sec / 60)), ($sec % 60)
+}
+
 if (-not (Test-Path $logDir)) { New-Item -ItemType Directory -Path $logDir -Force | Out-Null }
 Start-Transcript -Path $logFile -Append -Force | Out-Null

@@ -134,40 +142,65 @@ try {

    Remove-Item $certDir -Recurse -Force -ErrorAction SilentlyContinue

-    $duration = [int](New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds
+    $durationSec = [int](New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds
+    $durationHuman = Format-Duration $durationSec
    $srcBytes = (Get-ChildItem 'C:\sites\snolla' -Recurse -File -EA SilentlyContinue | Measure-Object Length -Sum).Sum
-    $totalGB = '{0:N2}' -f ($srcBytes / 1GB)
-    $msg = "RUVDS daily backup $today OK | $duration sec | size $totalGB GB | $exported certs"
-    Notify-Ntfy "RUVDS backup OK ($today)" $msg 'default' 'green_circle'
+    $sizeStr = '{0:N2} GB' -f ($srcBytes / 1GB)
+    $snapshotCount = if ($existing) { @($existing).Count } else { 1 }
+
+    # ntfy push (single line)
+    $ntfyBody = "$durationHuman, size=$sizeStr, snapshots=$snapshotCount, dest=$DestDisplay, certs=$exported"
+    Notify-Ntfy "$HostLabel backup OK $today" $ntfyBody 'default' 'green_circle'
+
+    # email (structured)
    $emailBody = @"
-RUVDS daily backup completed successfully.
+$HostLabel daily backup completed successfully.

 Date:      $today
-Duration: $duration seconds
-Size:     $totalGB GB
-Certs:    $exported PFX
-Source:   RUVDS ($env:COMPUTERNAME / 80.64.31.36)
-Dest:     kreknin:/volume1/NetBackup/ruvds-iis/$today/
+Duration:  $durationHuman
+Size:      $sizeStr
+Snapshots: $snapshotCount
+Source:    $SourceDisplay
+Dest:      $DestDisplay

-Components synced:
+Components:
 - sites/snolla
 - iis-config (applicationHost.config)
 - iis-backup-webconfiguration
- certs (LE PFX exports)
+- certs ($exported PFX exports)
 - ssh-config

 Log: $logFile
 "@
-    Notify-Email "RUVDS backup $today -- SUCCESS" $emailBody
-    Write-Host "`n=== DONE in $duration sec ==="
+    Notify-Email "[$HostLabel] backup OK $today" $emailBody
+    Write-Host "`n=== DONE in $durationHuman ==="

 } catch {
    $err = $_.Exception.Message
    Write-Host "`n=== FAILED: $err ===" -ForegroundColor Red
    Write-Host $_.ScriptStackTrace
-    $duration = [int](New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds
-    Notify-Ntfy "RUVDS backup FAILED ($today)" "After $duration sec: $err" 'high' 'red_circle'
-    Notify-Email "RUVDS backup $today -- FAILED" "After $duration sec: $err`n`nLog: $logFile"
+    $durationSec = [int](New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds
+    $durationHuman = Format-Duration $durationSec
+    $tailLog = ''
+    try { $tailLog = (Get-Content $logFile -Tail 40 -EA SilentlyContinue) -join "`n" } catch {}
+
+    # ntfy push (single line)
+    Notify-Ntfy "$HostLabel backup FAILED $today" "After $durationHuman`: $err. See $logFile" 'high' 'red_circle'
+
+    # email (structured)
+    $emailBody = @"
+$HostLabel daily backup FAILED.
+
+Date:     $today
+Duration: $durationHuman
+Error:    $err
+Source:   $SourceDisplay
+Log:      $logFile
+
+Tail (last 40 lines):
+$tailLog
+"@
+    Notify-Email "[$HostLabel] backup FAILED $today" $emailBody
    Stop-Transcript | Out-Null
    exit 1
 } finally {
--- a/scripts/ruvds-backup-daily-kreknin/smoke-notify.ps1
+++ b/scripts/ruvds-backup-daily-kreknin/smoke-notify.ps1
@@ -0,0 +1,65 @@
+# Smoke-test notify channels with NEW unified format.
+# Sends one TEST-tagged ntfy push + one TEST-tagged email — no backup run.
+# Used by unify-backup-notifications task to verify format/delivery without 80-min wait.
+
+$ErrorActionPreference = 'Stop'
+$base = 'C:\ProgramData\backup'
+$cfg = @{}
+Get-Content "$base\config.env" | Where-Object { $_ -match '^[A-Z_]+=' } | ForEach-Object {
+    $kv = $_ -split '=', 2
+    $cfg[$kv[0]] = $kv[1]
+}
+
+$today = Get-Date -Format 'yyyy-MM-dd'
+$HostLabel = 'RUVDS'
+$SourceDisplay = "RUVDS ($env:COMPUTERNAME / 80.64.31.36)"
+$DestDisplay = "kreknin:/volume1/NetBackup/ruvds-iis/$today/"
+$durationHuman = '1m02s'
+$sizeStr = '8.66 GB'
+$snapshotCount = 1
+$exported = 25
+$logFile = "C:\ProgramData\backup\logs\$today.log"
+
+# ntfy
+$pair = "$($cfg.NTFY_USER):$($cfg.NTFY_PASS)"
+$auth = 'Basic ' + [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($pair))
+$ntfyBody = "$durationHuman, size=$sizeStr, snapshots=$snapshotCount, dest=$DestDisplay, certs=$exported"
+try {
+    Invoke-RestMethod -Uri "$($cfg.NTFY_URL)/vds-backup" -Method POST `
+        -Headers @{ Authorization=$auth; Title="[TEST] $HostLabel backup OK $today"; Priority='default'; Tags='green_circle' } `
+        -Body $ntfyBody -ContentType 'text/plain' -ErrorAction Stop | Out-Null
+    Write-Host 'ntfy_ok'
+} catch { Write-Host ('ntfy_fail: ' + $_.Exception.Message) }
+
+# email
+$emailBody = @"
+$HostLabel daily backup completed successfully.
+
+Date:      $today
+Duration:  $durationHuman
+Size:      $sizeStr
+Snapshots: $snapshotCount
+Source:    $SourceDisplay
+Dest:      $DestDisplay
+
+Components:
+- sites/snolla
+- iis-config (applicationHost.config)
+- iis-backup-webconfiguration
+- certs ($exported PFX exports)
+- ssh-config
+
+Log: $logFile
+
+[TEST RUN — sent from unify-backup-notifications smoke]
+"@
+
+$secpass = ConvertTo-SecureString $cfg.SMTP_PASS -AsPlainText -Force
+$mailCred = New-Object PSCredential($cfg.SMTP_USER, $secpass)
+try {
+    Send-MailMessage -SmtpServer $cfg.SMTP_HOST -Port ([int]$cfg.SMTP_PORT) -UseSsl `
+        -Credential $mailCred -From $cfg.SMTP_FROM -To $cfg.OPS_NOTIFY_EMAIL `
+        -Subject "[TEST][$HostLabel] backup OK $today" -Body $emailBody -Encoding UTF8 `
+        -WarningAction SilentlyContinue -ErrorAction Stop
+    Write-Host 'email_ok'
+} catch { Write-Host ('email_fail: ' + $_.Exception.Message) }
--- a/scripts/vds-backup-rsync-kreknin/run.sh
+++ b/scripts/vds-backup-rsync-kreknin/run.sh
@@ -0,0 +1,172 @@
+#!/bin/bash
+# Daily VDS → kreknin backup. Triggered by /etc/cron.d/vds-backup at 05:00 MSK.
+set -euo pipefail
+
+ENV_FILE=/opt/stacks/backup/.env
+NTFY_ENV=/opt/stacks/ntfy/.env
+
+# shellcheck disable=SC1090
+source "$ENV_FILE"
+# shellcheck disable=SC1090
+source "$NTFY_ENV"
+
+TODAY=$(date +%Y-%m-%d)
+LOG_DIR=/var/log/vds-backup
+LOG="$LOG_DIR/$TODAY.log"
+LOCK=/tmp/vds-backup.lock
+DUMP_DIR=/tmp/vds-backup-dumps/$TODAY
+DEST_PATH="$DEST_BASE/$TODAY"
+LATEST="$DEST_BASE/latest"
+
+mkdir -p "$LOG_DIR" "$DUMP_DIR"
+exec >> "$LOG" 2>&1
+echo "===== VDS backup start $TODAY $(date -Is) ====="
+
+# single-instance lock
+exec 9>"$LOCK"
+if ! flock -n 9; then echo "already running, exiting"; exit 0; fi
+
+START=$(date +%s)
+SSH_OPTS="-i $SSH_KEY -o StrictHostKeyChecking=accept-new -o ConnectTimeout=30"
+
+HOST_LABEL=VDS
+SOURCE_DISPLAY="vds.kzntsv.site (89.253.255.94)"
+
+notify() {
+  local title="$1" msg="$2" tags="${3:-}" prio="${4:-default}"
+  curl -sS --max-time 10 -u "$NTFY_USER:$NTFY_PASS" \
+    -H "Title: $title" -H "Tags: $tags" -H "Priority: $prio" \
+    -d "$msg" "$NTFY_URL/vds-backup" > /dev/null || true
+}
+
+email_send() {
+  local subj="$1" body="$2"
+  printf "From: noreply@snolla.com\nTo: %s\nSubject: %s\n\n%s\n" \
+    "$EMAIL" "$subj" "$body" | msmtp "$EMAIL" || true
+}
+
+# Duration human-format: Xm YYs (e.g. "21m07s")
+fmt_duration() {
+  local s=$1
+  printf "%dm%02ds" $((s/60)) $((s%60))
+}
+
+on_error() {
+  local line=$1 rc=$2
+  local err="line $line, exit $rc"
+  local now=$(date +%s)
+  local dur=$((now - ${START:-now}))
+  local dur_h=$(fmt_duration "$dur")
+  local tail_log
+  tail_log=$(tail -40 "$LOG" 2>/dev/null || true)
+
+  echo "FATAL: $HOST_LABEL backup FAILED — $err"
+
+  # ntfy push (single line)
+  notify "$HOST_LABEL backup FAILED $TODAY" \
+         "After $dur_h: $err. See $LOG" \
+         red_circle high
+
+  # email (structured)
+  email_send "[$HOST_LABEL] backup FAILED $TODAY" \
+"$HOST_LABEL daily backup FAILED.
+
+Date:     $TODAY
+Duration: $dur_h
+Error:    $err
+Source:   $SOURCE_DISPLAY
+Log:      $LOG
+
+Tail (last 40 lines):
+$tail_log"
+  exit "$rc"
+}
+trap 'on_error $LINENO $?' ERR
+
+# === Step 1: DB dumps ===
+echo "--- DB dumps to $DUMP_DIR ---"
+
+docker exec postgres pg_dumpall -U postgres | gzip > "$DUMP_DIR/postgres.sql.gz"
+echo "postgres: $(stat -c %s "$DUMP_DIR/postgres.sql.gz") bytes"
+
+docker exec mariadb mariadb-dump -uroot -p"$MARIADB_ROOT_PASSWORD" --all-databases --single-transaction --quick 2>/dev/null | gzip > "$DUMP_DIR/mariadb.sql.gz"
+echo "mariadb: $(stat -c %s "$DUMP_DIR/mariadb.sql.gz") bytes"
+
+docker exec mongo mongodump --archive --quiet --ssl --sslAllowInvalidCertificates --uri="mongodb://${MONGO_INITDB_ROOT_USERNAME}:${MONGO_INITDB_ROOT_PASSWORD}@localhost:27017/?authSource=admin" 2>/dev/null | gzip > "$DUMP_DIR/mongo.archive.gz"
+echo "mongo: $(stat -c %s "$DUMP_DIR/mongo.archive.gz") bytes"
+
+docker exec redis redis-cli --tls --insecure -a "$REDIS_PASSWORD" --no-auth-warning SAVE > /dev/null
+docker cp redis:/data/dump.rdb "$DUMP_DIR/redis.rdb"
+gzip -f "$DUMP_DIR/redis.rdb"
+echo "redis: $(stat -c %s "$DUMP_DIR/redis.rdb.gz") bytes"
+
+# === Step 2: rsync to kreknin ===
+echo "--- rsync to $DEST_USER@$DEST_HOST:$DEST_PATH ---"
+ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "mkdir -p '$DEST_PATH'"
+
+LINK_DEST_ARG=""
+if ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "test -L '$LATEST' || test -d '$LATEST'" 2>/dev/null; then
+  LINK_DEST_ARG="--link-dest=$LATEST"
+fi
+
+rsync -aHh --info=stats2 --delete $LINK_DEST_ARG \
+  -e "ssh $SSH_OPTS" \
+  /opt/stacks/gitea \
+  /opt/stacks/verdaccio/storage \
+  /opt/stacks/verdaccio/config \
+  /opt/stacks/registry \
+  /opt/stacks/traefik \
+  /opt/stacks/portainer \
+  /opt/stacks/ntfy \
+  /opt/stacks/backup \
+  /opt/stacks/owncloud \
+  /etc/ssh \
+  /etc/ufw \
+  /etc/hosts \
+  /etc/docker \
+  "$DUMP_DIR" \
+  /home/vitya/.ssh \
+  "$DEST_USER@$DEST_HOST:$DEST_PATH/" || true
+
+# Update latest symlink
+ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "rm -f '$LATEST' && ln -s '$DEST_PATH' '$LATEST'"
+
+# === Step 3: retention prune ===
+echo "--- prune snapshots older than $RETENTION_DAYS days ---"
+ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "cd '$DEST_BASE' && ls -1d 20*-*-* 2>/dev/null | sort | head -n -$RETENTION_DAYS | xargs -r rm -rf"
+
+# === Step 4: local cleanup ===
+rm -rf "$DUMP_DIR"
+
+# === Step 5: report ===
+END=$(date +%s)
+DURATION=$((END - START))
+SIZE=$(ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "du -sh '$DEST_PATH' 2>/dev/null | cut -f1" || echo "?")
+SNAPSHOT_COUNT=$(ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "ls -1d $DEST_BASE/20*-*-* 2>/dev/null | wc -l" || echo "?")
+DURATION_HUMAN=$(fmt_duration "$DURATION")
+
+# ntfy push (single line)
+NTFY_BODY="$DURATION_HUMAN, size=$SIZE, snapshots=$SNAPSHOT_COUNT, dest=kreknin:$DEST_PATH"
+notify "$HOST_LABEL backup OK $TODAY" "$NTFY_BODY" green_circle
+
+# email (structured)
+EMAIL_BODY="$HOST_LABEL daily backup completed successfully.
+
+Date:      $TODAY
+Duration:  $DURATION_HUMAN
+Size:      $SIZE
+Snapshots: $SNAPSHOT_COUNT
+Source:    $SOURCE_DISPLAY
+Dest:      kreknin:$DEST_PATH
+
+Components:
+- /opt/stacks/{gitea,verdaccio,registry,traefik,portainer,ntfy,backup,owncloud}
+- /etc/{ssh,ufw,hosts,docker}
+- /home/vitya/.ssh
+- DB dumps: postgres, mariadb, mongo, redis
+
+Log: $LOG"
+email_send "[$HOST_LABEL] backup OK $TODAY" "$EMAIL_BODY"
+
+echo "$HOST_LABEL backup OK: $NTFY_BODY"
+echo "===== $HOST_LABEL backup done $(date -Is) ====="
--- a/scripts/windows-host-fallback-backup-daily/deploy.ps1
+++ b/scripts/windows-host-fallback-backup-daily/deploy.ps1
@@ -0,0 +1,27 @@
+# Deploy local repo run.ps1 -> C:\ProgramData\backup\run.ps1 with .bak rollback.
+# MUST be run elevated (ProgramData ACL = SYSTEM + Administrators).
+# Usage: open elevated PowerShell, cd to repo root, then:  & .\scripts\windows-host-fallback-backup-daily\deploy.ps1
+
+$ErrorActionPreference = 'Stop'
+$repoSrc = Join-Path $PSScriptRoot 'run.ps1'
+$dst = 'C:\ProgramData\backup\run.ps1'
+$bak = "$dst.bak-pre-unify"
+
+if (-not (Test-Path $repoSrc)) { throw "source missing: $repoSrc" }
+if (-not (Test-Path $dst))     { throw "destination missing: $dst (windows-host backup not initialized?)" }
+
+# Backup current (if no pre-unify backup yet)
+if (-not (Test-Path $bak)) {
+    Copy-Item $dst $bak -Force
+    Write-Host "  backup: $bak"
+} else {
+    Write-Host "  backup already present: $bak"
+}
+
+Copy-Item $repoSrc $dst -Force
+$srcHash = (Get-FileHash $repoSrc -Algorithm SHA256).Hash
+$dstHash = (Get-FileHash $dst    -Algorithm SHA256).Hash
+if ($srcHash -ne $dstHash) { throw "hash mismatch after copy: src=$srcHash dst=$dstHash" }
+Write-Host "  deployed:  $dst"
+Write-Host "  sha256:    $($dstHash.Substring(0,12))"
+Write-Host "  mtime:     $((Get-Item $dst).LastWriteTime)"
--- a/scripts/windows-host-fallback-backup-daily/run.ps1
+++ b/scripts/windows-host-fallback-backup-daily/run.ps1
@@ -31,6 +31,14 @@ $rcloneCfg = "$base\rclone.conf"
 $remoteBase = "kreknin:NetBackup/windows-host"
 $remoteToday = "$remoteBase/$today"

+$HostLabel = 'windows-host'
+$SourceDisplay = "windows-host ($env:COMPUTERNAME / 94.19.247.14)"
+$DestDisplay = "kreknin:/volume1/NetBackup/windows-host/$today/"
+
+function Format-Duration([int]$sec) {
+    '{0}m{1:D2}s' -f ([int]($sec / 60)), ($sec % 60)
+}
+
 if (-not (Test-Path $logDir)) { New-Item -ItemType Directory -Path $logDir -Force | Out-Null }
 Start-Transcript -Path $logFile -Append -Force | Out-Null

@@ -161,23 +169,29 @@ try {

    Remove-Item $mssqlLocalDir -Recurse -Force -ErrorAction SilentlyContinue

-    $duration = [int](New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds
+    $durationSec = [int](New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds
+    $durationHuman = Format-Duration $durationSec
    $sitesBytes = (Get-ChildItem 'C:\sites' -Recurse -File -EA SilentlyContinue | Measure-Object Length -Sum).Sum
    $minioBytes = (Get-ChildItem 'C:\Users\vitya\projects\docker\diskstation\minio\data' -Recurse -File -EA SilentlyContinue | Measure-Object Length -Sum).Sum
-    $totalGB = '{0:N2}' -f (($sitesBytes + $minioBytes) / 1GB)
+    $sizeStr = '{0:N2} GB' -f (($sitesBytes + $minioBytes) / 1GB)
+    $snapshotCount = if ($existing) { @($existing).Count } else { 1 }

-    $msg = "windows-host daily backup $today OK | $duration sec | $totalGB GB (sites+minio) | 5 DBs"
-    Notify-Ntfy "windows-host backup OK ($today)" $msg 'default' 'green_circle'
+    # ntfy push (single line)
+    $ntfyBody = "$durationHuman, size=$sizeStr, snapshots=$snapshotCount, dest=$DestDisplay"
+    Notify-Ntfy "$HostLabel backup OK $today" $ntfyBody 'default' 'green_circle'
+
+    # email (structured)
    $emailBody = @"
-windows-host (DESKTOP-NSEF0UK) daily backup completed successfully.
+$HostLabel daily backup completed successfully.

 Date:      $today
-Duration: $duration seconds
-Size:     $totalGB GB (sites+minio data; mssql .bak compressed separately)
-Source:   windows-host (DESKTOP-NSEF0UK / 94.19.247.14)
-Dest:     kreknin:/volume1/NetBackup/windows-host/$today/
+Duration:  $durationHuman
+Size:      $sizeStr
+Snapshots: $snapshotCount
+Source:    $SourceDisplay
+Dest:      $DestDisplay

-Components synced:
+Components:
 - mssql (5 DBs: MoreThenCms, StayerCalculator, StayerPrice, stostayer, TireService)
 - sites (C:\sites\)
 - minio (data dir)
@@ -186,16 +200,35 @@ Components synced:

 Log: $logFile
 "@
-    Notify-Email "windows-host backup $today -- SUCCESS" $emailBody
-    Write-Host "`n=== DONE in $duration sec ==="
+    Notify-Email "[$HostLabel] backup OK $today" $emailBody
+    Write-Host "`n=== DONE in $durationHuman ==="

 } catch {
    $err = $_.Exception.Message
    Write-Host "`n=== FAILED: $err ===" -ForegroundColor Red
    Write-Host $_.ScriptStackTrace
-    $duration = [int](New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds
-    Notify-Ntfy "windows-host backup FAILED ($today)" "After $duration sec: $err" 'high' 'red_circle'
-    Notify-Email "windows-host backup $today -- FAILED" "After $duration sec: $err`n`nLog: $logFile"
+    $durationSec = [int](New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds
+    $durationHuman = Format-Duration $durationSec
+    $tailLog = ''
+    try { $tailLog = (Get-Content $logFile -Tail 40 -EA SilentlyContinue) -join "`n" } catch {}
+
+    # ntfy push (single line)
+    Notify-Ntfy "$HostLabel backup FAILED $today" "After $durationHuman`: $err. See $logFile" 'high' 'red_circle'
+
+    # email (structured)
+    $emailBody = @"
+$HostLabel daily backup FAILED.
+
+Date:     $today
+Duration: $durationHuman
+Error:    $err
+Source:   $SourceDisplay
+Log:      $logFile
+
+Tail (last 40 lines):
+$tailLog
+"@
+    Notify-Email "[$HostLabel] backup FAILED $today" $emailBody
    Stop-Transcript | Out-Null
    exit 1
 } finally {
--- a/scripts/windows-host-fallback-backup-daily/smoke-notify.ps1
+++ b/scripts/windows-host-fallback-backup-daily/smoke-notify.ps1
@@ -0,0 +1,64 @@
+# Smoke-test notify channels with NEW unified format.
+# Sends one TEST-tagged ntfy push + one TEST-tagged email — no backup run.
+# MUST be run elevated (config.env ACL = SYSTEM + Administrators).
+
+$ErrorActionPreference = 'Stop'
+$base = 'C:\ProgramData\backup'
+$cfg = @{}
+Get-Content "$base\config.env" | Where-Object { $_ -match '^[A-Z_]+=' } | ForEach-Object {
+    $kv = $_ -split '=', 2
+    $cfg[$kv[0]] = $kv[1]
+}
+
+$today = Get-Date -Format 'yyyy-MM-dd'
+$HostLabel = 'windows-host'
+$SourceDisplay = "windows-host ($env:COMPUTERNAME / 94.19.247.14)"
+$DestDisplay = "kreknin:/volume1/NetBackup/windows-host/$today/"
+$durationHuman = '81m08s'
+$sizeStr = '23.10 GB'
+$snapshotCount = 1
+$logFile = "C:\ProgramData\backup\logs\$today.log"
+
+# ntfy
+$pair = "$($cfg.NTFY_USER):$($cfg.NTFY_PASS)"
+$auth = 'Basic ' + [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($pair))
+$ntfyBody = "$durationHuman, size=$sizeStr, snapshots=$snapshotCount, dest=$DestDisplay"
+try {
+    Invoke-RestMethod -Uri "$($cfg.NTFY_URL)/vds-backup" -Method POST `
+        -Headers @{ Authorization=$auth; Title="[TEST] $HostLabel backup OK $today"; Priority='default'; Tags='green_circle' } `
+        -Body $ntfyBody -ContentType 'text/plain' -ErrorAction Stop | Out-Null
+    Write-Host 'ntfy_ok'
+} catch { Write-Host ('ntfy_fail: ' + $_.Exception.Message) }
+
+# email
+$emailBody = @"
+$HostLabel daily backup completed successfully.
+
+Date:      $today
+Duration:  $durationHuman
+Size:      $sizeStr
+Snapshots: $snapshotCount
+Source:    $SourceDisplay
+Dest:      $DestDisplay
+
+Components:
+- mssql (5 DBs: MoreThenCms, StayerCalculator, StayerPrice, stostayer, TireService)
+- sites (C:\sites\)
+- minio (data dir)
+- traefik (config + acme.json)
+- iis-config (applicationHost.config) + iis-backup-webconfiguration
+
+Log: $logFile
+
+[TEST RUN — sent from unify-backup-notifications smoke]
+"@
+
+$secpass = ConvertTo-SecureString $cfg.SMTP_PASS -AsPlainText -Force
+$mailCred = New-Object PSCredential($cfg.SMTP_USER, $secpass)
+try {
+    Send-MailMessage -SmtpServer $cfg.SMTP_HOST -Port ([int]$cfg.SMTP_PORT) -UseSsl `
+        -Credential $mailCred -From $cfg.SMTP_FROM -To $cfg.OPS_NOTIFY_EMAIL `
+        -Subject "[TEST][$HostLabel] backup OK $today" -Body $emailBody -Encoding UTF8 `
+        -WarningAction SilentlyContinue -ErrorAction Stop
+    Write-Host 'email_ok'
+} catch { Write-Host ('email_fail: ' + $_.Exception.Message) }