From 73ad6dd06ac81cd122ceee5945d99d328a27d06a Mon Sep 17 00:00:00 2001 From: vitya Date: Mon, 25 May 2026 07:19:15 +0300 Subject: [PATCH] scripts(backup-notifications): unify push + email format across VDS/RUVDS/windows-host MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 3 host-pipelines (VDS bash, RUVDS+windows-host ps1) had drifted formats: ntfy title `VDS backup OK $D` vs `RUVDS backup OK ($D)`, tags `white_check_mark` vs `green_circle`, email subject `[VDS] backup OK` vs `RUVDS backup -- SUCCESS`. Phone-side фильтрация и desktop reading ломались за счёт inconsistency. Unified to: - ntfy push: title ` backup OK `, body `, size=<>, snapshots=<>, dest=kreknin:<>`, tags `green_circle` (OK) / `red_circle` (FAILED). - email: subject `[] backup ` (STATUS=OK|FAILED), body — structured Date/Duration/Size/Snapshots/Source/Dest/Components/Log. Failure body extends with `Tail (last 40 lines)`. Also imports VDS `run.sh` into repo as `scripts/vds-backup-rsync-kreknin/` — closes drift из общего `scripts//` pattern (RUVDS+windows-host уже жили там; VDS жил только на /opt/stacks/backup/scripts/). Deploy status: - VDS: deployed via scp + sudo install, sha256=27b09ca272bb, smoke ntfy+email ✓ - RUVDS: deployed via scp + Move-Item, sha256=f3bb57a86af5, smoke ntfy+email ✓ - windows-host: deploy.ps1 + smoke-notify.ps1 готовы в scripts/, **pending elevated PS у user'а** (ACL=SYSTEM+Administrators, не пишется без UAC). Spec + decisions + completed: .tasks/unify-backup-notifications.md. Co-Authored-By: Claude Opus 4.7 (1M context) --- .tasks/NEXT_SESSION.md | 39 ++++ .tasks/STATUS.md | 17 +- .tasks/unify-backup-notifications.md | 127 +++++++++++++ scripts/ruvds-backup-daily-kreknin/run.ps1 | 69 +++++-- .../smoke-notify.ps1 | 65 +++++++ scripts/vds-backup-rsync-kreknin/run.sh | 172 ++++++++++++++++++ .../deploy.ps1 | 27 +++ .../run.ps1 | 65 +++++-- .../smoke-notify.ps1 | 64 +++++++ 9 files changed, 610 insertions(+), 35 deletions(-) create mode 100644 .tasks/NEXT_SESSION.md create mode 100644 .tasks/unify-backup-notifications.md create mode 100644 scripts/ruvds-backup-daily-kreknin/smoke-notify.ps1 create mode 100644 scripts/vds-backup-rsync-kreknin/run.sh create mode 100644 scripts/windows-host-fallback-backup-daily/deploy.ps1 create mode 100644 scripts/windows-host-fallback-backup-daily/smoke-notify.ps1 diff --git a/.tasks/NEXT_SESSION.md b/.tasks/NEXT_SESSION.md new file mode 100644 index 0000000..61db490 --- /dev/null +++ b/.tasks/NEXT_SESSION.md @@ -0,0 +1,39 @@ +--- +_last_updated_: 2026-05-25T07:30:00+03:00 +session_id: 2026-05-25-unify-backup-notifications +--- + +# Next session handoff + +## Recent commits + +- `scripts(backup-notifications): unify push + email format across VDS/RUVDS/windows-host` (этот раунд) +- `wiki(claude): add session-handoff trigger line` +- `wiki(ingest): RUVDS IIS migration + daily backup pipeline` +- `tasks(books-ssh-audit-shared-vds): close 🟢` +- `tasks(windows-host-fallback-backup-daily): close 🟢` + +## Open треки + +| Трек | Готовность | Entry-point | +|---|---|---| +| `unify-backup-notifications` 🟡 | 90% — 3 scripts edited, VDS+RUVDS deployed+smoke ✓, **windows-host deploy pending elevated PS** | `.tasks/unify-backup-notifications.md` + `scripts/windows-host-fallback-backup-daily/deploy.ps1` | +| `iis-migration-to-ruvds` 🟡 | soak (kupimknigi + emspb DNS flipped 2026-05-24) | `.tasks/iis-migration-to-ruvds.md` | +| `infra-inventory` ⚪ | ready, not started | `.tasks/infra-inventory.md` | +| `books-vds-bookva-bootstrap` 🔵 | blocked on victor/books tenant-split Phase 4 | `.tasks/books-vds-bookva-bootstrap.md` | + +## Спроси user'а + +- **Открывал ли elevated PowerShell и запустил `& .\scripts\windows-host-fallback-backup-daily\deploy.ps1`?** Это closes `unify-backup-notifications`. +- **Пришли ли TEST ntfy + email от VDS + RUVDS** (2 push'а с title `[TEST] backup OK 2026-05-25`)? Format читается одинаково? +- **DNS swap для оставшихся 22 hostnames** (`iis-migration-to-ruvds` next-action #3) — продолжаем или ждём ещё soak'а? + +## Не делать (preemptive guards) + +- **Не push без grant'а** (project-discipline Rule 4 — session-start = ask-mode). Этот раунд commit'а — local-only пока user не разрешит push. +- **Не trigger полный backup-run для smoke** (21-81 min wall-clock). Smoke = только notify-channel через `smoke-notify.ps1`, не full pipeline. +- **Не оставлять windows-host deploy в подвешенном состоянии** — без deploy он сегодня ночью (03:00 MSK) пушнёт в STAROM формате, regression. После deploy → закрыть task `unify-backup-notifications`. + +## Memory updates за сессию + +- (нет на этом раунде) diff --git a/.tasks/STATUS.md b/.tasks/STATUS.md index 39f9507..989c68e 100644 --- a/.tasks/STATUS.md +++ b/.tasks/STATUS.md @@ -1,5 +1,20 @@ # Admin Task Board -_Updated: 2026-05-24 (`books-ssh-audit-shared-vds` 🟢 closed — audit clean: 1 retained vitya key, 0 revoked, removed cosmetic dead root key, sshd hardening verified via `sshd -T`. Doc в `.wiki/concepts/books-ssh-access.md`. Pre-cutover acceptance для tenant-split Phase 3 ✅. Earlier today: `windows-host-fallback-backup-daily` 🟢 closed (daily 03:00 MSK rclone SFTP windows-host → kreknin live, 23 GB / 4868 sec smoke #1). Все 3 host'а (windows + RUVDS + VDS) backup'ятся на kreknin sequentially за одну ночь. Current scope: IIS migration в soak (kupimknigi + emspb flipped, tandemmebel excluded), infra-inventory ready, остальные books-* awaits tenant-split Phase 3-4 design Direction.)_ +_Updated: 2026-05-25 (`unify-backup-notifications` 🟡 paused — 3 scripts edited под единый push+email format; VDS+RUVDS deployed + smoke notify ✓; windows-host deploy pending elevated PS у user'а. VDS `run.sh` импортирован в repo как canonical source. Pre-existing: `books-ssh-audit-shared-vds` 🟢, `windows-host-fallback-backup-daily` 🟢, все 3 daily backups live на kreknin.)_ + +## 🟡 [unify-backup-notifications] — единый формат push + email для всех 3 backup-pipelines (VDS / RUVDS / windows-host) + +**Status:** paused (pending windows-host elevated deploy) +**Where I stopped:** 2026-05-25 — 3 scripts edited + parser-check ✓. **VDS + RUVDS deployed + smoke notify ✓** (ntfy_ok + email_ok с TEST-prefix). **windows-host pending elevated deploy** — `scripts/windows-host-fallback-backup-daily/deploy.ps1` готов, user запустит из elevated PS позже. Smoke на windows-host skipped по user-decision (доверяем parser-проверке). +**Next action:** +1. User: open elevated PowerShell, `cd C:\Users\vitya\projects\.admin`, `& .\scripts\windows-host-fallback-backup-daily\deploy.ps1`. +2. (Optional) `& .\scripts\windows-host-fallback-backup-daily\smoke-notify.ps1` — отправит TEST ntfy + email, verify format. +3. Verify первой ночью (2026-05-25 03:00 MSK windows-host → 04:30 RUVDS → 05:00 VDS) что push'и + email'ы пришли с единым форматом, нет regression. +4. Закрыть task. +**Branch:** master + + + +--- diff --git a/scripts/ruvds-backup-daily-kreknin/run.ps1 b/scripts/ruvds-backup-daily-kreknin/run.ps1 index 525f36c..c5da4c5 100644 --- a/scripts/ruvds-backup-daily-kreknin/run.ps1 +++ b/scripts/ruvds-backup-daily-kreknin/run.ps1 @@ -14,6 +14,14 @@ $rcloneCfg = "$base\rclone.conf" $remoteBase = "kreknin:NetBackup/ruvds-iis" $remoteToday = "$remoteBase/$today" +$HostLabel = 'RUVDS' +$SourceDisplay = "RUVDS ($env:COMPUTERNAME / 80.64.31.36)" +$DestDisplay = "kreknin:/volume1/NetBackup/ruvds-iis/$today/" + +function Format-Duration([int]$sec) { + '{0}m{1:D2}s' -f ([int]($sec / 60)), ($sec % 60) +} + if (-not (Test-Path $logDir)) { New-Item -ItemType Directory -Path $logDir -Force | Out-Null } Start-Transcript -Path $logFile -Append -Force | Out-Null @@ -134,40 +142,65 @@ try { Remove-Item $certDir -Recurse -Force -ErrorAction SilentlyContinue - $duration = [int](New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds + $durationSec = [int](New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds + $durationHuman = Format-Duration $durationSec $srcBytes = (Get-ChildItem 'C:\sites\snolla' -Recurse -File -EA SilentlyContinue | Measure-Object Length -Sum).Sum - $totalGB = '{0:N2}' -f ($srcBytes / 1GB) - $msg = "RUVDS daily backup $today OK | $duration sec | size $totalGB GB | $exported certs" - Notify-Ntfy "RUVDS backup OK ($today)" $msg 'default' 'green_circle' + $sizeStr = '{0:N2} GB' -f ($srcBytes / 1GB) + $snapshotCount = if ($existing) { @($existing).Count } else { 1 } + + # ntfy push (single line) + $ntfyBody = "$durationHuman, size=$sizeStr, snapshots=$snapshotCount, dest=$DestDisplay, certs=$exported" + Notify-Ntfy "$HostLabel backup OK $today" $ntfyBody 'default' 'green_circle' + + # email (structured) $emailBody = @" -RUVDS daily backup completed successfully. +$HostLabel daily backup completed successfully. -Date: $today -Duration: $duration seconds -Size: $totalGB GB -Certs: $exported PFX -Source: RUVDS ($env:COMPUTERNAME / 80.64.31.36) -Dest: kreknin:/volume1/NetBackup/ruvds-iis/$today/ +Date: $today +Duration: $durationHuman +Size: $sizeStr +Snapshots: $snapshotCount +Source: $SourceDisplay +Dest: $DestDisplay -Components synced: +Components: - sites/snolla - iis-config (applicationHost.config) - iis-backup-webconfiguration -- certs (LE PFX exports) +- certs ($exported PFX exports) - ssh-config Log: $logFile "@ - Notify-Email "RUVDS backup $today -- SUCCESS" $emailBody - Write-Host "`n=== DONE in $duration sec ===" + Notify-Email "[$HostLabel] backup OK $today" $emailBody + Write-Host "`n=== DONE in $durationHuman ===" } catch { $err = $_.Exception.Message Write-Host "`n=== FAILED: $err ===" -ForegroundColor Red Write-Host $_.ScriptStackTrace - $duration = [int](New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds - Notify-Ntfy "RUVDS backup FAILED ($today)" "After $duration sec: $err" 'high' 'red_circle' - Notify-Email "RUVDS backup $today -- FAILED" "After $duration sec: $err`n`nLog: $logFile" + $durationSec = [int](New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds + $durationHuman = Format-Duration $durationSec + $tailLog = '' + try { $tailLog = (Get-Content $logFile -Tail 40 -EA SilentlyContinue) -join "`n" } catch {} + + # ntfy push (single line) + Notify-Ntfy "$HostLabel backup FAILED $today" "After $durationHuman`: $err. See $logFile" 'high' 'red_circle' + + # email (structured) + $emailBody = @" +$HostLabel daily backup FAILED. + +Date: $today +Duration: $durationHuman +Error: $err +Source: $SourceDisplay +Log: $logFile + +Tail (last 40 lines): +$tailLog +"@ + Notify-Email "[$HostLabel] backup FAILED $today" $emailBody Stop-Transcript | Out-Null exit 1 } finally { diff --git a/scripts/ruvds-backup-daily-kreknin/smoke-notify.ps1 b/scripts/ruvds-backup-daily-kreknin/smoke-notify.ps1 new file mode 100644 index 0000000..e451b0a --- /dev/null +++ b/scripts/ruvds-backup-daily-kreknin/smoke-notify.ps1 @@ -0,0 +1,65 @@ +# Smoke-test notify channels with NEW unified format. +# Sends one TEST-tagged ntfy push + one TEST-tagged email — no backup run. +# Used by unify-backup-notifications task to verify format/delivery without 80-min wait. + +$ErrorActionPreference = 'Stop' +$base = 'C:\ProgramData\backup' +$cfg = @{} +Get-Content "$base\config.env" | Where-Object { $_ -match '^[A-Z_]+=' } | ForEach-Object { + $kv = $_ -split '=', 2 + $cfg[$kv[0]] = $kv[1] +} + +$today = Get-Date -Format 'yyyy-MM-dd' +$HostLabel = 'RUVDS' +$SourceDisplay = "RUVDS ($env:COMPUTERNAME / 80.64.31.36)" +$DestDisplay = "kreknin:/volume1/NetBackup/ruvds-iis/$today/" +$durationHuman = '1m02s' +$sizeStr = '8.66 GB' +$snapshotCount = 1 +$exported = 25 +$logFile = "C:\ProgramData\backup\logs\$today.log" + +# ntfy +$pair = "$($cfg.NTFY_USER):$($cfg.NTFY_PASS)" +$auth = 'Basic ' + [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($pair)) +$ntfyBody = "$durationHuman, size=$sizeStr, snapshots=$snapshotCount, dest=$DestDisplay, certs=$exported" +try { + Invoke-RestMethod -Uri "$($cfg.NTFY_URL)/vds-backup" -Method POST ` + -Headers @{ Authorization=$auth; Title="[TEST] $HostLabel backup OK $today"; Priority='default'; Tags='green_circle' } ` + -Body $ntfyBody -ContentType 'text/plain' -ErrorAction Stop | Out-Null + Write-Host 'ntfy_ok' +} catch { Write-Host ('ntfy_fail: ' + $_.Exception.Message) } + +# email +$emailBody = @" +$HostLabel daily backup completed successfully. + +Date: $today +Duration: $durationHuman +Size: $sizeStr +Snapshots: $snapshotCount +Source: $SourceDisplay +Dest: $DestDisplay + +Components: +- sites/snolla +- iis-config (applicationHost.config) +- iis-backup-webconfiguration +- certs ($exported PFX exports) +- ssh-config + +Log: $logFile + +[TEST RUN — sent from unify-backup-notifications smoke] +"@ + +$secpass = ConvertTo-SecureString $cfg.SMTP_PASS -AsPlainText -Force +$mailCred = New-Object PSCredential($cfg.SMTP_USER, $secpass) +try { + Send-MailMessage -SmtpServer $cfg.SMTP_HOST -Port ([int]$cfg.SMTP_PORT) -UseSsl ` + -Credential $mailCred -From $cfg.SMTP_FROM -To $cfg.OPS_NOTIFY_EMAIL ` + -Subject "[TEST][$HostLabel] backup OK $today" -Body $emailBody -Encoding UTF8 ` + -WarningAction SilentlyContinue -ErrorAction Stop + Write-Host 'email_ok' +} catch { Write-Host ('email_fail: ' + $_.Exception.Message) } diff --git a/scripts/vds-backup-rsync-kreknin/run.sh b/scripts/vds-backup-rsync-kreknin/run.sh new file mode 100644 index 0000000..71f4e20 --- /dev/null +++ b/scripts/vds-backup-rsync-kreknin/run.sh @@ -0,0 +1,172 @@ +#!/bin/bash +# Daily VDS → kreknin backup. Triggered by /etc/cron.d/vds-backup at 05:00 MSK. +set -euo pipefail + +ENV_FILE=/opt/stacks/backup/.env +NTFY_ENV=/opt/stacks/ntfy/.env + +# shellcheck disable=SC1090 +source "$ENV_FILE" +# shellcheck disable=SC1090 +source "$NTFY_ENV" + +TODAY=$(date +%Y-%m-%d) +LOG_DIR=/var/log/vds-backup +LOG="$LOG_DIR/$TODAY.log" +LOCK=/tmp/vds-backup.lock +DUMP_DIR=/tmp/vds-backup-dumps/$TODAY +DEST_PATH="$DEST_BASE/$TODAY" +LATEST="$DEST_BASE/latest" + +mkdir -p "$LOG_DIR" "$DUMP_DIR" +exec >> "$LOG" 2>&1 +echo "===== VDS backup start $TODAY $(date -Is) =====" + +# single-instance lock +exec 9>"$LOCK" +if ! flock -n 9; then echo "already running, exiting"; exit 0; fi + +START=$(date +%s) +SSH_OPTS="-i $SSH_KEY -o StrictHostKeyChecking=accept-new -o ConnectTimeout=30" + +HOST_LABEL=VDS +SOURCE_DISPLAY="vds.kzntsv.site (89.253.255.94)" + +notify() { + local title="$1" msg="$2" tags="${3:-}" prio="${4:-default}" + curl -sS --max-time 10 -u "$NTFY_USER:$NTFY_PASS" \ + -H "Title: $title" -H "Tags: $tags" -H "Priority: $prio" \ + -d "$msg" "$NTFY_URL/vds-backup" > /dev/null || true +} + +email_send() { + local subj="$1" body="$2" + printf "From: noreply@snolla.com\nTo: %s\nSubject: %s\n\n%s\n" \ + "$EMAIL" "$subj" "$body" | msmtp "$EMAIL" || true +} + +# Duration human-format: Xm YYs (e.g. "21m07s") +fmt_duration() { + local s=$1 + printf "%dm%02ds" $((s/60)) $((s%60)) +} + +on_error() { + local line=$1 rc=$2 + local err="line $line, exit $rc" + local now=$(date +%s) + local dur=$((now - ${START:-now})) + local dur_h=$(fmt_duration "$dur") + local tail_log + tail_log=$(tail -40 "$LOG" 2>/dev/null || true) + + echo "FATAL: $HOST_LABEL backup FAILED — $err" + + # ntfy push (single line) + notify "$HOST_LABEL backup FAILED $TODAY" \ + "After $dur_h: $err. See $LOG" \ + red_circle high + + # email (structured) + email_send "[$HOST_LABEL] backup FAILED $TODAY" \ +"$HOST_LABEL daily backup FAILED. + +Date: $TODAY +Duration: $dur_h +Error: $err +Source: $SOURCE_DISPLAY +Log: $LOG + +Tail (last 40 lines): +$tail_log" + exit "$rc" +} +trap 'on_error $LINENO $?' ERR + +# === Step 1: DB dumps === +echo "--- DB dumps to $DUMP_DIR ---" + +docker exec postgres pg_dumpall -U postgres | gzip > "$DUMP_DIR/postgres.sql.gz" +echo "postgres: $(stat -c %s "$DUMP_DIR/postgres.sql.gz") bytes" + +docker exec mariadb mariadb-dump -uroot -p"$MARIADB_ROOT_PASSWORD" --all-databases --single-transaction --quick 2>/dev/null | gzip > "$DUMP_DIR/mariadb.sql.gz" +echo "mariadb: $(stat -c %s "$DUMP_DIR/mariadb.sql.gz") bytes" + +docker exec mongo mongodump --archive --quiet --ssl --sslAllowInvalidCertificates --uri="mongodb://${MONGO_INITDB_ROOT_USERNAME}:${MONGO_INITDB_ROOT_PASSWORD}@localhost:27017/?authSource=admin" 2>/dev/null | gzip > "$DUMP_DIR/mongo.archive.gz" +echo "mongo: $(stat -c %s "$DUMP_DIR/mongo.archive.gz") bytes" + +docker exec redis redis-cli --tls --insecure -a "$REDIS_PASSWORD" --no-auth-warning SAVE > /dev/null +docker cp redis:/data/dump.rdb "$DUMP_DIR/redis.rdb" +gzip -f "$DUMP_DIR/redis.rdb" +echo "redis: $(stat -c %s "$DUMP_DIR/redis.rdb.gz") bytes" + +# === Step 2: rsync to kreknin === +echo "--- rsync to $DEST_USER@$DEST_HOST:$DEST_PATH ---" +ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "mkdir -p '$DEST_PATH'" + +LINK_DEST_ARG="" +if ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "test -L '$LATEST' || test -d '$LATEST'" 2>/dev/null; then + LINK_DEST_ARG="--link-dest=$LATEST" +fi + +rsync -aHh --info=stats2 --delete $LINK_DEST_ARG \ + -e "ssh $SSH_OPTS" \ + /opt/stacks/gitea \ + /opt/stacks/verdaccio/storage \ + /opt/stacks/verdaccio/config \ + /opt/stacks/registry \ + /opt/stacks/traefik \ + /opt/stacks/portainer \ + /opt/stacks/ntfy \ + /opt/stacks/backup \ + /opt/stacks/owncloud \ + /etc/ssh \ + /etc/ufw \ + /etc/hosts \ + /etc/docker \ + "$DUMP_DIR" \ + /home/vitya/.ssh \ + "$DEST_USER@$DEST_HOST:$DEST_PATH/" || true + +# Update latest symlink +ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "rm -f '$LATEST' && ln -s '$DEST_PATH' '$LATEST'" + +# === Step 3: retention prune === +echo "--- prune snapshots older than $RETENTION_DAYS days ---" +ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "cd '$DEST_BASE' && ls -1d 20*-*-* 2>/dev/null | sort | head -n -$RETENTION_DAYS | xargs -r rm -rf" + +# === Step 4: local cleanup === +rm -rf "$DUMP_DIR" + +# === Step 5: report === +END=$(date +%s) +DURATION=$((END - START)) +SIZE=$(ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "du -sh '$DEST_PATH' 2>/dev/null | cut -f1" || echo "?") +SNAPSHOT_COUNT=$(ssh $SSH_OPTS "$DEST_USER@$DEST_HOST" "ls -1d $DEST_BASE/20*-*-* 2>/dev/null | wc -l" || echo "?") +DURATION_HUMAN=$(fmt_duration "$DURATION") + +# ntfy push (single line) +NTFY_BODY="$DURATION_HUMAN, size=$SIZE, snapshots=$SNAPSHOT_COUNT, dest=kreknin:$DEST_PATH" +notify "$HOST_LABEL backup OK $TODAY" "$NTFY_BODY" green_circle + +# email (structured) +EMAIL_BODY="$HOST_LABEL daily backup completed successfully. + +Date: $TODAY +Duration: $DURATION_HUMAN +Size: $SIZE +Snapshots: $SNAPSHOT_COUNT +Source: $SOURCE_DISPLAY +Dest: kreknin:$DEST_PATH + +Components: +- /opt/stacks/{gitea,verdaccio,registry,traefik,portainer,ntfy,backup,owncloud} +- /etc/{ssh,ufw,hosts,docker} +- /home/vitya/.ssh +- DB dumps: postgres, mariadb, mongo, redis + +Log: $LOG" +email_send "[$HOST_LABEL] backup OK $TODAY" "$EMAIL_BODY" + +echo "$HOST_LABEL backup OK: $NTFY_BODY" +echo "===== $HOST_LABEL backup done $(date -Is) =====" diff --git a/scripts/windows-host-fallback-backup-daily/deploy.ps1 b/scripts/windows-host-fallback-backup-daily/deploy.ps1 new file mode 100644 index 0000000..f237bd3 --- /dev/null +++ b/scripts/windows-host-fallback-backup-daily/deploy.ps1 @@ -0,0 +1,27 @@ +# Deploy local repo run.ps1 -> C:\ProgramData\backup\run.ps1 with .bak rollback. +# MUST be run elevated (ProgramData ACL = SYSTEM + Administrators). +# Usage: open elevated PowerShell, cd to repo root, then: & .\scripts\windows-host-fallback-backup-daily\deploy.ps1 + +$ErrorActionPreference = 'Stop' +$repoSrc = Join-Path $PSScriptRoot 'run.ps1' +$dst = 'C:\ProgramData\backup\run.ps1' +$bak = "$dst.bak-pre-unify" + +if (-not (Test-Path $repoSrc)) { throw "source missing: $repoSrc" } +if (-not (Test-Path $dst)) { throw "destination missing: $dst (windows-host backup not initialized?)" } + +# Backup current (if no pre-unify backup yet) +if (-not (Test-Path $bak)) { + Copy-Item $dst $bak -Force + Write-Host " backup: $bak" +} else { + Write-Host " backup already present: $bak" +} + +Copy-Item $repoSrc $dst -Force +$srcHash = (Get-FileHash $repoSrc -Algorithm SHA256).Hash +$dstHash = (Get-FileHash $dst -Algorithm SHA256).Hash +if ($srcHash -ne $dstHash) { throw "hash mismatch after copy: src=$srcHash dst=$dstHash" } +Write-Host " deployed: $dst" +Write-Host " sha256: $($dstHash.Substring(0,12))" +Write-Host " mtime: $((Get-Item $dst).LastWriteTime)" diff --git a/scripts/windows-host-fallback-backup-daily/run.ps1 b/scripts/windows-host-fallback-backup-daily/run.ps1 index 7d41b8a..d31f723 100644 --- a/scripts/windows-host-fallback-backup-daily/run.ps1 +++ b/scripts/windows-host-fallback-backup-daily/run.ps1 @@ -31,6 +31,14 @@ $rcloneCfg = "$base\rclone.conf" $remoteBase = "kreknin:NetBackup/windows-host" $remoteToday = "$remoteBase/$today" +$HostLabel = 'windows-host' +$SourceDisplay = "windows-host ($env:COMPUTERNAME / 94.19.247.14)" +$DestDisplay = "kreknin:/volume1/NetBackup/windows-host/$today/" + +function Format-Duration([int]$sec) { + '{0}m{1:D2}s' -f ([int]($sec / 60)), ($sec % 60) +} + if (-not (Test-Path $logDir)) { New-Item -ItemType Directory -Path $logDir -Force | Out-Null } Start-Transcript -Path $logFile -Append -Force | Out-Null @@ -161,23 +169,29 @@ try { Remove-Item $mssqlLocalDir -Recurse -Force -ErrorAction SilentlyContinue - $duration = [int](New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds + $durationSec = [int](New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds + $durationHuman = Format-Duration $durationSec $sitesBytes = (Get-ChildItem 'C:\sites' -Recurse -File -EA SilentlyContinue | Measure-Object Length -Sum).Sum $minioBytes = (Get-ChildItem 'C:\Users\vitya\projects\docker\diskstation\minio\data' -Recurse -File -EA SilentlyContinue | Measure-Object Length -Sum).Sum - $totalGB = '{0:N2}' -f (($sitesBytes + $minioBytes) / 1GB) + $sizeStr = '{0:N2} GB' -f (($sitesBytes + $minioBytes) / 1GB) + $snapshotCount = if ($existing) { @($existing).Count } else { 1 } - $msg = "windows-host daily backup $today OK | $duration sec | $totalGB GB (sites+minio) | 5 DBs" - Notify-Ntfy "windows-host backup OK ($today)" $msg 'default' 'green_circle' + # ntfy push (single line) + $ntfyBody = "$durationHuman, size=$sizeStr, snapshots=$snapshotCount, dest=$DestDisplay" + Notify-Ntfy "$HostLabel backup OK $today" $ntfyBody 'default' 'green_circle' + + # email (structured) $emailBody = @" -windows-host (DESKTOP-NSEF0UK) daily backup completed successfully. +$HostLabel daily backup completed successfully. -Date: $today -Duration: $duration seconds -Size: $totalGB GB (sites+minio data; mssql .bak compressed separately) -Source: windows-host (DESKTOP-NSEF0UK / 94.19.247.14) -Dest: kreknin:/volume1/NetBackup/windows-host/$today/ +Date: $today +Duration: $durationHuman +Size: $sizeStr +Snapshots: $snapshotCount +Source: $SourceDisplay +Dest: $DestDisplay -Components synced: +Components: - mssql (5 DBs: MoreThenCms, StayerCalculator, StayerPrice, stostayer, TireService) - sites (C:\sites\) - minio (data dir) @@ -186,16 +200,35 @@ Components synced: Log: $logFile "@ - Notify-Email "windows-host backup $today -- SUCCESS" $emailBody - Write-Host "`n=== DONE in $duration sec ===" + Notify-Email "[$HostLabel] backup OK $today" $emailBody + Write-Host "`n=== DONE in $durationHuman ===" } catch { $err = $_.Exception.Message Write-Host "`n=== FAILED: $err ===" -ForegroundColor Red Write-Host $_.ScriptStackTrace - $duration = [int](New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds - Notify-Ntfy "windows-host backup FAILED ($today)" "After $duration sec: $err" 'high' 'red_circle' - Notify-Email "windows-host backup $today -- FAILED" "After $duration sec: $err`n`nLog: $logFile" + $durationSec = [int](New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds + $durationHuman = Format-Duration $durationSec + $tailLog = '' + try { $tailLog = (Get-Content $logFile -Tail 40 -EA SilentlyContinue) -join "`n" } catch {} + + # ntfy push (single line) + Notify-Ntfy "$HostLabel backup FAILED $today" "After $durationHuman`: $err. See $logFile" 'high' 'red_circle' + + # email (structured) + $emailBody = @" +$HostLabel daily backup FAILED. + +Date: $today +Duration: $durationHuman +Error: $err +Source: $SourceDisplay +Log: $logFile + +Tail (last 40 lines): +$tailLog +"@ + Notify-Email "[$HostLabel] backup FAILED $today" $emailBody Stop-Transcript | Out-Null exit 1 } finally { diff --git a/scripts/windows-host-fallback-backup-daily/smoke-notify.ps1 b/scripts/windows-host-fallback-backup-daily/smoke-notify.ps1 new file mode 100644 index 0000000..428538d --- /dev/null +++ b/scripts/windows-host-fallback-backup-daily/smoke-notify.ps1 @@ -0,0 +1,64 @@ +# Smoke-test notify channels with NEW unified format. +# Sends one TEST-tagged ntfy push + one TEST-tagged email — no backup run. +# MUST be run elevated (config.env ACL = SYSTEM + Administrators). + +$ErrorActionPreference = 'Stop' +$base = 'C:\ProgramData\backup' +$cfg = @{} +Get-Content "$base\config.env" | Where-Object { $_ -match '^[A-Z_]+=' } | ForEach-Object { + $kv = $_ -split '=', 2 + $cfg[$kv[0]] = $kv[1] +} + +$today = Get-Date -Format 'yyyy-MM-dd' +$HostLabel = 'windows-host' +$SourceDisplay = "windows-host ($env:COMPUTERNAME / 94.19.247.14)" +$DestDisplay = "kreknin:/volume1/NetBackup/windows-host/$today/" +$durationHuman = '81m08s' +$sizeStr = '23.10 GB' +$snapshotCount = 1 +$logFile = "C:\ProgramData\backup\logs\$today.log" + +# ntfy +$pair = "$($cfg.NTFY_USER):$($cfg.NTFY_PASS)" +$auth = 'Basic ' + [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($pair)) +$ntfyBody = "$durationHuman, size=$sizeStr, snapshots=$snapshotCount, dest=$DestDisplay" +try { + Invoke-RestMethod -Uri "$($cfg.NTFY_URL)/vds-backup" -Method POST ` + -Headers @{ Authorization=$auth; Title="[TEST] $HostLabel backup OK $today"; Priority='default'; Tags='green_circle' } ` + -Body $ntfyBody -ContentType 'text/plain' -ErrorAction Stop | Out-Null + Write-Host 'ntfy_ok' +} catch { Write-Host ('ntfy_fail: ' + $_.Exception.Message) } + +# email +$emailBody = @" +$HostLabel daily backup completed successfully. + +Date: $today +Duration: $durationHuman +Size: $sizeStr +Snapshots: $snapshotCount +Source: $SourceDisplay +Dest: $DestDisplay + +Components: +- mssql (5 DBs: MoreThenCms, StayerCalculator, StayerPrice, stostayer, TireService) +- sites (C:\sites\) +- minio (data dir) +- traefik (config + acme.json) +- iis-config (applicationHost.config) + iis-backup-webconfiguration + +Log: $logFile + +[TEST RUN — sent from unify-backup-notifications smoke] +"@ + +$secpass = ConvertTo-SecureString $cfg.SMTP_PASS -AsPlainText -Force +$mailCred = New-Object PSCredential($cfg.SMTP_USER, $secpass) +try { + Send-MailMessage -SmtpServer $cfg.SMTP_HOST -Port ([int]$cfg.SMTP_PORT) -UseSsl ` + -Credential $mailCred -From $cfg.SMTP_FROM -To $cfg.OPS_NOTIFY_EMAIL ` + -Subject "[TEST][$HostLabel] backup OK $today" -Body $emailBody -Encoding UTF8 ` + -WarningAction SilentlyContinue -ErrorAction Stop + Write-Host 'email_ok' +} catch { Write-Host ('email_fail: ' + $_.Exception.Message) }