admin/.tasks/ruvds-backup-daily-kreknin.md

# ruvds-backup-daily-kreknin

## Goal

Ежедневный backup с RUVDS (`80.64.31.36` / Win Server 2025 Core) → [[../entities/kreknin-synology]] (`195.19.90.188 / kreknin.site`) в **04:30 MSK** (за 30 мин до `[vds-backup-rsync-kreknin]` 05:00, чтобы не пересекать uplink). После каждого прохода — ntfy push на topic `vds-backup` (тот же что VDS backup пушит — общий agg-канал).

Параллель `[vds-backup-rsync-kreknin]` 🟢 (VDS Ubuntu → kreknin via rsync `--link-dest`), но source = Windows Server Core, поэтому:
- Cron нет → Windows Task Scheduler
- rsync нативного нет → **rclone** sync over SFTP (см. Decisions ниже)
- root-cron-as-uid недоступен → SYSTEM scheduled task

## Scope (what to backup)

Минимум для disaster-recovery RUVDS state:

| Path on RUVDS | Reason | Approx size |
|---|---|---|
| `C:\sites\snolla\` | CMS site content + Web.config + media | 8.66 GB (grows slowly) |
| `C:\Windows\System32\inetsrv\config\applicationHost.config` | IIS site/binding/apppool config | <1 MB |
| Backup-WebConfiguration export | IIS native config snapshot (`Backup-WebConfiguration -Name ...`) → `%SystemRoot%\System32\inetsrv\backup\` | <5 MB |
| `C:\ProgramData\ssh\` (sshd_config + administrators_authorized_keys) | SSH access state | <50 KB |
| LE certs export (`Get-ChildItem Cert:\LocalMachine\My` → Export-PfxCertificate с временным пасс'ом + git-tracked) | для restore без re-extract из traefik acme.json | <1 MB |

**Не бэкапить:**
- `C:\Windows\` system files — RUVDS rebuild = fresh install (~5 мин через RUVDS panel).
- `C:\Program Files\` — installed software re-installable from scripts (bootstrap.ps1).
- IIS logs (`C:\inetpub\logs\LogFiles\`) — диагностические, не recovery-critical. Optional.

## Open questions

- [ ] **Tool:** rclone sync over SFTP (см. Decisions ниже — recommended). Alternative — restic (encrypted dedupe) — defer как для VDS; RUVDS↔kreknin link через public internet, но обе стороны ours, single-hop. Add encryption если threat model changes.
- [ ] **Retention:** 7 daily snapshots — следуем VDS-pattern. Per-day dirs (`/volume1/NetBackup/ruvds-iis/2026-05-25/`) — без hardlink dedup, 8.66 GB × 7 = ~60 GB на kreknin (5.7T free, easy).
- [ ] **Auth — RUVDS→kreknin SSH key:** генерировать новый ed25519 на RUVDS (`ssh-keygen` ships with OpenSSH client/server caps), pubkey добавить в `/volume1/homes/vitya/.ssh/authorized_keys` на kreknin через user. Private key хранится в `C:\ProgramData\backup\ssh_key` (ACL: SYSTEM + Administrators only).
- [ ] **Notification:** ntfy `vds-backup` topic — reuse (общий backup-канал, удобно для phone alerts). Title включает source-host чтобы отличать VDS-backup от RUVDS-backup.
- [ ] **Encryption-at-rest на kreknin:** none (как для VDS). RUVDS data = sites content (public web pages anyway) + config files. Не sensitive enough для encryption overhead в transitional setup.

## Implementation sketch

### 1. rclone install + config

```powershell
# Install rclone (one-time)
$ver = (Invoke-RestMethod 'https://downloads.rclone.org/version.txt').Trim('v').Trim()
$url = "https://downloads.rclone.org/v$ver/rclone-v$ver-windows-amd64.zip"
$tmp = "$env:TEMP\rclone.zip"
Invoke-WebRequest $url -OutFile $tmp -UseBasicParsing
Expand-Archive $tmp -DestinationPath "$env:TEMP\rclone" -Force
New-Item -ItemType Directory 'C:\Program Files\rclone' -Force | Out-Null
Copy-Item "$env:TEMP\rclone\rclone-*-windows-amd64\rclone.exe" 'C:\Program Files\rclone\' -Force
[Environment]::SetEnvironmentVariable('Path', "$env:Path;C:\Program Files\rclone", 'Machine')

# Generate SSH key для kreknin auth (one-time)
ssh-keygen -t ed25519 -f C:\ProgramData\backup\kreknin-key -N '""' -C 'ruvds-backup'
icacls C:\ProgramData\backup\kreknin-key /inheritance:r /grant 'SYSTEM:(F)' /grant 'Administrators:(F)'
# Затем user добавляет .pub в kreknin /volume1/homes/vitya/.ssh/authorized_keys

# rclone config — SFTP remote
@'
[kreknin]
type = sftp
host = 195.19.90.188
user = vitya
key_file = C:\ProgramData\backup\kreknin-key
disable_hashcheck = true
'@ | Set-Content C:\ProgramData\backup\rclone.conf -Encoding ASCII
```

### 2. Backup script `C:\ProgramData\backup\run.ps1`

```powershell
$ErrorActionPreference = 'Stop'
$today = Get-Date -Format 'yyyy-MM-dd'
$start = Get-Date
$logFile = "C:\ProgramData\backup\logs\$today.log"
New-Item -ItemType Directory (Split-Path $logFile) -Force | Out-Null
Start-Transcript -Path $logFile -Append -Force | Out-Null

try {
    # 1. IIS config snapshot
    Backup-WebConfiguration -Name "daily-$today" -Force | Out-Null
    $iisBackupDir = "$env:SystemRoot\System32\inetsrv\backup\daily-$today"

    # 2. Export cert store (LocalMachine\My) to PFX with timestamp passphrase
    $pfxPass = ConvertTo-SecureString 'pfximport' -AsPlainText -Force  # TODO move to pass-equivalent
    $certDir = "C:\ProgramData\backup\certs-$today"
    New-Item -ItemType Directory $certDir -Force | Out-Null
    Get-ChildItem Cert:\LocalMachine\My | Where-Object { $_.HasPrivateKey } | ForEach-Object {
        $name = ($_.Subject -replace '[^A-Za-z0-9.-]','_').Substring(0, [math]::Min(40, $_.Subject.Length))
        try {
            Export-PfxCertificate -Cert $_ -FilePath "$certDir\$name.pfx" -Password $pfxPass -ErrorAction SilentlyContinue | Out-Null
        } catch {}
    }

    # 3. rclone sync — push to dated dir on kreknin
    & 'C:\Program Files\rclone\rclone.exe' sync `
        --config C:\ProgramData\backup\rclone.conf `
        --log-file $logFile --log-level INFO `
        --transfers 4 --checkers 8 `
        C:\sites\snolla `
        "kreknin:/volume1/NetBackup/ruvds-iis/$today/sites/snolla/"
    if ($LASTEXITCODE -ne 0) { throw "rclone snolla failed" }

    & 'C:\Program Files\rclone\rclone.exe' copy `
        --config C:\ProgramData\backup\rclone.conf `
        C:\Windows\System32\inetsrv\config\applicationHost.config `
        "kreknin:/volume1/NetBackup/ruvds-iis/$today/iis-config/"

    & 'C:\Program Files\rclone\rclone.exe' sync `
        --config C:\ProgramData\backup\rclone.conf `
        $iisBackupDir `
        "kreknin:/volume1/NetBackup/ruvds-iis/$today/iis-backup-$today/"

    & 'C:\Program Files\rclone\rclone.exe' sync `
        --config C:\ProgramData\backup\rclone.conf `
        $certDir `
        "kreknin:/volume1/NetBackup/ruvds-iis/$today/certs/"

    & 'C:\Program Files\rclone\rclone.exe' sync `
        --config C:\ProgramData\backup\rclone.conf `
        C:\ProgramData\ssh `
        "kreknin:/volume1/NetBackup/ruvds-iis/$today/ssh-config/"

    # 4. Retention prune — keep last 7 daily snapshots
    & 'C:\Program Files\rclone\rclone.exe' lsd `
        --config C:\ProgramData\backup\rclone.conf `
        'kreknin:/volume1/NetBackup/ruvds-iis/' |
        Where-Object { $_ -match '\d{4}-\d{2}-\d{2}' } |
        ForEach-Object { ($_ -split '\s+')[-1] } |
        Sort-Object | Select-Object -SkipLast 7 |
        ForEach-Object {
            & 'C:\Program Files\rclone\rclone.exe' purge `
                --config C:\ProgramData\backup\rclone.conf `
                "kreknin:/volume1/NetBackup/ruvds-iis/$_"
        }

    # 5. Cleanup local temp
    Remove-Item $certDir -Recurse -Force
    Remove-Item $iisBackupDir -Recurse -Force -ErrorAction SilentlyContinue

    # 6. Notification — ntfy success
    $duration = (New-TimeSpan -Start $start -End (Get-Date)).TotalSeconds
    $msg = "RUVDS backup $today OK | $([math]::Round($duration,0))s"
    $ntfyToken = (Get-Content C:\ProgramData\backup\ntfy-token.txt)
    Invoke-RestMethod -Uri 'https://ntfy.vds.kzntsv.site/vds-backup' `
        -Method POST `
        -Headers @{ Authorization = "Bearer $ntfyToken"; Title = "RUVDS backup success"; Priority = 'default'; Tags = 'green_circle' } `
        -Body $msg

} catch {
    # Notification — ntfy failure
    $msg = "RUVDS backup $today FAILED: $($_.Exception.Message)"
    $ntfyToken = (Get-Content C:\ProgramData\backup\ntfy-token.txt -ErrorAction SilentlyContinue)
    if ($ntfyToken) {
        Invoke-RestMethod -Uri 'https://ntfy.vds.kzntsv.site/vds-backup' `
            -Method POST `
            -Headers @{ Authorization = "Bearer $ntfyToken"; Title = "RUVDS backup FAILED"; Priority = 'high'; Tags = 'red_circle' } `
            -Body $msg
    }
    throw
} finally {
    Stop-Transcript | Out-Null
}
```

### 3. Schedule via Windows Task Scheduler

```powershell
$action = New-ScheduledTaskAction -Execute 'powershell.exe' `
    -Argument '-NoProfile -ExecutionPolicy Bypass -File C:\ProgramData\backup\run.ps1'
$trigger = New-ScheduledTaskTrigger -Daily -At '04:30'
$principal = New-ScheduledTaskPrincipal -UserId 'SYSTEM' -LogonType ServiceAccount -RunLevel Highest
$settings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries -StartWhenAvailable -MultipleInstances IgnoreNew
Register-ScheduledTask -TaskName 'RUVDS-Backup-Daily' `
    -Action $action -Trigger $trigger -Principal $principal -Settings $settings -Force
```

## Decisions log

- **2026-05-24 (task creation):** Tool = **rclone**, not robocopy-over-SMB and not restic. Reasoning:
  - rsync на Win Server Core отсутствует (no cygwin/WSL on Core). Установка cygwin/MSYS — лишний пакет, security surface.
  - robocopy + SMB-mount kreknin требует Synology SMB включён + Windows credentials cached + UNC reliability over WAN — fragile.
  - rclone — single .exe (~50 MB), SFTP backend native, reliable, well-supported on Server Core, log-friendly.
  - restic — encrypted+dedupe — overkill пока link trusted; defer like VDS.

- **2026-05-24:** **Run as SYSTEM**, не пользовательский account. SYSTEM имеет full доступ к `C:\sites\` + `C:\Windows\System32\inetsrv\` + cert store, не требует stored-password в Task Scheduler. Pattern matches `[vds-backup-rsync-kreknin]` root-cron decision.

- **2026-05-24:** **PFX export `pfximport` password** — temporary, will move to pass-equivalent on RUVDS (или derived из машинной creds). Open question: how to handle pass-on-Windows-Core analog. Defer пока migration не cutover'нется полностью.

## Open questions (additional)

- [ ] **Когда стартовать:** сейчас (parallel с soak window `[iis-migration-to-ruvds]`) или после full cutover? Recommendation: **сейчас** — даже до full cutover, backup snolla content на RUVDS даёт rollback-evidence если RUVDS state corrupt'нется. Lateral risk minimal — backup идёт через ssh-key auth, кroil-кroil traffic.
- [ ] **kreknin SSH ACL on `/volume1/homes/vitya/`** — текущий ACL разрешает `vitya` user. RUVDS пушит как `vitya@kreknin` через SFTP — нужна добавка pubkey в authorized_keys. Без user action не сработает.
- [ ] **Backup `C:\sites\snolla` size growth** — CMS file uploads через админку могут вырасти со временем. Monitor disk usage на kreknin через quarterly check.

## Completed steps

- [x] **2026-05-24 ~10:18:** rclone v1.74.2 installed на RUVDS (`C:\Program Files\rclone\rclone.exe`).
- [x] **2026-05-24 ~10:18:** ed25519 SSH key generated `C:\ProgramData\backup\kreknin-key`, pubkey deployed в kreknin `/var/services/homes/vitya/.ssh/authorized_keys` через VDS pivot (source machine не имеет direct SSH к kreknin).
- [x] **2026-05-24 ~10:19:** `C:\ProgramData\backup\rclone.conf` (SFTP remote `kreknin`, disable_hashcheck), `config.env` (ntfy creds), `run.ps1` deployed; ACL = SYSTEM + Administrators.
- [x] **2026-05-24 10:19:47:** ScheduledTask `RUVDS-Backup-Daily` registered (daily 04:30 MSK, SYSTEM, 2h timeout).
- [x] **2026-05-24 10:21:13 – 11:42:57:** initial full sync 8.66 GB snolla site + 4 small components → kreknin (Завершилось ~82 мин из-за home-uplink throttling — production cron не affected since RUVDS uplink direct).
- [x] **2026-05-24 11:46:51:** run #1 incremental smoke ✓ 20 sec — все 5 components ok, retention prune ok (1 snapshot kept), ntfy success sent.
- [x] **2026-05-24:** Scripts checked into repo `scripts/ruvds-backup-daily-kreknin/` (setup.ps1 + run.ps1 + README).

## Closed

**2026-05-24 11:46:51** — пайплайн live, run #1 verified end-to-end. Закрывает Open question "Backup strategy для RUVDS IIS" в `[iis-migration-to-ruvds]`.

**Bugs fixed during smoke (зафиксированы в Decisions log):**
- `Backup-WebConfiguration -Force` параметра нет на этом IIS — используем `Get-WebConfigurationBackup + Remove-WebConfigurationBackup` если exists + plain `Backup-WebConfiguration`.
- rclone `--log-file` указывал на тот же путь что и `Start-Transcript` → file lock. Убрали `--log-file` (transcript captures всё необходимое).
- rclone NOTICE на stderr + `$ErrorActionPreference = 'Stop'` = PS terminating error даже c `2>$null`. Решение: `Invoke-Rclone` wrapper-функция temporarily switches к `ErrorActionPreference = 'Continue'`.
- ssh-keyscan'енный known_hosts не proходит rclone go-sftp library (key mismatch). Решение: убрать `known_hosts_file` из rclone.conf, полагаться на key-auth.

## Open follow-ups (не блокер)

- [ ] PFX export pass = `ruvds-backup-pfx` plaintext в `run.ps1` — temporary. TODO: pass-equivalent на Windows (gpg4win + bash-pass), или derived from machine-creds (DPAPI scoped to SYSTEM).
- [ ] Retention prune ещё не сработал (day 1 — 1 snapshot < 7). Verify в day 8 что purge action correct.
- [ ] Phone-side ntfy verify — user проверит что push с title `RUVDS backup OK` приходит на ntfy app, тот же channel что VDS backup.

## Notes

- **Триггер:** task создан 2026-05-24, можно стартовать paralleltno с soak'ом RUVDS. Не блокировать full DNS swap (если backup pipeline ломается — rollback на source IIS:8089 всегда возможен).
- **Integration:** ntfy topic `vds-backup` — общий с VDS backup. Title differentiator чтобы phone-side фильтровать.
- **Атомарный revert (откат всей задачи):**
  ```powershell
  # На RUVDS:
  Unregister-ScheduledTask -TaskName 'RUVDS-Backup-Daily' -Confirm:$false
  Remove-Item C:\ProgramData\backup -Recurse -Force
  # На kreknin: (через SSH)
  ssh vitya@195.19.90.188 'rm -rf /volume1/NetBackup/ruvds-iis'
  ```
- **Relation to [iis-migration-to-ruvds]:** этот backup закрывает "Backup strategy для RUVDS IIS" open-question из migration task. Когда migration переходит в 🟢, эту задачу нужно явно отметить как dependency-closer.
- **Future evolution:** когда snolla-on-node будет ready и IIS станет obsolete (long-term goal per migration task notes) — этот backup можно retire вместе с RUVDS.

<!-- created-by: vitya / 2026-05-24 / trigger: post-iis-migration-partial-cutover, while-soak-window-open -->