# vds-backup-rsync-kreknin

## Goal

Ежедневный rsync-бэкап важных директорий с VDS (`89.253.255.94 / vds.kzntsv.site`) → [[kreknin-synology]] (`195.19.90.188 / kreknin.site`) в 05:00 MSK. После каждого прохода — email-нотификация на `vitya.kuznetsov@gmail.com` со статусом (success/fail + transferred bytes + duration).

## Open questions

- [x] ~~**Что бэкапить:**~~ Resolved 2026-05-20. Sources в `/opt/stacks/backup/scripts/run.sh`:
  - `/opt/stacks/{gitea,verdaccio/storage,verdaccio/config,registry,traefik,portainer,ntfy,backup}`
  - `/opt/stacks/owncloud` (added 2026-05-21 для [[owncloud-vds-deploy]] — oCIS data + config + .env + compose; post-import будет ~25GB, hardlink-incremental после первой ночи)
  - `/etc/{ssh,ufw,hosts,docker}`
  - `/home/vitya/.ssh`
  - DB dumps в `$DUMP_DIR` (pg/maria/mongo/redis) — добавляются в rsync source list.
  - **Не бэкапим:** `/opt/stacks/databases/*/data` (raw datadirs — running container locks; dumps вместо).
- [x] ~~**Куда на kreknin:**~~ `/volume1/NetBackup/vds-kzntsv/<YYYY-MM-DD>/` + symlink `latest`. Retention 7 daily snapshots (RETENTION_DAYS env). Weekly/monthly — defer (overkill для текущих ~12G).
- [x] ~~**SSH key:**~~ продолжаем `/home/vitya/.ssh/id_ed25519_kreknin` — скрипт читает абсолютным path (work as root too).
- [x] ~~**Tool:**~~ rsync `--link-dest` (built-in, hardlink-incremental). Encrypted (restic/borg) defer — VDS↔kreknin link trusted (оба ours), encryption key management = ops overhead. Если канал potentially compromised — переключить позже.
- [x] ~~**Email механизм:**~~ `msmtp` + `msmtp-mta` apt-installed на VDS. Config в `/etc/msmtprc` (chmod 600 root:root, system-wide since cron runs as root). SMTP `smtp.yandex.ru:465` + creds `noreply@snolla.com` из `noreply-snolla-smtp.env`. Test send → Yandex 250 OK ✅.
- [ ] **Phone confirm:** user verifies ntfy push на `vds-backup` topic пришёл из реального cron run (или manual run).

## Implementation sketch

`/etc/cron.d/vds-backup`:

```cron
# 05:00 MSK ежедневно
0 5 * * * vitya /opt/stacks/backup/scripts/run.sh
```

`/opt/stacks/backup/scripts/run.sh`:

```bash
#!/bin/bash
set -e
SOURCE_DIRS=(/opt/stacks/gitea/data /opt/stacks/verdaccio/storage /opt/stacks/verdaccio/config /opt/stacks/registry/docker /opt/stacks/registry/auth /opt/stacks/traefik /opt/stacks/portainer/data /etc/ssh /etc/ufw /etc/hosts /etc/docker /etc/letsencrypt)
DEST_BASE=/volume1/NetBackup/vds-kzntsv
TODAY=$(date +%Y-%m-%d)
LATEST=$DEST_BASE/latest

START_TIME=$(date +%s)
LOG=/tmp/backup-$TODAY.log

# DB dumps первым шагом — pg/maria/mongo в /tmp/db-dumps/$TODAY/, потом включаются в rsync
mkdir -p /tmp/db-dumps/$TODAY
docker exec postgres pg_dumpall -U postgres > /tmp/db-dumps/$TODAY/postgres.sql
docker exec mariadb mariadb-dump --all-databases -u root -p"$MARIA_PASS" > /tmp/db-dumps/$TODAY/mariadb.sql
docker exec mongo mongodump --out=/tmp/mongo --uri="mongodb://root:$MONGO_PASS@localhost:27017?tls=true&tlsAllowInvalidCertificates=true"
docker exec redis redis-cli --tls --insecure -a "$REDIS_PASS" --rdb /data/dump.rdb

# rsync с --link-dest для hardlink-incremental
rsync -azh --link-dest=$LATEST "${SOURCE_DIRS[@]}" /tmp/db-dumps/$TODAY \
  -e "ssh -i ~/.ssh/id_ed25519_kreknin" \
  vitya@195.19.90.188:$DEST_BASE/$TODAY/

ssh -i ~/.ssh/id_ed25519_kreknin vitya@195.19.90.188 \
  "rm -f $LATEST && ln -sf $DEST_BASE/$TODAY $LATEST"

END_TIME=$(date +%s)
DURATION=$((END_TIME - START_TIME))
SIZE=$(du -sh $DEST_BASE/$TODAY 2>/dev/null | cut -f1)

# Email via msmtp
echo "Subject: VDS backup $TODAY — SUCCESS

Size: $SIZE, duration: ${DURATION}s
Source: vds.kzntsv.site
Dest: kreknin.site:$DEST_BASE/$TODAY/
" | msmtp vitya.kuznetsov@gmail.com
```

(скетч; добавить retention prune — `find $DEST_BASE -maxdepth 1 -type d -name "20*" | sort | head -n -7 | xargs rm -rf`, error trap, etc.)

## Decisions log

- **2026-05-20** — rsync `--link-dest` chosen over restic/borg. Built-in, no encryption key management, VDS↔kreknin trusted link. Encryption defer if threat model changes.
- **2026-05-20** — Cron runs as **root**, not vitya. Initial vitya run hit ~30 Permission Denied (gitea/portainer container-owned files, /etc/ssh host keys, /etc/ufw rules). Trade-off: больше privilege чем нужно, но альтернатива (sudo wrap rsync) — сложнее. Mitigation: `/etc/msmtprc` chmod 600 root:root, `/opt/stacks/backup/.env` chmod 600.
- **2026-05-20** — DB dump approach: `docker exec <container>` с CLI-passed credentials, не URI params. Mongo требует **legacy `--ssl --sslAllowInvalidCertificates`** (НЕ `--tls --tlsInsecure` despite --help listing `--tlsInsecure` — fails as unknown option pre-`--ssl`).
- **2026-05-20** — Retention начали с 7 daily snapshots (RETENTION_DAYS=7). Weekly/monthly defer — 12G/day @ 7 days = 84G headroom вне hardlinks, на kreknin 5.6T free; нет смысла усложнять.
- **2026-05-20** — Cron не был установлен на VDS из-коробки на Ubuntu 24.04 — apt install cron был нужен. Lesson для будущих vds-* tasks: проверять `which cron` в preflight.
- **2026-05-20** — msmtp `/etc/msmtprc` system-wide (а не `~/.msmtprc`) потому что cron как root → msmtp picks up /etc/ first.
- **2026-05-20** — Smoke run (2026-05-20 23:37 → 00:00) successful: 71,867 files / 11.74G transferred в 21m27s, all 4 DB dumps OK, latest symlink updated, ntfy + email sent.

## Completed steps

- [x] msmtp + msmtp-mta apt-installed; `/etc/msmtprc` written с Yandex SMTP creds (chmod 600 root:root)
- [x] `/opt/stacks/backup/.env` создан с DB passwords + kreknin creds + retention (chmod 600)
- [x] `/opt/stacks/backup/scripts/run.sh` написан (bash + flock single-instance + ERR trap)
- [x] DB dumps protocol: pg_dumpall, mariadb-dump --single-transaction, mongodump --ssl --sslAllowInvalidCertificates, redis-cli SAVE + docker cp
- [x] rsync sources finalized: /opt/stacks/*, /etc/{ssh,ufw,hosts,docker}, /home/vitya/.ssh, $DUMP_DIR
- [x] `/var/log/vds-backup/` log dir (pre-created via sudo + chown vitya)
- [x] `/etc/cron.d/vds-backup` installed (root, `0 5 * * *`)
- [x] apt install cron (Ubuntu 24.04 не имел его out-of-box); cron.service active + enabled
- [x] Smoke run as root: 71,867 files / 11.74G на kreknin:/volume1/NetBackup/vds-kzntsv/2026-05-20, latest symlink set, ntfy publish 200, email Yandex 250 OK
- [x] Permission errors из vitya-run выявлены и устранены переходом на root cron

## Notes

- Триггер: после `[[vds-kzntsv-bootstrap]]` Phase 3 + verdaccio/registry стабильны (done).
- Integration с [[vds-ntfy-push]] — backup publish на topic `vds-backup` (admin auth).
- Retention vs disk: kreknin 5.7T free, daily 7 × ~12G full ≈ 85G headroom (hardlinks делают incremental ~1G/day после day 1).
- **acme.json** проходит через rsync (внутри /opt/stacks/traefik) — содержит LE private keys, но VDS↔kreknin link trusted, kreknin ACL restricted to vitya. Acceptable trade-off vs шифрование через restic.
- **Atomic revert:** `ssh vitya@89.253.255.94 'sudo systemctl stop cron; sudo rm /etc/cron.d/vds-backup; sudo apt-get remove -y cron msmtp msmtp-mta; sudo rm -rf /etc/msmtprc /opt/stacks/backup /var/log/vds-backup'`