Skip to content

Runbook: monitoring, update-notifications & container healthchecks

The observability layer, stood up in the 2026-07-14 session. Three complementary tools plus a repo-wide healthcheck pass. Backups are deliberately NOT here — they need a separate, off-box target (planned: restic + Backrest).

Backup scope — MUST include the gitignored stacks/*/.env files. They hold irreplaceable secrets that exist nowhere else (Arcane ENCRYPTION_KEY/JWT_SECRET, Homarr key, Matrix registration token, MQTT passwords, Beszel/Diun tokens, Postmoogle bot password + POSTMOOGLE_DATA_SECRET). Losing them = lockouts and undecryptable data (see docs/runbooks/arcane-auth-recovery.md). Also back up Postmoogle's data/postmoogle.db (mailbox↔room mappings). Until a real backup exists, keep an encrypted off-box copy of the .env files.

Concern Tool URL Stack
"Is it up?" (availability) Uptime Kuma up.fmm.house stacks/uptime-kuma
"How's it doing?" (metrics / btop view) Beszel bz.fmm.house stacks/beszel
"Is there a new image?" (moving tags, notify-only) Diun — (no UI) stacks/diun
"Is there a new version?" (pinned tags, PRs) Renovate Forgejo dashboard stacks/renovate
Per-container health badges Docker HEALTHCHECKs — in each compose.yml

Uptime Kuma (up.fmm.house)

  • v2.x, SQLite (data/kuma.db). Admin: albert.
  • No PostgreSQL support — Uptime Kuma only does SQLite or MariaDB/MySQL for its own storage (verified in-source: supportedDBTypes = ["mariadb","sqlite"]). SQLite is correct for a single instance; don't try to point it at the Postgres boxes.
  • No REST API — monitors are managed over Socket.IO (the web UI) only. The service monitors were inserted directly into the monitor table (only id is required; all other columns default), then UK was restarted to load them. To insert while UK is running, use its own root shell: docker exec uptime-kuma sqlite3 /app/data/kuma.db "INSERT …" (the data/ dir is root-owned so a host-side write fails), then docker restart uptime-kuma to load it. Back up kuma.db first (docker exec uptime-kuma cp …).
  • Monitors 1–18 are HTTP(s) against each *.fmm.house service, accepted_statuscodes = ["200-299","300-399","401","403"] so auth-gated/redirecting services don't false-down.
  • Monitor #19 AdGuard DNS (fmm.house :53) (type dns) — a real DNS check: resolves fmm.house A against 192.168.86.49:53 every 60 s. This catches AdGuard's DNS being dead even when the HTTP monitor #2 (its web UI) is green — the exact "healthy but dead" gap from the 2026-07-16 reboot (see docs/runbooks/dns.md §6.4).
  • Notifications → Matrix "Homelab Updates" (the same room Diun posts to). One matrix notifier (notification id 1, is_default=1 so new monitors inherit it) is attached to all monitors. It posts via http://continuwuity:8008 (UK is on matrix_default) using @diun's access token — so up/down alerts land alongside the image-update alerts. Wired by DB insert (no UI): login as @diun → build config {type:matrix, homeserverUrl, internalRoomId, accessToken} → INSERT INTO notification … + link every monitor in monitor_notification → recreate UK. ⚠️ Token caveat: the stored access token is a @diun login session; if it's ever revoked (logout-all / expiry), UK alerts go silent — re-login as @diun and update notification.config. A dedicated @uptimekuma bot would be cleaner long-term.

Beszel (bz.fmm.house) — hub + agent

PocketBase-based; admin identity is an email (albert@fmm.house). Outbound mail (password reset, alerts) sends via Postmoogle as beszel@fmm.house → Matrix; those SMTP settings + the appURL live in the PocketBase DB (data/hub/data.db, _params.settings), not compose. The host shows up as system ascertain-verse.

⚠ Patched hub image — stacks/beszel/patch/. The hub runs a locally-built beszel-hub-patched:<ver>, not stock henrygd/beszel. Why: the stock login form has a hidden website honeypot validated with v.literal(""); Android password managers (Bitwarden/Chrome) autofill it, so validation fails and the login silently won't submit on mobile — a known, still-open upstream bug (#1011), unfixed as of 0.18.7. Our patch flips that schema to v.any() so the field always validates (bots are still caught by PocketBase's auth rate-limiting, and this instance is Tailscale/LAN-only + single-user, so the honeypot bought nothing — verified in the built bundle: website:ut("") → website:ut()). The Dockerfile fetches the pinned source, applies a one-line sed guarded by a grep that fails the build if the anchor ever moves, builds the web UI + Go hub, and mirrors upstream's scratch image. Update flow: bump the version in patch/Dockerfile and the image: tag + build.args in compose.yml, then docker compose build beszel && docker compose up -d beszel. Retire the patch and return to the stock image once upstream ships a fix (Diun flags the new release). Patch-free workarounds: Firefox on Android, or /_/ (PocketBase admin login — no honeypot).

Architecture: beszel (hub, UI on 8090, Traefik-fronted) + beszel-agent (this host). The agent runs in URL / universal-token mode: it connects out to HUB_URL=http://beszel:8090 with a TOKEN (a universal token) + KEY (the hub's SSH public key) and auto-registers. Both live in stacks/beszel/.env (gitignored).

To add another host: run a beszel-agent there with the same HUB_URL, KEY, and a universal TOKEN — it self-registers under its hostname. The hub public key is at GET /api/beszel/getkey (authenticated); universal tokens live in the universal_tokens collection.

Beszel is also the btop-style container view — the agent (docker.sock mounted) reports per-container CPU/mem/net for all running containers. It does not track container up/down or Docker health (that's the healthcheck badges + Uptime Kuma).

Diun (stacks/diun) — image-update notifier (no UI)

Watches every running container (watchByDefault: true) by image digest, every 6h, and fans out to two channels (config in stacks/diun/diun.yml, secrets in .env):

  1. Matrix — posts to the "Homelab Updates" room via bot @diun:matrix.fmm.house (reaches the homeserver internally at continuwuity:8008 over the matrix_default net).
  2. MQTT — publishes JSON to topic docker/diun on mosquitto (over ha_default), which the HA automation diun_image_update_notify turns into a phone push (notify.mobile_app_albert_s_phone).

mosquitto wiring: a dedicated diun user in stacks/ha/mosquitto/passwd + an ACL entry (user diun / topic readwrite docker/diun) in stacks/ha/mosquitto/acl. ⚠️ Those are single-file bind mounts — a Write/Edit that replaces the inode won't reach the container; append in place or docker restart mosquitto (see gotcha in CONTEXT.md).

Digest vs. new-version: by default Diun notifies when a watched tag's digest changes (floating tags, rebuilds). Since most images here are hand-pinned to a version, that won't catch "v1.71.0 is out." For true new-version alerts on a service, add label diun.watch_repo=true (optionally diun.include_tags=^\d+\.\d+\.\d+$) to its compose service.

docker exec diun diun notif test sends a test through all notifiers.

Renovate (stacks/renovate) — new-version PRs for pinned tags

Diun's gap (above) is exactly Renovate's job: it parses the hand-pinned image tags in our compose files and surfaces available upgrades as a "🐳 Image updates" Dependency Dashboard issue on Forgejo, opening a PR only for updates you tick. Config: renovate.json (repo root) + stacks/renovate/compose.yml (renovate/renovate:41, looped every 6h; platform: gitea since Forgejo is Gitea-API-compatible; endpoint https://git.fmm.house/api/v1). Token + repo list in stacks/renovate/.env (gitignored). No docker socket needed — it reaches Forgejo + registries over the network.

⏸ Not deployed yet — blocked on Forgejo setup (needs you): 1. Host the repo on Forgejo. The homelab repo is currently local-only (no git remote). Create homelab on git.fmm.house (enable Issues — the dashboard is an issue), then git remote add origin https://git.fmm.house/<user>/homelab.git && git push -u origin main. (Repo is push-safe: only *.example.env are tracked, all real secrets are gitignored.) 2. Generate a token: Forgejo → Settings → Applications → scopes write:repository, write:issue, read:user. 3. Put both in stacks/renovate/.env: RENOVATE_TOKEN=… and RENOVATE_REPOSITORIES=["<user>/homelab"], then docker compose -f stacks/renovate/compose.yml up -d. First run creates the dashboard issue; docker logs -f renovate to watch.

Container healthchecks

The 2026-07-14 pass added a Docker HEALTHCHECK to every container that could support one (so the status badge reads healthy instead of none). Each probe was verified to exit 0 inside its container before being applied, and each service was recreated individually. End state: 0 unhealthy.

Patterns used (see each compose.yml): - App CLI: beszel /beszel health, beszel-agent /agent health, diun diun healthcheck. - HTTP: curl -fsS or busybox wget -q against a real 2xx path (forgejo /api/healthz, ocis /healthz, peertube /api/v1/config, homarr /, adguard :8083/, etc.). - TCP-only: nc -z localhost <port> (mosquitto 1883, heisenbridge 9898); python socket (wyoming 10200/10300) or bash /dev/tcp (moonlight-web) where no nc exists. - redis-cli ping, node -e require('http').get(...) (trmnl byos), etc.

Skipped (3) — no self-contained probe possible; not faked: - continuwuity — distroless image: no shell, no curl/wget/nc, and conduwuit has no health subcommand. Would need a probe binary baked into the image. - controller-watch, trmnl-nowplaying — pure worker sidecars with no listening port; a real liveness check would require app changes.

To add a check to a distroless service later, bake a tiny static probe into a wrapper image or use a sidecar; don't invent a probe that can't actually run in-container.

Gotcha: a wrong healthcheck marks a working container unhealthy and can block anything that depends_on: condition: service_healthy. Always verify the probe exits 0 first.