Runbook: Arcane login broken after a container recreate¶

Symptom: docker.fmm.house shows the login page but every attempt returns 401 (POST /api/auth/login → 401), even with the correct password. Happened once in the 2026-07-14 port cleanup after arcane was recreated.

Cause¶

Arcane derives its auth from ENCRYPTION_KEY / JWT_SECRET in the compose env and does not persist them (nothing in the data volume, no kv/settings key). The account's password is validated against whatever secret the running container had. If those secrets change relative to when the account was created — e.g. they were absent (ephemeral, regenerated each start) and later pinned — a recreate invalidates every stored password. The user/role/data are all intact; only verification fails.

Prevention (done): ENCRYPTION_KEY and JWT_SECRET live in stacks/arcane/.env (gitignored, referenced by the compose), so recreates reuse the same secret and logins keep working. Never change or regenerate them, or you'll re-trigger this.

Recovery — reset the password through Arcane's own API¶

Because the reset goes through Arcane, it re-hashes against the current secret (a raw DB write would need to replicate the hashing/pepper, so don't). Uses a temporary admin API key.

cd stacks/arcane
# 1. generate a static admin key and inject it temporarily
KEY=$(docker exec arcane /app/arcane generate api-key | tail -1)   # arc_...
#    add  ADMIN_STATIC_API_KEY: <KEY>  to the arcane env, then:
docker compose up -d arcane            # recreate with the key

# 2. find the user id (the key authenticates as the admin user)
curl -s -H "X-API-Key: $KEY" https://docker.fmm.house/api/auth/me   # -> data.id

# 3. reset the password (admin endpoint; re-hashes with the live secret)
curl -s -X PUT -H "X-API-Key: $KEY" -H 'Content-Type: application/json' \
  -d '{"username":"albert","password":"<new-temp-password>"}' \
  https://docker.fmm.house/api/users/<id>

# 4. remove ADMIN_STATIC_API_KEY from the env and recreate — the reset password persists
docker compose up -d arcane

Then log in and change the temp password in the UI. Notes: - API key header is X-API-Key; the admin user is albert (renamed from the default arcane). UserUpdateUser has no roles field, so a password/username change won't strip the admin role. - Same trick (X-API-Key or a login-token Authorization: Bearer) works for any admin API op, e.g. renaming the user via PUT /api/users/{id}. - Arcane reaches Docker through a read-only socket-proxy (arcane-docker-proxy), not the raw socket.

Update 2026-07-21 — the above recovery FAILS on v2.4.0 because the admin was renamed¶

The ADMIN_STATIC_API_KEY reconciliation only binds the key to a default admin user literally named arcane (getDefaultAdminUser, hardcoded — there is no env override for the admin username in this build; confirmed by strings on /app/arcane). Because we renamed the account to albert, startup logs WRN Default admin user not found, skipping default admin API key reconciliation username=arcane, the key is never wired up, and every request 401s — so steps 20-42 above no longer work as written.

Working recovery (rename the account back, briefly, at the DB layer): password is argon2id peppered with the secret, so it can't be set from a raw DB write — but the username can. The store is SQLite at arcane.db (volume arcane_arcane-data); roles live in user_role_assignments keyed by user-id, so a username change doesn't affect admin.

cd stacks/arcane
KEY=$(docker exec arcane /app/arcane generate api-key | grep -oE 'arc_[A-Za-z0-9._-]+' | head -1)
# rename albert -> arcane while arcane is stopped (root helper container; no host sudo needed)
docker stop arcane
docker run --rm -e SQL="UPDATE users SET username='arcane' WHERE username='albert';" \
  -v arcane_arcane-data:/data alpine sh -c 'apk add -q sqlite; sqlite3 /data/arcane.db "$SQL"'
# start with the key (compose.override.yml injecting ADMIN_STATIC_API_KEY: ${ADMIN_STATIC_API_KEY})
ADMIN_STATIC_API_KEY="$KEY" docker compose -f compose.yml -f compose.override.yml up -d arcane
ID=$(curl -s -H "X-API-Key: $KEY" https://docker.fmm.house/api/auth/me | python3 -c 'import sys,json;print(json.load(sys.stdin)["data"]["id"])')
# one PUT resets the password AND renames back to albert:
curl -s -X PUT -H "X-API-Key: $KEY" -H 'Content-Type: application/json' \
  -d '{"username":"albert","password":"<new-temp>"}' https://docker.fmm.house/api/users/$ID
rm compose.override.yml && docker compose up -d arcane   # remove key, recreate clean

Clean up afterward: reconciliation persists a managed row in api_keys named Static Admin API Key (the 2026-07-14 run left one lingering for a week). Its plaintext is gone, but delete the stale record: DELETE FROM api_keys WHERE name='Static Admin API Key'; (stop arcane first, or delete via the API with apikeys:delete).

Better long-term fix: rename the account back to the default arcane for good, so the documented static-key recovery keeps working — or stand up OIDC so you're never locked out of local auth.