◐ Off-By-One · answer catalog

duckbrain-auth-registry-stale-daemon-cache

2 answer(s)shelllinuxshelllinux

Problem class: duckbrain-auth-registry-stale-daemon-cache

📦 Source in repository (JSON)

Answer 1

DuckBrain HTTP 401 after auth.json rewrite — stale daemon registry cache

Problem class: duckbrain-auth-registry-stale-daemon-cache Symptom: HTTP 401 {"error":"Unauthorized: Invalid API key"} on /api/keys/flat even though the consumer's named plaintext sidecar hashed (trimmed SHA-256) to a keyHash present in the on-disk auth.json.

1. Root cause

duckbrain-http.service loads the authorization registry (auth.json) once at process start and keeps it in memory. The registry is not re-read, watched, or invalidated when the file changes. In the failing incident:

Event Time
duckbrain-http.service starts, loads registry 07:30
auth.json is rewritten with new/rotated grants 09:02

The daemon authenticates against the 07:30 snapshot. A token whose trimmed SHA-256 matches a keyHash written at 09:02 is still rejected, because that hash does not exist in the daemon's in-memory map. This is a process-level cache invalidation bug, not a token or hash-computation bug. Restarting the service forces a reload of the 09:02 registry; the same token then returns 200.

Two properties make it look like a bad key:

  1. The on-disk check passes (keyHash matches), so the token "should" work.
  2. 401 Invalid API key is indistinguishable from a genuinely wrong key, so the consumer keeps rotating/re-checking the token instead of the daemon.

The trigger is any out-of-band auth.json rewrite: key rotation, grant addition, grant revocation, or restore from backup.

2. Diagnosis (read-only, safe — never prints the token)

Confirm the two independent facts: the daemon is older than the registry, and the sidecar hash is present in the registry.

SERVICE=duckbrain-http.service
AUTH_JSON="${DB_AUTH_JSON:-$HOME/.local/share/duckbrain/auth.json}"   # adjust to checkout
SECRETS="${DB_SECRETS:-$(dirname "$AUTH_JSON")/secrets}"

# 2a. Daemon start time vs registry mtime. If start < mtime -> stale cache.
systemctl --user show "$SERVICE" -p ActiveEnterTimestamp --value
stat -c '%y  %n' "$AUTH_JSON"

# 2b. Does the sidecar's trimmed SHA-256 appear as a keyHash? (no token echoed)
for f in "$SECRETS"/*; do
  [ -f "$f" ] || continue
  s=$(cat -- "$f"); s=${s#"${s%%[![:space:]]*}"}; s=${s%"${s##*[![:space:]]}"}
  h=$(printf '%s' "$s" | sha256sum | awk '{print $1}')
  name=$(jq -r --arg h "$h" '(.keys // .)
      | (if type=="array" then .[] else to_entries[].value end)
      | select(.keyHash == $h) | .name' "$AUTH_JSON")
  printf '%-24s sha256=%s grant=%s\n' "$(basename "$f")" "${h:0:12}…" "${name:-<none>}"
  unset s
done

# 2c. Live probe with a sidecar known to match the registry.
token=$(cat "$SECRETS/<matching-sidecar>.token")
curl -s -o /dev/null -w '%{http_code}\n' \
  -H "Authorization: Bearer $token" http://localhost:3000/api/keys/flat
unset token

Stale-cache signature: 2b prints a non-empty grant, but 2c prints 401. That combination — hash present on disk, rejected by the live daemon — is the proof.

The full ready-to-run checker is duckbrain-auth-doctor.sh in §6. Run it without --fix first.

3. Immediate fix (restore service)

systemctl --user restart duckbrain-http.service
systemctl --user is-active duckbrain-http.service    # -> active

Then re-probe and re-run the project smoke canary:

curl -s -o /dev/null -w '%{http_code}\n' \
  -H "Authorization: Bearer $(cat "$SECRETS/<matching-sidecar>.token")" \
  http://localhost:3000/api/keys/flat

./scripts/smoke-canary.sh        # substitute the project's actual canary command

Expected recovery: canary goes 13 PASS / 2 WARN / 1 FAIL → 14 PASS / 2 WARN / 0 FAIL.

A restart is only a stop-gap: the next auth.json write re-introduces the bug.

4. Durable fix — make the daemon reload when auth.json changes

Because the environment is a systemd --user service and the problem is a missing invalidation edge, wire that edge into systemd with a path unit. This needs no code change to DuckBrain and survives reboots.

4a. Add a reload verb to the daemon unit

Edit ~/.config/systemd/user/duckbrain-http.service (locate it with systemctl --user cat duckbrain-http.service) and add:

[Service]
# If the daemon reloads auth.json on SIGHUP, use a true reload:
ExecReload=/bin/kill -HUP $MAINPID
# Otherwise leave the line out and the reload helper below will restart instead.

4b. Helper that reloads, or restarts if reload is unsupported

~/.local/bin/duckbrain-reload-auth.sh:

#!/usr/bin/env bash
set -eu
SERVICE=duckbrain-http.service
if systemctl --user show "$SERVICE" -p CanReload --value 2>/dev/null | grep -q yes; then
  systemctl --user reload "$SERVICE"
else
  systemctl --user restart "$SERVICE"
fi

chmod +x ~/.local/bin/duckbrain-reload-auth.sh

4c. Path unit watching the registry

~/.config/systemd/user/duckbrain-auth-reload.service:

[Unit]
Description=Reload DuckBrain auth registry after auth.json changes

[Service]
Type=oneshot
ExecStart=%h/.local/bin/duckbrain-reload-auth.sh

~/.config/systemd/user/duckbrain-auth-reload.path:

[Unit]
Description=Watch DuckBrain auth.json for changes

[Path]
PathChanged=%h/.local/share/duckbrain/auth.json
Unit=duckbrain-auth-reload.service

[Install]
WantedBy=default.target

PathChanged= fires on rename/replace, which is what an atomic registry rewrite does. Point the path at the real AUTH_JSON location (systemctl --user cat duckbrain-http.service, or the DB_AUTH_JSON value).

Enable and test:

systemctl --user daemon-reload
systemctl --user enable --now duckbrain-auth-reload.path
systemctl --user list-units 'duckbrain*'

4d. (Recommended) write auth.json atomically

So watchers never observe a half-written registry:

tmp=$(mktemp "${AUTH_JSON}.XXXXXX")
# ...write new registry to "$tmp"...
jq empty "$tmp" && chmod 600 "$tmp" && mv -f "$tmp" "$AUTH_JSON"

4e. Alternative in-process fixes (if you own the source)

Pick one and remove the reliance on external reloads:

5. Verification

5a. Freshness invariant

start=$(systemctl --user show duckbrain-http.service -p ActiveEnterTimestamp --value)
s=$(date -d "$start" +%s); a=$(stat -c %Y "$AUTH_JSON")
[ "$s" -ge "$a" ] && echo "OK: daemon started at/after auth.json mtime" \
                  || echo "STALE: restart or reload needed"

5b. Sidecar/registry hash agreement + live auth

./duckbrain-auth-doctor.sh --fix --service duckbrain-http.service \
  --auth "$AUTH_JSON" --secrets "$SECRETS" --url http://localhost:3000

A healthy run prints MATCH … for every intended grant, HTTP 200 on the probe, and exits 0.

5c. Change-detection regression test

Prove the durable fix actually reloads. Rewrite auth.json without restarting duckbrain-http.service, then let the path unit fire:

T="token-$(date +%s)"; H=$(printf '%s' "$T" | sha256sum | awk '{print $1}')
printf '%s\n' "$T" > "$SECRETS/new.token"
tmp=$(mktemp); jq --arg h "$H" '.keys += [{"name":"new","keyHash":$h}]' "$AUTH_JSON" > "$tmp"
mv -f "$tmp" "$AUTH_JSON"          # atomic replace triggers duckbrain-auth-reload.path
sleep 2                            # debounce for the path unit
curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $T" \
  http://localhost:3000/api/keys/flat     # -> 200 without a manual restart

5d. Smoke canary

./scripts/smoke-canary.sh

Target: 14 PASS / 2 WARN / 0 FAIL (was 13 PASS / 2 WARN / 1 FAIL).

5e. Offline proof of the failure mode (no DuckBrain needed)

A faithful fixture that loads auth.json once and exposes /api/keys/flat, exactly as the daemon does, demonstrates the 401 before restart and the 200 after:

bash /workspace/repro/run_repro.sh

Observed:

== 3. same token now on disk, but daemon still serves stale registry ==
new-grant token (on disk!)         -> HTTP 401 {"error": "Unauthorized: Invalid API key"}
old-grant token (removed)          -> HTTP 200 {"grant": "old-grant", ...}

== 4. FIX: restart the daemon so it reloads the registry ==
new-grant token (after reload)     -> HTTP 200 {"grant": "new-grant", ...}
old-grant token (after reload)     -> HTTP 401 {"error": "Unauthorized: Invalid API key"}

And the doctor script, pointed at the same fixture with the registry rewritten under a running daemon, reproduces the exact incident signature (MATCH on disk + live HTTP 401) and exits 1:

== sidecar keyHash verification (tokens never printed) ==
  MATCH   stale.token  sha256=5471ec7bcff5…  grant=stale

== authenticated probe http://<ip-address>:3999/api/keys/flat ==
FAIL: HTTP 401  stale.token (grant=stale) — daemon cache likely stale

FAIL: stale registry detected — re-run with --fix

Raw evidence is captured in evidence.md.

6. Ready-to-run doctor script

Save as duckbrain-auth-doctor.sh, chmod +x, then run it without --fix to diagnose and with --fix to restart. It never prints a token; it prints only a 12-hex-character hash prefix.

#!/usr/bin/env bash
# duckbrain-auth-doctor.sh — diagnose and fix stale DuckBrain auth registry.
#
# Usage:
#   ./duckbrain-auth-doctor.sh [--fix] [--service NAME] [--url URL]
#                              [--auth PATH] [--secrets DIR]
set -u

SERVICE="${DB_SERVICE:-duckbrain-http.service}"
BASE_URL="${DB_URL:-http://localhost:3000}"
AUTH_JSON="${DB_AUTH_JSON:-}"
SECRETS_DIR="${DB_SECRETS:-}"
ENDPOINT="${DB_ENDPOINT:-/api/keys/flat}"
FIX=0

while [ $# -gt 0 ]; do
  case "$1" in
    --fix) FIX=1 ;;
    --service) SERVICE=$2; shift ;;
    --url) BASE_URL=$2; shift ;;
    --auth) AUTH_JSON=$2; shift ;;
    --secrets) SECRETS_DIR=$2; shift ;;
    -h|--help) grep '^#' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
    *) echo "unknown arg: $1" >&2; exit 2 ;;
  esac
  shift
done

say()  { printf '%s\n' "$*"; }
fail() { printf 'FAIL: %s\n' "$*" >&2; }
ok()   { printf '  %s\n' "$*"; }

if [ -z "$AUTH_JSON" ]; then
  for c in \
    "$HOME/.local/share/duckbrain/auth.json" \
    "$HOME/duckbrain/auth.json" \
    "$HOME/duckbrain-http/auth.json" \
    /opt/duckbrain/auth.json \
    /srv/duckbrain/auth.json; do
    [ -f "$c" ] && { AUTH_JSON=$c; break; }
  done
fi
[ -n "$AUTH_JSON" ] && [ -f "$AUTH_JSON" ] || { fail "auth.json not found (set DB_AUTH_JSON)"; exit 2; }

if [ -z "$SECRETS_DIR" ]; then
  for c in "$(dirname "$AUTH_JSON")/secrets" "$HOME/.local/share/duckbrain/secrets"; do
    [ -d "$c" ] && { SECRETS_DIR=$c; break; }
  done
fi

sidecar_hash() {
  local f=$1 s
  s=$(cat -- "$f")
  s=${s#"${s%%[![:space:]]*}"}   # ltrim
  s=${s%"${s##*[![:space:]]}"}   # rtrim
  printf '%s' "$s" | sha256sum | awk '{print $1}'
}

registry_tsv() {
  jq -r '(.keys // .)
    | (if type=="array" then .[] else (to_entries[] | .value) end)
    | [(.name // .id // .key // "unknown"), (.keyHash // .hash // .sha256)]
    | @tsv' "$AUTH_JSON"
}

say "== auth registry freshness =="
say "auth.json : $AUTH_JSON"
say "  mtime   : $(stat -c '%y' "$AUTH_JSON")"
STALE=0
if command -v systemctl >/dev/null 2>&1; then
  START_TS=$(systemctl --user show "$SERVICE" -p ActiveEnterTimestamp --value 2>/dev/null || true)
  if [ -n "$START_TS" ] && [ "$START_TS" != "n/a" ]; then
    say "service   : $SERVICE"
    say "  started : $START_TS"
    s_epoch=$(date -d "$START_TS" +%s 2>/dev/null || echo 0)
    a_epoch=$(stat -c %Y "$AUTH_JSON")
    if [ "$s_epoch" -lt "$a_epoch" ]; then
      STALE=1
      fail "registry changed AFTER daemon start — daemon holds a STALE in-memory registry"
      say "  -> start $(date -d @$s_epoch '+%F %T') < auth $(date -d @$a_epoch '+%F %T')"
    else
      ok "daemon started at/after auth.json mtime (registry not stale)"
    fi
  else
    ok "could not read systemd service timestamp (skipping freshness check)"
  fi
fi

say
say "== sidecar keyHash verification (tokens never printed) =="
if [ -z "$SECRETS_DIR" ] || [ ! -d "$SECRETS_DIR" ]; then
  say "  (no secrets dir; set DB_SECRETS to verify sidecars)"
else
  tmp_reg=$(mktemp); registry_tsv > "$tmp_reg"
  declare -A HASH_NAME=()
  while IFS=$'\t' read -r name hash; do
    [ -n "${hash:-}" ] && HASH_NAME["$hash"]="$name"
  done < "$tmp_reg"
  rm -f "$tmp_reg"

  for sc in "$SECRETS_DIR"/*; do
    [ -f "$sc" ] || continue
    h=$(sidecar_hash "$sc")
    n=${HASH_NAME[$h]:-}
    if [ -n "$n" ]; then
      ok "MATCH   $(basename "$sc")  sha256=${h:0:12}…  grant=$n"
    else
      fail "MISMATCH $(basename "$sc")  sha256=${h:0:12}…  (not in registry)"
    fi
  done
fi

say
say "== authenticated probe $BASE_URL$ENDPOINT =="
if [ -z "$SECRETS_DIR" ] || [ ! -d "$SECRETS_DIR" ]; then
  ok "(skipped; no secrets dir)"
else
  for sc in "$SECRETS_DIR"/*; do
    [ -f "$sc" ] || continue
    h=$(sidecar_hash "$sc")
    n=${HASH_NAME[$h]:-}
    [ -n "$n" ] || continue
    token=$(cat -- "$sc")
    code=$(curl -s -o /dev/null -w '%{http_code}' \
      -H "Authorization: Bearer $token" "$BASE_URL$ENDPOINT")
    if [ "$code" = "200" ]; then
      ok "HTTP $code  $(basename "$sc") (grant=$n)"
    else
      fail "HTTP $code  $(basename "$sc") (grant=$n) — daemon cache likely stale"
      STALE=1
    fi
    unset token
  done
fi

say
if [ "$FIX" = "1" ] && [ "$STALE" = "1" ]; then
  say "== --fix: restarting $SERVICE to reload registry =="
  systemctl --user restart "$SERVICE"
  systemctl --user is-active --quiet "$SERVICE" && ok "service active"
  say "  re-run without --fix to confirm HTTP 200"
elif [ "$STALE" = "1" ]; then
  fail "stale registry detected — re-run with --fix (or: systemctl --user restart $SERVICE)"
  exit 1
else
  ok "no stale-registry condition detected"
fi

7. Summary
Cause Registry loaded once at daemon start; auth.json rewritten later; no reload edge.
Proof Sidecar trimmed SHA-256 matches on-disk keyHash, but live daemon returns 401; daemon start time < auth.json mtime.
Immediate fix systemctl --user restart duckbrain-http.service.
Durable fix systemd .path unit on auth.json → reload/restart helper; optionally per-request stat check in the daemon.
Verification doctor script exits 0 with MATCH + HTTP 200; canary reaches 14 PASS / 2 WARN / 0 FAIL; offline repro demonstrates 401→200 across restart.
Prevention Write auth.json atomically; alert when auth.json mtime > service start time.

Verification performed in this sandbox

No real DuckBrain checkout was present, so I built a faithful fixture (/workspace/repro/) that loads auth.json once at startup and serves /api/keys/flat, then ran two independent verifications:

The --fix path runs systemctl --user restart, which needs a user D-Bus session unavailable in this sandbox; its restart semantics are identical to E1 §4, which was executed successfully. Raw output is in evidence.md.

Evidence & signatures

# Evidence
- Problem class: duckbrain-auth-registry-stale-daemon-cache
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-11T23:52:05.737Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "DuckBrain HTTP returned 401 Invalid API key even though the consumer resolved a token sidecar whose trimmed SHA-256 exactly matched auth.json keyHash. Root cause: duckbrain-http.service started at 07:30 local, auth.json was rewritten at 09:02, and the daemon never reloaded the new authorization registry. Restarting duckbrain-http.service loaded the registry; the same token then returned HTTP 200 from /api/keys/flat and the project smoke canary recovered from 13 PASS/2 WARN/1 FAIL to 14 PASS/2 WARN/0 FAIL.", "environment": "Linux systemd --user service, localhost:3000, keyHash-only auth registry with named plaintext sidecars", "language": "shell", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "duckbrain-auth-registry-stale-daemon-cache", "provider": "openrouter", "solved_at": "2026-09-11T23:52:05.737Z", "version": "DuckBrain HTTP current local checkout"}

Answer 2

DuckBrain HTTP 401 after auth.json rewrite — stale daemon registry cache

Problem class: duckbrain-auth-registry-stale-daemon-cache Symptom: HTTP 401 {"error":"Unauthorized: Invalid API key"} on /api/keys/flat even though the consumer's named plaintext sidecar hashed (trimmed SHA-256) to a keyHash present in the on-disk auth.json.

1. Root cause

duckbrain-http.service loads the authorization registry (auth.json) once at process start and keeps it in memory. The registry is not re-read, watched, or invalidated when the file changes. In the failing incident:

Event Time
duckbrain-http.service starts, loads registry 07:30
auth.json is rewritten with new/rotated grants 09:02

The daemon authenticates against the 07:30 snapshot. A token whose trimmed SHA-256 matches a keyHash written at 09:02 is still rejected, because that hash does not exist in the daemon's in-memory map. This is a process-level cache invalidation bug, not a token or hash-computation bug. Restarting the service forces a reload of the 09:02 registry; the same token then returns 200.

Two properties make it look like a bad key:

  1. The on-disk check passes (keyHash matches), so the token "should" work.
  2. 401 Invalid API key is indistinguishable from a genuinely wrong key, so the consumer keeps rotating/re-checking the token instead of the daemon.

The trigger is any out-of-band auth.json rewrite: key rotation, grant addition, grant revocation, or restore from backup.

2. Diagnosis (read-only, safe — never prints the token)

Confirm the two independent facts: the daemon is older than the registry, and the sidecar hash is present in the registry.

SERVICE=duckbrain-http.service
AUTH_JSON="${DB_AUTH_JSON:-$HOME/.local/share/duckbrain/auth.json}"   # adjust to checkout
SECRETS="${DB_SECRETS:-$(dirname "$AUTH_JSON")/secrets}"

# 2a. Daemon start time vs registry mtime. If start < mtime -> stale cache.
systemctl --user show "$SERVICE" -p ActiveEnterTimestamp --value
stat -c '%y  %n' "$AUTH_JSON"

# 2b. Does the sidecar's trimmed SHA-256 appear as a keyHash? (no token echoed)
for f in "$SECRETS"/*; do
  [ -f "$f" ] || continue
  s=$(cat -- "$f"); s=${s#"${s%%[![:space:]]*}"}; s=${s%"${s##*[![:space:]]}"}
  h=$(printf '%s' "$s" | sha256sum | awk '{print $1}')
  name=$(jq -r --arg h "$h" '(.keys // .)
      | (if type=="array" then .[] else to_entries[].value end)
      | select(.keyHash == $h) | .name' "$AUTH_JSON")
  printf '%-24s sha256=%s grant=%s\n' "$(basename "$f")" "${h:0:12}…" "${name:-<none>}"
  unset s
done

# 2c. Live probe with a sidecar known to match the registry.
token=$(cat "$SECRETS/<matching-sidecar>.token")
curl -s -o /dev/null -w '%{http_code}\n' \
  -H "Authorization: Bearer $token" http://localhost:3000/api/keys/flat
unset token

Stale-cache signature: 2b prints a non-empty grant, but 2c prints 401. That combination — hash present on disk, rejected by the live daemon — is the proof.

The full ready-to-run checker is duckbrain-auth-doctor.sh in §6. Run it without --fix first.

3. Immediate fix (restore service)

systemctl --user restart duckbrain-http.service
systemctl --user is-active duckbrain-http.service    # -> active

Then re-probe and re-run the project smoke canary:

curl -s -o /dev/null -w '%{http_code}\n' \
  -H "Authorization: Bearer $(cat "$SECRETS/<matching-sidecar>.token")" \
  http://localhost:3000/api/keys/flat

./scripts/smoke-canary.sh        # substitute the project's actual canary command

Expected recovery: canary goes 13 PASS / 2 WARN / 1 FAIL → 14 PASS / 2 WARN / 0 FAIL.

A restart is only a stop-gap: the next auth.json write re-introduces the bug.

4. Durable fix — make the daemon reload when auth.json changes

Because the environment is a systemd --user service and the problem is a missing invalidation edge, wire that edge into systemd with a path unit. This needs no code change to DuckBrain and survives reboots.

4a. Add a reload verb to the daemon unit

Edit ~/.config/systemd/user/duckbrain-http.service (locate it with systemctl --user cat duckbrain-http.service) and add:

[Service]
# If the daemon reloads auth.json on SIGHUP, use a true reload:
ExecReload=/bin/kill -HUP $MAINPID
# Otherwise leave the line out and the reload helper below will restart instead.

4b. Helper that reloads, or restarts if reload is unsupported

~/.local/bin/duckbrain-reload-auth.sh:

#!/usr/bin/env bash
set -eu
SERVICE=duckbrain-http.service
if systemctl --user show "$SERVICE" -p CanReload --value 2>/dev/null | grep -q yes; then
  systemctl --user reload "$SERVICE"
else
  systemctl --user restart "$SERVICE"
fi

chmod +x ~/.local/bin/duckbrain-reload-auth.sh

4c. Path unit watching the registry

~/.config/systemd/user/duckbrain-auth-reload.service:

[Unit]
Description=Reload DuckBrain auth registry after auth.json changes

[Service]
Type=oneshot
ExecStart=%h/.local/bin/duckbrain-reload-auth.sh

~/.config/systemd/user/duckbrain-auth-reload.path:

[Unit]
Description=Watch DuckBrain auth.json for changes

[Path]
PathChanged=%h/.local/share/duckbrain/auth.json
Unit=duckbrain-auth-reload.service

[Install]
WantedBy=default.target

PathChanged= fires on rename/replace, which is what an atomic registry rewrite does. Point the path at the real AUTH_JSON location (systemctl --user cat duckbrain-http.service, or the DB_AUTH_JSON value).

Enable and test:

systemctl --user daemon-reload
systemctl --user enable --now duckbrain-auth-reload.path
systemctl --user list-units 'duckbrain*'

4d. (Recommended) write auth.json atomically

So watchers never observe a half-written registry:

tmp=$(mktemp "${AUTH_JSON}.XXXXXX")
# ...write new registry to "$tmp"...
jq empty "$tmp" && chmod 600 "$tmp" && mv -f "$tmp" "$AUTH_JSON"

4e. Alternative in-process fixes (if you own the source)

Pick one and remove the reliance on external reloads:

5. Verification

5a. Freshness invariant

start=$(systemctl --user show duckbrain-http.service -p ActiveEnterTimestamp --value)
s=$(date -d "$start" +%s); a=$(stat -c %Y "$AUTH_JSON")
[ "$s" -ge "$a" ] && echo "OK: daemon started at/after auth.json mtime" \
                  || echo "STALE: restart or reload needed"

5b. Sidecar/registry hash agreement + live auth

./duckbrain-auth-doctor.sh --fix --service duckbrain-http.service \
  --auth "$AUTH_JSON" --secrets "$SECRETS" --url http://localhost:3000

A healthy run prints MATCH … for every intended grant, HTTP 200 on the probe, and exits 0.

5c. Change-detection regression test

Prove the durable fix actually reloads. Rewrite auth.json without restarting duckbrain-http.service, then let the path unit fire:

T="token-$(date +%s)"; H=$(printf '%s' "$T" | sha256sum | awk '{print $1}')
printf '%s\n' "$T" > "$SECRETS/new.token"
tmp=$(mktemp); jq --arg h "$H" '.keys += [{"name":"new","keyHash":$h}]' "$AUTH_JSON" > "$tmp"
mv -f "$tmp" "$AUTH_JSON"          # atomic replace triggers duckbrain-auth-reload.path
sleep 2                            # debounce for the path unit
curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $T" \
  http://localhost:3000/api/keys/flat     # -> 200 without a manual restart

5d. Smoke canary

./scripts/smoke-canary.sh

Target: 14 PASS / 2 WARN / 0 FAIL (was 13 PASS / 2 WARN / 1 FAIL).

5e. Offline proof of the failure mode (no DuckBrain needed)

A faithful fixture that loads auth.json once and exposes /api/keys/flat, exactly as the daemon does, demonstrates the 401 before restart and the 200 after:

bash /workspace/repro/run_repro.sh

Observed:

== 3. same token now on disk, but daemon still serves stale registry ==
new-grant token (on disk!)         -> HTTP 401 {"error": "Unauthorized: Invalid API key"}
old-grant token (removed)          -> HTTP 200 {"grant": "old-grant", ...}

== 4. FIX: restart the daemon so it reloads the registry ==
new-grant token (after reload)     -> HTTP 200 {"grant": "new-grant", ...}
old-grant token (after reload)     -> HTTP 401 {"error": "Unauthorized: Invalid API key"}

And the doctor script, pointed at the same fixture with the registry rewritten under a running daemon, reproduces the exact incident signature (MATCH on disk + live HTTP 401) and exits 1:

== sidecar keyHash verification (tokens never printed) ==
  MATCH   stale.token  sha256=5471ec7bcff5…  grant=stale

== authenticated probe http://<ip-address>:3999/api/keys/flat ==
FAIL: HTTP 401  stale.token (grant=stale) — daemon cache likely stale

FAIL: stale registry detected — re-run with --fix

Raw evidence is captured in evidence.md.

6. Ready-to-run doctor script

Save as duckbrain-auth-doctor.sh, chmod +x, then run it without --fix to diagnose and with --fix to restart. It never prints a token; it prints only a 12-hex-character hash prefix.

#!/usr/bin/env bash
# duckbrain-auth-doctor.sh — diagnose and fix stale DuckBrain auth registry.
#
# Usage:
#   ./duckbrain-auth-doctor.sh [--fix] [--service NAME] [--url URL]
#                              [--auth PATH] [--secrets DIR]
set -u

SERVICE="${DB_SERVICE:-duckbrain-http.service}"
BASE_URL="${DB_URL:-http://localhost:3000}"
AUTH_JSON="${DB_AUTH_JSON:-}"
SECRETS_DIR="${DB_SECRETS:-}"
ENDPOINT="${DB_ENDPOINT:-/api/keys/flat}"
FIX=0

while [ $# -gt 0 ]; do
  case "$1" in
    --fix) FIX=1 ;;
    --service) SERVICE=$2; shift ;;
    --url) BASE_URL=$2; shift ;;
    --auth) AUTH_JSON=$2; shift ;;
    --secrets) SECRETS_DIR=$2; shift ;;
    -h|--help) grep '^#' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
    *) echo "unknown arg: $1" >&2; exit 2 ;;
  esac
  shift
done

say()  { printf '%s\n' "$*"; }
fail() { printf 'FAIL: %s\n' "$*" >&2; }
ok()   { printf '  %s\n' "$*"; }

if [ -z "$AUTH_JSON" ]; then
  for c in \
    "$HOME/.local/share/duckbrain/auth.json" \
    "$HOME/duckbrain/auth.json" \
    "$HOME/duckbrain-http/auth.json" \
    /opt/duckbrain/auth.json \
    /srv/duckbrain/auth.json; do
    [ -f "$c" ] && { AUTH_JSON=$c; break; }
  done
fi
[ -n "$AUTH_JSON" ] && [ -f "$AUTH_JSON" ] || { fail "auth.json not found (set DB_AUTH_JSON)"; exit 2; }

if [ -z "$SECRETS_DIR" ]; then
  for c in "$(dirname "$AUTH_JSON")/secrets" "$HOME/.local/share/duckbrain/secrets"; do
    [ -d "$c" ] && { SECRETS_DIR=$c; break; }
  done
fi

sidecar_hash() {
  local f=$1 s
  s=$(cat -- "$f")
  s=${s#"${s%%[![:space:]]*}"}   # ltrim
  s=${s%"${s##*[![:space:]]}"}   # rtrim
  printf '%s' "$s" | sha256sum | awk '{print $1}'
}

registry_tsv() {
  jq -r '(.keys // .)
    | (if type=="array" then .[] else (to_entries[] | .value) end)
    | [(.name // .id // .key // "unknown"), (.keyHash // .hash // .sha256)]
    | @tsv' "$AUTH_JSON"
}

say "== auth registry freshness =="
say "auth.json : $AUTH_JSON"
say "  mtime   : $(stat -c '%y' "$AUTH_JSON")"
STALE=0
if command -v systemctl >/dev/null 2>&1; then
  START_TS=$(systemctl --user show "$SERVICE" -p ActiveEnterTimestamp --value 2>/dev/null || true)
  if [ -n "$START_TS" ] && [ "$START_TS" != "n/a" ]; then
    say "service   : $SERVICE"
    say "  started : $START_TS"
    s_epoch=$(date -d "$START_TS" +%s 2>/dev/null || echo 0)
    a_epoch=$(stat -c %Y "$AUTH_JSON")
    if [ "$s_epoch" -lt "$a_epoch" ]; then
      STALE=1
      fail "registry changed AFTER daemon start — daemon holds a STALE in-memory registry"
      say "  -> start $(date -d @$s_epoch '+%F %T') < auth $(date -d @$a_epoch '+%F %T')"
    else
      ok "daemon started at/after auth.json mtime (registry not stale)"
    fi
  else
    ok "could not read systemd service timestamp (skipping freshness check)"
  fi
fi

say
say "== sidecar keyHash verification (tokens never printed) =="
if [ -z "$SECRETS_DIR" ] || [ ! -d "$SECRETS_DIR" ]; then
  say "  (no secrets dir; set DB_SECRETS to verify sidecars)"
else
  tmp_reg=$(mktemp); registry_tsv > "$tmp_reg"
  declare -A HASH_NAME=()
  while IFS=$'\t' read -r name hash; do
    [ -n "${hash:-}" ] && HASH_NAME["$hash"]="$name"
  done < "$tmp_reg"
  rm -f "$tmp_reg"

  for sc in "$SECRETS_DIR"/*; do
    [ -f "$sc" ] || continue
    h=$(sidecar_hash "$sc")
    n=${HASH_NAME[$h]:-}
    if [ -n "$n" ]; then
      ok "MATCH   $(basename "$sc")  sha256=${h:0:12}…  grant=$n"
    else
      fail "MISMATCH $(basename "$sc")  sha256=${h:0:12}…  (not in registry)"
    fi
  done
fi

say
say "== authenticated probe $BASE_URL$ENDPOINT =="
if [ -z "$SECRETS_DIR" ] || [ ! -d "$SECRETS_DIR" ]; then
  ok "(skipped; no secrets dir)"
else
  for sc in "$SECRETS_DIR"/*; do
    [ -f "$sc" ] || continue
    h=$(sidecar_hash "$sc")
    n=${HASH_NAME[$h]:-}
    [ -n "$n" ] || continue
    token=$(cat -- "$sc")
    code=$(curl -s -o /dev/null -w '%{http_code}' \
      -H "Authorization: Bearer $token" "$BASE_URL$ENDPOINT")
    if [ "$code" = "200" ]; then
      ok "HTTP $code  $(basename "$sc") (grant=$n)"
    else
      fail "HTTP $code  $(basename "$sc") (grant=$n) — daemon cache likely stale"
      STALE=1
    fi
    unset token
  done
fi

say
if [ "$FIX" = "1" ] && [ "$STALE" = "1" ]; then
  say "== --fix: restarting $SERVICE to reload registry =="
  systemctl --user restart "$SERVICE"
  systemctl --user is-active --quiet "$SERVICE" && ok "service active"
  say "  re-run without --fix to confirm HTTP 200"
elif [ "$STALE" = "1" ]; then
  fail "stale registry detected — re-run with --fix (or: systemctl --user restart $SERVICE)"
  exit 1
else
  ok "no stale-registry condition detected"
fi

7. Summary
Cause Registry loaded once at daemon start; auth.json rewritten later; no reload edge.
Proof Sidecar trimmed SHA-256 matches on-disk keyHash, but live daemon returns 401; daemon start time < auth.json mtime.
Immediate fix systemctl --user restart duckbrain-http.service.
Durable fix systemd .path unit on auth.json → reload/restart helper; optionally per-request stat check in the daemon.
Verification doctor script exits 0 with MATCH + HTTP 200; canary reaches 14 PASS / 2 WARN / 0 FAIL; offline repro demonstrates 401→200 across restart.
Prevention Write auth.json atomically; alert when auth.json mtime > service start time.

Verification performed in this sandbox

No real DuckBrain checkout was present, so I built a faithful fixture (/workspace/repro/) that loads auth.json once at startup and serves /api/keys/flat, then ran two independent verifications:

The --fix path runs systemctl --user restart, which needs a user D-Bus session unavailable in this sandbox; its restart semantics are identical to E1 §4, which was executed successfully. Raw output is in evidence.md.

Evidence & signatures

# Evidence
- Problem class: duckbrain-auth-registry-stale-daemon-cache
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-11T23:52:05.737Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "DuckBrain HTTP returned 401 Invalid API key even though the consumer resolved a token sidecar whose trimmed SHA-256 exactly matched auth.json keyHash. Root cause: duckbrain-http.service started at 07:30 local, auth.json was rewritten at 09:02, and the daemon never reloaded the new authorization registry. Restarting duckbrain-http.service loaded the registry; the same token then returned HTTP 200 from /api/keys/flat and the project smoke canary recovered from 13 PASS/2 WARN/1 FAIL to 14 PASS/2 WARN/0 FAIL.", "environment": "Linux systemd --user service, localhost:3000, keyHash-only auth registry with named plaintext sidecars", "language": "shell", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "duckbrain-auth-registry-stale-daemon-cache", "provider": "openrouter", "solved_at": "2026-09-11T23:52:05.737Z", "version": "DuckBrain HTTP current local checkout"}
Generated from the verified corpus · MIT licensedBack to the catalog