ops-stale-binary-deploy-verify

Answer

Root cause. The Foreman tick's "fix committed but not live" was not a code problem — it was a deploy/verification problem, confirmed by two misreadings:

1. The deploy rebuilt nothing. The binary's mtime pre-dated the fix commit. The deploy step ran make against a stamp/cache artifact instead of the source, so make concluded "up to date" and never recompiled the fix into the binary.
2. pgrep self-match faked a restart. pgrep -f <query> matches any cmdline containing the query string — including the shell that invoked pgrep itself. That returned a "recent" PID that was the checker, not the service, so monitoring declared the new binary live when it wasn't.

Fix — four parts: force a real rebuild with an mtime-vs-commit guard, rely on Restart=always for restart, then verify by endpoint behavior, and provision a scratch read-only instance for OB-GAP-020.

1. Deploy that actually rebuilds, with a commit-time guard (the key defect — svc.go is the real prerequisite, no stamp games; fail-closed if the binary is older than the fix commit):

# Fixed Makefile: binary depends on the source; make rebuilds when svc.go is newer.
svc: svc.go
    go build -o svc svc.go

#!/bin/sh
# deploy.sh — fixed: rebuild, then prove the fix is compiled in.
set -e
git pull --ff-only            # bring in fix commit OB-GAP-024
make svc

FIX_TS=$(git show -s --format=%ct HEAD)   # fix commit epoch
BIN_TS=$(stat -c %Y svc)
if [ "$BIN_TS" -lt "$FIX_TS" ]; then
    echo "FATAL: binary older than fix commit ($BIN_TS < $FIX_TS); force rebuild" >&2
    make -B svc
fi
[ "$(stat -c %Y svc)" -ge "$FIX_TS" ] || { echo "FATAL: rebuild failed" >&2; exit 1; }
echo "deploy ok: svc mtime=$(stat -c %Y svc) >= fix commit=$FIX_TS"

2. systemd unit with Restart=always — kill is sufficient to pick up the new binary:

[Service]
ExecStart=/opt/obgap/svc -readonly
Restart=always
RestartSec=0.2
ReadOnlyPaths=/var/lib/obgap     # scratch instance: discover allowed, submit 403
ProtectSystem=strict

3. Restart + live-verify by endpoint, not by process listing:

systemctl kill -s KILL obgap         # Restart=always respawns with the new binary
# poll the ENDPOINT until the fix is live (this is the deploy proof):
for i in $(seq 1 30); do
  status=$(curl -s localhost:18080/detail/OB-GAP-024 | sed -n 's/.*"status":"\([^"]*\)".*/\1/p')
  [ "$status" = "VERIFIED" ] && break; sleep 0.5
done
[ "$status" = "VERIFIED" ] || { echo "deploy NOT live"; exit 1; }

4. Read-only scratch instance for OB-GAP-020 (discover 200 / submit 403) — the -readonly runtime flag rejects writes with 403 while serving discovery:

http.HandleFunc("/discover", ...)                  // -> 200 always
http.HandleFunc("/submit", func(w, r) {
    if readonly { w.WriteHeader(403); return }     // scratch: submit forbidden
    ...
})

5. Fix the monitor: replace pgrep -f svc (self-matching) with an exact-name check plus a behavioral probe — pgrep -x svc for liveness only, and the HTTP detail-status probe for deploy truth.