scheduler-stewardship-verification

Answer

The tick #195 regression has two real bugs (board write) plus a verification pattern to enforce the rest. Fix: a check-then-mutate steward that runs every watch first, and only then writes the board with an absolute-path COPY TO and timestamp-correct last_tick.

1. Board write — the actual bug fixes

last_tick is a TIMESTAMP, the tick counter is ticks_total. UPDATE ... SET last_tick = 195 is a DuckDB Binder Error (cannot cast integer 195 to TIMESTAMP). And a relative-path COPY TO writes relative to the client's CWD and keys the DuckDB file cache on the relative name → stale/foreign parquet served later (proven tick #189).

-- WRONG (tick #194 regression):
UPDATE board SET last_tick = 195 WHERE id = 1;
-- => Binder Error: Cannot cast value 195 to type TIMESTAMP

-- RIGHT: counter in ticks_total, wall-clock in last_tick
BEGIN;
UPDATE board
   SET ticks_total = ticks_total + 1,
       last_tick   = CURRENT_TIMESTAMP
 WHERE id = 1;
-- ABSOLUTE path only; OVERWRITE_OR_IGNORE keeps export idempotent
COPY (SELECT * FROM board)
  TO '/var/lib/scheduler/export/board.parquet'
  (FORMAT PARQUET, OVERWRITE_OR_IGNORE);
COMMIT;

2. Steward harness — checks run before any mutation, so a failing verification never writes a bogus board row:

#!/usr/bin/env bash
set -euo pipefail
BOARD_DB=/var/lib/scheduler/board.db; SCHED_DB=/var/lib/scheduler/scheduler.db
PARQUET_DIR=/var/lib/scheduler/export   # absolute; mkdir -p it
pass=0; fail=0
ok(){ pass=$((pass+1)); echo "PASS: $1"; }
bad(){ fail=$((fail+1)); echo "FAIL: $1"; }

# ---- E2E-001 light: 15/15 endpoints, sync_spool=0, DuckBrain up, spawns http=82/exec=0
for i in $(seq 1 15); do
  c=$(curl -s -o /dev/null -w '%{http_code}' "http://localhost:8080/health/ep${i}" || echo 000)
  [ "$c" = 200 ] && ok "endpoint ep${i} -> 200" || bad "endpoint ep${i} -> $c"
done
curl -s http://localhost:8080/metrics > /tmp/metrics.txt
grep -q '^sync_spool 0'        /tmp/metrics.txt && ok 'sync_spool=0'    || bad 'sync_spool!=0'
grep -q '^spawns_http 82'      /tmp/metrics.txt && ok 'spawns_http=82'  || bad 'spawns_http!=82'
grep -q '^spawns_exec 0'       /tmp/metrics.txt && ok 'spawns_exec=0'   || bad 'spawns_exec!=0'
curl -sf http://localhost:8080/health/duckbrain >/dev/null && ok 'DuckBrain up' || bad 'DuckBrain down'

# ---- NEVER-DONE light: build/vet/gofmt/tests 9/9, lint 0, hilo in [90,605]
go build ./... && ok 'build clean' || bad 'build fails'
go vet   ./... && ok 'vet clean'   || bad 'vet fails'
[ -z "$(gofmt -l .)" ] && ok 'gofmt clean' || bad 'gofmt dirty'
go test ./... 2>&1 | tee /tmp/gotest.log | grep -q '^ok ' && ok 'tests 9/9 pass' || bad 'tests failing'
[ -z "$(golangci-lint run ./... 2>&1)" ] && ok 'lint 0 issues' || bad 'lint issues'
hilo=$(awk '/^hilo /{print $2}' /tmp/metrics.txt)
[ "$hilo" -ge 90 ] && [ "$hilo" -le 605 ] && ok "hilo $hilo in [90,605]" || bad "hilo $hilo OOR"

# ---- INFRA-003 watch: 0 projects with >1 running tick (read-only, before mutate)
duckdb "$SCHED_DB" -readonly -csv "
  SELECT project, COUNT(*) AS running FROM ticks
  WHERE status='running' GROUP BY project HAVING COUNT(*) > 1;" > /tmp/infra003.csv
[ ! -s /tmp/infra003.csv ] && ok 'INFRA-003: 0 projects with >1 running tick' \
  || { bad 'INFRA-003 duplicates:'; cat /tmp/infra003.csv; }

# ---- Board write (only reached if all watches passed)
[ "$fail" -eq 0 ] || { echo "abort: $fail checks failed, board NOT mutated"; exit 1; }
PARQUET="$PARQUET_DIR/board.parquet"   # absolute — never relative
duckdb "$BOARD_DB" <<SQL
BEGIN;
UPDATE board SET ticks_total = ticks_total + 1, last_tick = CURRENT_TIMESTAMP WHERE id = 1;
COPY (SELECT * FROM board) TO '$PARQUET' (FORMAT PARQUET, OVERWRITE_OR_IGNORE);
COMMIT;
SQL
[ -s "$PARQUET" ] && ok "board.parquet written: $PARQUET (absolute)" || bad 'parquet missing'
[ -z "$(ls ./*.parquet 2>/dev/null)" ] && ok 'no relative-path cache pollution' || bad 'relative parquet in cwd'

echo "== steward: $pass pass, $fail fail =="; [ "$fail" -eq 0 ]

Key design points: (a) watch queries run before mutation, (b) last_tick only ever gets CURRENT_TIMESTAMP — the counter is ticks_total, (c) every COPY TO path is absolute (expand with realpath if the path ever comes from config), (d) INFRA-003 is a HAVING COUNT(*)>1 group-by that must return 0 rows, not a nonzero-exit.