go-scheduler-test-dispatches-live-tick
Diagnosis and fix complete. The self-contained write-up is at /workspace/scheduler/SOLUTION-SCHED-GAP-1734.md.
TestUsagePoolLeaseReleasedWhenStartFails treated an empty SCHEDULER_DISPATCH_TARGETS file as "dispatch disabled". It is not: per dispatch_targets.go, empty/missing means "no lane is remote", so every lane keeps its historical local transport. The test's spawner (NewSpawner(db, 1)) had no gateway client and noExecFallback == false, so Spawn() fell through to exec.Command("hermes", …) — the live CLI/gateway. The SQLite RAISE(IGNORE) trigger makes StartRunning return nil (only the row update is skipped), so the slot pool proceeds past the cleared gate and dispatches. Confirmed by code trace and a safe repro:
SPAWN: start-fails-dbg tick=… pid=… workdir=/tmp
LEAK: hermes invoked with: chat -q [Scheduler tick: start-fails-dbg-…] … Workdir: /tmp.
It's intermittent because SlotPool.Spawn is fire-and-forget and Wait returns when Running()==0; when the goroutine loses the race, test cleanup closes the DB first (sql: database is closed) and nothing leaks — which is why only some suite runs produced a phantom tick.
internal/scheduler/usage_pool_policy_test.go — added newHermeticBookkeepingSpawner (no gateway client + SetNoExecFallback(true)) and used it in the two bookkeeping fixtures, removing the misleading empty-targets setup. dispatch_targets.go — documented the "empty ≠ disabled" footgun at EnvDispatchTargets. Byte semantics of the targets file are intentionally unchanged.
hermes first on PATH, zero hermes invocations occur; the fixed spawner now logs DEFERRED … no gateway client and exec fallback disabled.TestUsagePool*, TestRemote*, TestDispatchLeg* pass (1.045s), go build ./... clean.Scheduler tick: start-fails- lines in ~/.hermes/foreman/logs/agent.log before/after — count must not increase while the lease assertion still passes.Fleet-side detection: fixture-named tick + workdir /tmp + empty worker model/provider = leaked test dispatch — report, do not provision.
# Evidence - Problem class: go-scheduler-test-dispatches-live-tick - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-10-10T10:49:22.131Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "SYMPTOM: a scheduler unit test mints a REAL foreman tick and dispatches a live Hermes session. Phantom tick ids of the shape <fixture-name>-<YYYY-MM-DD-HH-MM-SS> appear in the fleet one per suite run: start-fails-2026-10-05-05-28-58, start-fails-2026-10-07-01-07-44, start-fails-2026-10-07-01-07-50 (double dispatch, 6s apart, consistent with two suite invocations or a -race re-run), start-fails-2026-10-07-17-59-06, start-fails-2026-10-10-10-39-28. Each leaked session consumes one of the scheduler's 10 slots plus a whole foreman turn; the 2026-10-05 instance ran with workdir /tmp, no scheduler project row, an empty worker model/provider, and still went as far as a branch-ref push and a DuckBrain write. ROOT CAUSE: internal/scheduler/usage_pool_policy_test.go, TestUsagePoolLeaseReleasedWhenStartFails, sets SCHEDULER_DISPATCH_TARGETS to a file written with an EMPTY body (writeDispatchTargets(t, \"\")) believing that neutralizes dispatch. Per internal/scheduler/dispatch_targets.go a missing-or-empty targets file means 'no lane is remote', i.e. every lane keeps its historical LOCAL behavior = spawn through the gateway. The test therefore ENABLES the very dispatch leg it meant to disable. pool.Spawn(PackedProject{Name:\"start-fails\", Workdir:\"/tmp\"}) then mints the tick id and drives the spawn to the endpoint leg; resolveTickEndpoint (spawn.go, gateway_endpoint.go) merges the project row's lane/namespace endpoint with the spawner's global gateway, and a project row carrying no gateway_url (the test insert has only name/repo_url/workdir) falls through to the GLOBAL configured endpoint, which is the live gateway. SCOPE DETAIL: sibling fixtures leak nothing - the slot-timeout fixture never clears the slot gate, and the panic fixture uses a nil spawner in a child process - so the offender is specifically the trigger-refused path where the tick clears the slot gate before the queued-to-running status update is ignored. REPRO HAZARD: do NOT reproduce by running go test -run TestUsagePoolLeaseReleasedWhenStartFails on a box with the gateway live; that run itself leaks another tick. VERIFY instead by counting 'Scheduler tick: start-fails-' lines in ~/.hermes/foreman/logs/agent.log before and after the run (the count must NOT increase) while the usage-lease-release assertion still passes. FIX: neutralize the dispatch leg inside the test (point the spawner's gateway endpoint at an unreachable address, or inject a test-double gateway client) instead of relying on an empty dispatch-targets file; optionally add a build-time guard that a test-constructed Spawner never resolves the global endpoint. FLEET-SIDE DETECTION for any future leak: a tick whose id carries a test-fixture name, whose workdir is /tmp, and whose worker model/provider are empty is a leaked test dispatch, not a project - diagnose and report, do NOT provision it.", "environment": "control box with the fleet gateway LIVE; coding-hermes scheduler repo (github.com/coding-hermes/scheduler); suite run as go test -short -p 1 ./...", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "go-scheduler-test-dispatches-live-tick", "provider": "openrouter", "solved_at": "2026-10-10T10:49:22.132Z", "version": ""}