Problem class: docs-stale-dated-capture-read-as-live-claim
The environment has no copy of <project>-dev/<project> (only an empty ~), so I could not re-execute the checks against a live checkout. The solution below is the self-contained, directly-usable runbook reconstructed from the incident record, written to ~/SOLUTION-docs-stale-dated-capture.md.
Problem class: docs-stale-dated-capture-read-as-live-claim
Board item: CR-GAP-060 (docs-sweep T6), umbrella CR-GAP-053
Repo: <project>-dev/<project>
Outcome: tier1 PASS + tier2 PASS on attempt 4 (.gitreins/history/2026-09-16/17b60c0c)
A docs truth-pass tick had every content sub-claim accepted and still failed the umbrella criterion "the family contains no false claim at HEAD." The cause was a repo-shipped artifact, not the edited prose:
examples/hermes-gateway-demo/TRANSCRIPT-2026-08-20.md:52-58
body session_id slot : ''
NOTE: body session_id slot rendered empty - known template-engine gap
(resolvePath cannot descend into struct-typed EnvelopeMeta ...).
See README "Known gap".
Two defects: (1) stale live claim — the slot was fixed six weeks earlier (CR-GAP-037); (2) dangling pointer — the README section was renamed to ### Session-id body slot (fixed by CR-GAP-037). The judge read the file as current because the README called it "real captured output of the last run ... never hand-written."
Generalizable rule: a transcript is evidence, not a status page — but only if the repo says so in-file, next to the artifact.
(a) Never hand-edit a capture. Leave byte-identical; record the hash:
CAPTURE=examples/hermes-gateway-demo/TRANSCRIPT-2026-08-20.md
git hash-object "$CAPTURE"; sha256sum "$CAPTURE" # record in commit body
(b) Regenerate by running the demo unmodified, requiring rc=0, committing the script's own output under TRANSCRIPT-<YYYY-MM-DD>.md:
cd examples/hermes-gateway-demo
set -o pipefail
./run-demo.sh 2>&1 | tee "/tmp/demo-$(date +%F).log"
[ "${PIPESTATUS[0]}" -eq 0 ] || exit 1
cp "/tmp/demo-$(date +%F).log" "TRANSCRIPT-$(date +%F).md"
grep -q "PASS: template body carried the session_id (body slot populated)" \
"TRANSCRIPT-$(date +%F).md"
(c) Make the README state the rule and name which capture predates which fix:
### Transcripts (dated records, not current state)
`TRANSCRIPT-<YYYY-MM-DD>.md` files are dated records of the run on that date,
generated by `run-demo.sh` (never hand-written). A transcript predating a fix
still shows that date's behaviour.
- `TRANSCRIPT-2026-08-20.md` — predates the session-id body-slot fix (CR-GAP-037).
- `TRANSCRIPT-2026-09-16.md` — post-fix; shows the PASS line.
(d) Re-grep the family — only the framing sentence and the dated capture may match:
git grep -n "known template-engine gap\|body session_id slot : ''"
(e) Secret-scan the new public artifact before committing.
(f) Add a CI guard (scripts/check-captures.sh) asserting every tracked capture is referenced by the README; wire into make docs-check.
The round-2 FAIL came from a false premise the foreman wrote into the brief ("no v1 delivery surface populates thread_id") — true only before CR-FEAT-011 (ae71cbc), disproved against internal/registry/handler.go:47-51,584, internal/webhook/webhook.go:128, docs/openapi.yaml:486-488. Settling probe: boot HEAD on a free scratch port with CR_REQUIRE_AGENT_SIG=false CR_GUARD_ENABLED=false, register an agent (POST /agents requires public_key; any 64-hex works) with schema_template: hermes-http-gateway → logging sink, deliver a top-level thread_id, read the sink body (thr-live-1).
Cap exceeded: Input token budget (9.0M) exceeded (9.0M used) with used == cap means raise the budget, not a merits failure. Size from .gitreins/usage.jsonl (tokens_in 9,019,468); raise the top-level evaluator.max_input_tokens to 12M and max_time to 30m. Next runs completed at 2.73M/1.82M. Decide capped-vs-merits by the log line, not the stage label — both leave tier-2 INCOMPLETE.
On 53cd87f (on 0bbf461 + 7566105; cap chore 901c9f8):
| Check | Command | Result |
|---|---|---|
| Docs | make docs-check |
rc=0 |
| Build | go build ./... && go vet ./... |
rc=0 |
| Unit | go test ./... |
13/13 ok |
| E2E | fresh HEAD, port 18781, holder-pid asserted | 14/14 |
| thread_id probe | server 18795 + sink 18796 | thr-live-1 |
| Remotes | git remote -v |
parity 0 |
| CI | pipeline | green |
4th judge run: tier1 PASS + tier2 PASS, .gitreins/history/2026-09-16/17b60c0c. New TRANSCRIPT-2026-09-16.md shows the PASS line; TRANSCRIPT-2026-08-20.md unchanged (sha256 fe5024572dc8...); stale-phrase grep hits only the framing sentence and the dated capture.
Audit → hash first → never hand-edit → regenerate from the unmodified script → frame the rule in-file → re-grep → secret-scan → CI guard → verify the brief's premises against HEAD → on tier-2 INCOMPLETE, read for Cap exceeded before calling it merits.
Full version saved at ~/SOLUTION-docs-stale-dated-capture.md (includes the complete probe scripts, commit-body template, and guard script). To actually re-verify end-to-end, clone the repo and run the commands in §3–§6 against a HEAD build.
# Evidence - Problem class: docs-stale-dated-capture-read-as-live-claim - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-16T11:52:28.103Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "SYMPTOM: a docs truth-pass tick (<project> CR-GAP-060, docs-sweep T6: examples/ + AGENTS.md + .env.example + Makefile) had every content sub-claim accepted by the tier-2 judge and still failed the umbrella criterion 'the <family> contains no false claim at HEAD', because a DATED CAPTURE tracked in the repo still printed the pre-fix behaviour of a bug fixed six weeks earlier. Concretely examples/hermes-gateway-demo/TRANSCRIPT-2026-08-20.md:52-58 still read `body session_id slot : ''` plus `NOTE: body session_id slot rendered empty - known template-engine gap (resolvePath cannot descend into struct-typed EnvelopeMeta ...). See README \"Known gap\".` - stale text AND a dangling pointer, because the README section it names had been renamed to '### Session-id body slot (fixed by CR-GAP-037)'. The judge's own reasoning: the README described the file as 'real captured output of the last run (generated by the script, never hand-written)', so the capture counts as a CURRENT claim, not history. ROOT CAUSE: (1) prose truth-passes are scoped to the doc files they edit, while a repo-shipped capture is an artifact of the SAME family and keeps asserting whatever was true on its date; (2) the README's description of the capture ('output of the last run') makes the dated artifact read as the latest state; (3) the earlier round of the same tick had explicitly blessed 'leave the dated transcript - it is evidence' and the next judge round overruled it, so the judgement call is unsettled unless the artifact is framed in-file. FIX SHAPE (apply to any docs family that ships transcripts/captures): (a) NEVER hand-edit a capture - editing recorded lines falsifies evidence worse than the drift; leave it byte-identical and record its sha256 in the commit body; (b) generate a FRESH capture by running the demo script unmodified (rc=0) and commit the script's own output under the repo's existing TRANSCRIPT-<YYYY-MM-DD>.md naming - the fixed script's PASS branch then becomes repo-resident proof of the fix; (c) make the README state the RULE, not just point at filenames: a transcript is a dated record of the run on that date, so one predating a fix still shows the pre-fix behaviour, and name which capture predates which fix so the dangling pointer resolves; (d) re-grep the family afterwards - the only acceptable hits for the stale phrase are the framing sentence and the dated capture itself; (e) secret-scan the new public artifact before committing. SECOND, INDEPENDENT LESSON from the same tick, worth folding in: verify the DISPATCH BRIEF's own premises, not only the board card. The round-2 FAIL was caused by a false premise the foreman wrote into the brief ('no v1 delivery surface populates the envelope's thread_id', true only before CR-FEAT-011/ae71cbc); the worker propagated it faithfully into two places and the judge disproved it against internal/registry/handler.go:47-51,584 + internal/webhook/webhook.go:128 + docs/openapi.yaml:486-488. The settling probe for that class of claim: boot a HEAD binary with CR_REQUIRE_AGENT_SIG=false CR_GUARD_ENABLED=false on a proven-free scratch port, register an agent whose webhook uses schema_template hermes-http-gateway pointing at a tiny logging HTTP sink (note POST /agents REQUIRES public_key - any 64-hex value works for a probe), deliver with a top-level thread_id, and read the sink's recorded POST body (body thread_id arrived as 'thr-live-1'). BUDGET NOTE for the same tick: the first judge run died 'Cap exceeded: Input token budget (9.0M) exceeded (9.0M used)' = budget starvation (used == configured cap, fix direction UP), sized from the measured .gitreins/usage.jsonl row tokens_in 9,019,468; after raising the evaluator max_input_tokens to 12M (the token knob existed in ONE place, the top-level evaluator block) and max_time to 30m, the next two runs completed at 2.73M and 1.82M with full merits verdicts. A merits FAIL keeps the tier-2 status word INCOMPLETE - decide capped-vs-merits by the LOG MESSAGE (a 'Cap exceeded' line) and not by the stage label. VERIFICATION: after the fix, the 4th judge run returned tier1 PASS + tier2 PASS on the final commit (verdict .gitreins/history/2026-09-16/17b60c0c), with the fresh TRANSCRIPT-2026-09-16.md showing 'PASS: template body carried the session_id (body slot populated)' and the 2026-08-20 capture unchanged (sha256 fe5024572dc8...).", "environment": "<project> (Go agent-to-agent bus, 13 packages, public repo, JSONL foreman board, gitreins tier-1/tier-2 harness); docs-sweep tick writing prose claims to examples/ and agent-facing files; tier-2 judge with file_scope: full.", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "docs-stale-dated-capture-read-as-live-claim", "provider": "openrouter", "solved_at": "2026-09-16T11:52:28.103Z", "version": ""}The environment has no copy of <project>-dev/<project> (only an empty ~), so I could not re-execute the checks against a live checkout. The solution below is the self-contained, directly-usable runbook reconstructed from the incident record, written to ~/SOLUTION-docs-stale-dated-capture.md.
Problem class: docs-stale-dated-capture-read-as-live-claim
Board item: CR-GAP-060 (docs-sweep T6), umbrella CR-GAP-053
Repo: <project>-dev/<project>
Outcome: tier1 PASS + tier2 PASS on attempt 4 (.gitreins/history/2026-09-16/17b60c0c)
A docs truth-pass tick had every content sub-claim accepted and still failed the umbrella criterion "the family contains no false claim at HEAD." The cause was a repo-shipped artifact, not the edited prose:
examples/hermes-gateway-demo/TRANSCRIPT-2026-08-20.md:52-58
body session_id slot : ''
NOTE: body session_id slot rendered empty - known template-engine gap
(resolvePath cannot descend into struct-typed EnvelopeMeta ...).
See README "Known gap".
Two defects: (1) stale live claim — the slot was fixed six weeks earlier (CR-GAP-037); (2) dangling pointer — the README section was renamed to ### Session-id body slot (fixed by CR-GAP-037). The judge read the file as current because the README called it "real captured output of the last run ... never hand-written."
Generalizable rule: a transcript is evidence, not a status page — but only if the repo says so in-file, next to the artifact.
(a) Never hand-edit a capture. Leave byte-identical; record the hash:
CAPTURE=examples/hermes-gateway-demo/TRANSCRIPT-2026-08-20.md
git hash-object "$CAPTURE"; sha256sum "$CAPTURE" # record in commit body
(b) Regenerate by running the demo unmodified, requiring rc=0, committing the script's own output under TRANSCRIPT-<YYYY-MM-DD>.md:
cd examples/hermes-gateway-demo
set -o pipefail
./run-demo.sh 2>&1 | tee "/tmp/demo-$(date +%F).log"
[ "${PIPESTATUS[0]}" -eq 0 ] || exit 1
cp "/tmp/demo-$(date +%F).log" "TRANSCRIPT-$(date +%F).md"
grep -q "PASS: template body carried the session_id (body slot populated)" \
"TRANSCRIPT-$(date +%F).md"
(c) Make the README state the rule and name which capture predates which fix:
### Transcripts (dated records, not current state)
`TRANSCRIPT-<YYYY-MM-DD>.md` files are dated records of the run on that date,
generated by `run-demo.sh` (never hand-written). A transcript predating a fix
still shows that date's behaviour.
- `TRANSCRIPT-2026-08-20.md` — predates the session-id body-slot fix (CR-GAP-037).
- `TRANSCRIPT-2026-09-16.md` — post-fix; shows the PASS line.
(d) Re-grep the family — only the framing sentence and the dated capture may match:
git grep -n "known template-engine gap\|body session_id slot : ''"
(e) Secret-scan the new public artifact before committing.
(f) Add a CI guard (scripts/check-captures.sh) asserting every tracked capture is referenced by the README; wire into make docs-check.
The round-2 FAIL came from a false premise the foreman wrote into the brief ("no v1 delivery surface populates thread_id") — true only before CR-FEAT-011 (ae71cbc), disproved against internal/registry/handler.go:47-51,584, internal/webhook/webhook.go:128, docs/openapi.yaml:486-488. Settling probe: boot HEAD on a free scratch port with CR_REQUIRE_AGENT_SIG=false CR_GUARD_ENABLED=false, register an agent (POST /agents requires public_key; any 64-hex works) with schema_template: hermes-http-gateway → logging sink, deliver a top-level thread_id, read the sink body (thr-live-1).
Cap exceeded: Input token budget (9.0M) exceeded (9.0M used) with used == cap means raise the budget, not a merits failure. Size from .gitreins/usage.jsonl (tokens_in 9,019,468); raise the top-level evaluator.max_input_tokens to 12M and max_time to 30m. Next runs completed at 2.73M/1.82M. Decide capped-vs-merits by the log line, not the stage label — both leave tier-2 INCOMPLETE.
On 53cd87f (on 0bbf461 + 7566105; cap chore 901c9f8):
| Check | Command | Result |
|---|---|---|
| Docs | make docs-check |
rc=0 |
| Build | go build ./... && go vet ./... |
rc=0 |
| Unit | go test ./... |
13/13 ok |
| E2E | fresh HEAD, port 18781, holder-pid asserted | 14/14 |
| thread_id probe | server 18795 + sink 18796 | thr-live-1 |
| Remotes | git remote -v |
parity 0 |
| CI | pipeline | green |
4th judge run: tier1 PASS + tier2 PASS, .gitreins/history/2026-09-16/17b60c0c. New TRANSCRIPT-2026-09-16.md shows the PASS line; TRANSCRIPT-2026-08-20.md unchanged (sha256 fe5024572dc8...); stale-phrase grep hits only the framing sentence and the dated capture.
Audit → hash first → never hand-edit → regenerate from the unmodified script → frame the rule in-file → re-grep → secret-scan → CI guard → verify the brief's premises against HEAD → on tier-2 INCOMPLETE, read for Cap exceeded before calling it merits.
Full version saved at ~/SOLUTION-docs-stale-dated-capture.md (includes the complete probe scripts, commit-body template, and guard script). To actually re-verify end-to-end, clone the repo and run the commands in §3–§6 against a HEAD build.
# Evidence - Problem class: docs-stale-dated-capture-read-as-live-claim - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-16T11:52:28.103Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "SYMPTOM: a docs truth-pass tick (<project> CR-GAP-060, docs-sweep T6: examples/ + AGENTS.md + .env.example + Makefile) had every content sub-claim accepted by the tier-2 judge and still failed the umbrella criterion 'the <family> contains no false claim at HEAD', because a DATED CAPTURE tracked in the repo still printed the pre-fix behaviour of a bug fixed six weeks earlier. Concretely examples/hermes-gateway-demo/TRANSCRIPT-2026-08-20.md:52-58 still read `body session_id slot : ''` plus `NOTE: body session_id slot rendered empty - known template-engine gap (resolvePath cannot descend into struct-typed EnvelopeMeta ...). See README \"Known gap\".` - stale text AND a dangling pointer, because the README section it names had been renamed to '### Session-id body slot (fixed by CR-GAP-037)'. The judge's own reasoning: the README described the file as 'real captured output of the last run (generated by the script, never hand-written)', so the capture counts as a CURRENT claim, not history. ROOT CAUSE: (1) prose truth-passes are scoped to the doc files they edit, while a repo-shipped capture is an artifact of the SAME family and keeps asserting whatever was true on its date; (2) the README's description of the capture ('output of the last run') makes the dated artifact read as the latest state; (3) the earlier round of the same tick had explicitly blessed 'leave the dated transcript - it is evidence' and the next judge round overruled it, so the judgement call is unsettled unless the artifact is framed in-file. FIX SHAPE (apply to any docs family that ships transcripts/captures): (a) NEVER hand-edit a capture - editing recorded lines falsifies evidence worse than the drift; leave it byte-identical and record its sha256 in the commit body; (b) generate a FRESH capture by running the demo script unmodified (rc=0) and commit the script's own output under the repo's existing TRANSCRIPT-<YYYY-MM-DD>.md naming - the fixed script's PASS branch then becomes repo-resident proof of the fix; (c) make the README state the RULE, not just point at filenames: a transcript is a dated record of the run on that date, so one predating a fix still shows the pre-fix behaviour, and name which capture predates which fix so the dangling pointer resolves; (d) re-grep the family afterwards - the only acceptable hits for the stale phrase are the framing sentence and the dated capture itself; (e) secret-scan the new public artifact before committing. SECOND, INDEPENDENT LESSON from the same tick, worth folding in: verify the DISPATCH BRIEF's own premises, not only the board card. The round-2 FAIL was caused by a false premise the foreman wrote into the brief ('no v1 delivery surface populates the envelope's thread_id', true only before CR-FEAT-011/ae71cbc); the worker propagated it faithfully into two places and the judge disproved it against internal/registry/handler.go:47-51,584 + internal/webhook/webhook.go:128 + docs/openapi.yaml:486-488. The settling probe for that class of claim: boot a HEAD binary with CR_REQUIRE_AGENT_SIG=false CR_GUARD_ENABLED=false on a proven-free scratch port, register an agent whose webhook uses schema_template hermes-http-gateway pointing at a tiny logging HTTP sink (note POST /agents REQUIRES public_key - any 64-hex value works for a probe), deliver with a top-level thread_id, and read the sink's recorded POST body (body thread_id arrived as 'thr-live-1'). BUDGET NOTE for the same tick: the first judge run died 'Cap exceeded: Input token budget (9.0M) exceeded (9.0M used)' = budget starvation (used == configured cap, fix direction UP), sized from the measured .gitreins/usage.jsonl row tokens_in 9,019,468; after raising the evaluator max_input_tokens to 12M (the token knob existed in ONE place, the top-level evaluator block) and max_time to 30m, the next two runs completed at 2.73M and 1.82M with full merits verdicts. A merits FAIL keeps the tier-2 status word INCOMPLETE - decide capped-vs-merits by the LOG MESSAGE (a 'Cap exceeded' line) and not by the stage label. VERIFICATION: after the fix, the 4th judge run returned tier1 PASS + tier2 PASS on the final commit (verdict .gitreins/history/2026-09-16/17b60c0c), with the fresh TRANSCRIPT-2026-09-16.md showing 'PASS: template body carried the session_id (body slot populated)' and the 2026-08-20 capture unchanged (sha256 fe5024572dc8...).", "environment": "<project> (Go agent-to-agent bus, 13 packages, public repo, JSONL foreman board, gitreins tier-1/tier-2 harness); docs-sweep tick writing prose claims to examples/ and agent-facing files; tier-2 judge with file_scope: full.", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "docs-stale-dated-capture-read-as-live-claim", "provider": "openrouter", "solved_at": "2026-09-16T11:52:28.103Z", "version": ""}