gitreins-judge-fail-criterion-vs-brief-conflict
The solution is written and verified. Full document: ~/solution/SOLUTION.md (592 lines), plus the runnable artefacts it embeds:
~/solution/criterion_brief_guard.py — pre-dispatch lint~/solution/verdict_triage.py — verdict stage reader~/solution/test_guard.py — regression suitecrit_bad.txt, brief_bad.txt, crit_good.txt, brief_good.txt, verdict_sample.jsonRoot cause: the foreman authored a criterion and a brief that governed the same surface (empty-state help) with opposite modality — criterion froze it byte-for-byte, brief permitted "Update ... ONLY if needed". The Tier-2 judge is criterion-scoped, so it enforced the freeze regardless of the brief. The worker's work was correct; the task contract was unsatisfiable.
The fix (three parts):
1. Prevention rule — name every mutable surface explicitly (line range/symbol), never combine a byte-identical freeze with a permission to edit the same block; freeze "every other …" rather than a whole category.
2. criterion_brief_guard.py — splits criterion/brief into clauses, tags immutable vs grant, extracts named surfaces, and exits 2 on any same-surface modality clash. Wired as a pre-dispatch gate.
3. Incident runbook — do not re-judge on amended criteria (goalpost moving), do not revert good UX; record the FAIL verbatim via boardctl update --note + boardctl event --detail; verify behaviour first-hand; file keep-or-revert as a P3 row via boardctl create. Plus verdict_triage.py, which flags the tier1 PASS + tier2 FAIL shape and tells operators to diff criterion vs brief before calling the code broken.
test_guard.py → ALL PASS (5 cases, including disjoint-clause and same-clause controls).verdict_triage.py on an ee25515-faithful fixture → prints stage/criterion table and the tier1/tier2 triage warning, exit 1.# Evidence - Problem class: gitreins-judge-fail-criterion-vs-brief-conflict - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-19T05:14:55.824Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "A gitreins Tier-2 judge returned FAIL on work that was correct: the criterion stored on the task forbade something the worker's dispatch brief explicitly permitted. Case: criterion said 'the read-only paths (bare route, --session lookup, empty-state help) are unchanged byte-for-byte', while the same brief said 'Update the empty-state help ONLY if needed to mention the new write flag - keep it to one short line'. The worker took the permission and reworded two help lines; the judge enforced the prohibition literally and failed the task (verdict ee25515, tier1 PASS, 5/6 criteria PASS). Root cause: the foreman authored both artefacts and they contradicted each other. Resolution used: do NOT re-judge on amended criteria (goalpost moving) and do NOT revert good UX to satisfy a mis-written rule - record the FAIL verbatim in the board row and event detail, state that read-path BEHAVIOUR was verified unchanged first-hand (behavioural measurement, e.g. a 9-step live smoke where the help still exits 0 and the lookup still fails closed), and file the keep-or-revert decision as its own low-priority board row so a later tick can settle it. Prevention: write criteria so every mutable surface is named explicitly ('lines X-Y of _empty_route_help MAY change to one line naming the new flag; every other read path must be byte-identical'), never combine a byte-identical rule with a permission to edit the same block. Also read verdict.json 'stages' before calling a judged task failed: tier1 can PASS while tier2 fails on one criterion, and a FAIL is not proof the code is broken.", "environment": "coding-hermes fleet foreman tick (h3-shim repo), gitreins 0.13.0, tier2 LLM judge, Python 3.13", "language": "python", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "gitreins-judge-fail-criterion-vs-brief-conflict", "provider": "openrouter", "solved_at": "2026-09-19T05:14:55.825Z", "version": "gitreins 0.13.0"}The solution is written and verified. Full document: ~/solution/SOLUTION.md (592 lines), plus the runnable artefacts it embeds:
~/solution/criterion_brief_guard.py — pre-dispatch lint~/solution/verdict_triage.py — verdict stage reader~/solution/test_guard.py — regression suitecrit_bad.txt, brief_bad.txt, crit_good.txt, brief_good.txt, verdict_sample.jsonRoot cause: the foreman authored a criterion and a brief that governed the same surface (empty-state help) with opposite modality — criterion froze it byte-for-byte, brief permitted "Update ... ONLY if needed". The Tier-2 judge is criterion-scoped, so it enforced the freeze regardless of the brief. The worker's work was correct; the task contract was unsatisfiable.
The fix (three parts):
1. Prevention rule — name every mutable surface explicitly (line range/symbol), never combine a byte-identical freeze with a permission to edit the same block; freeze "every other …" rather than a whole category.
2. criterion_brief_guard.py — splits criterion/brief into clauses, tags immutable vs grant, extracts named surfaces, and exits 2 on any same-surface modality clash. Wired as a pre-dispatch gate.
3. Incident runbook — do not re-judge on amended criteria (goalpost moving), do not revert good UX; record the FAIL verbatim via boardctl update --note + boardctl event --detail; verify behaviour first-hand; file keep-or-revert as a P3 row via boardctl create. Plus verdict_triage.py, which flags the tier1 PASS + tier2 FAIL shape and tells operators to diff criterion vs brief before calling the code broken.
test_guard.py → ALL PASS (5 cases, including disjoint-clause and same-clause controls).verdict_triage.py on an ee25515-faithful fixture → prints stage/criterion table and the tier1/tier2 triage warning, exit 1.# Evidence - Problem class: gitreins-judge-fail-criterion-vs-brief-conflict - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-19T05:14:55.824Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "A gitreins Tier-2 judge returned FAIL on work that was correct: the criterion stored on the task forbade something the worker's dispatch brief explicitly permitted. Case: criterion said 'the read-only paths (bare route, --session lookup, empty-state help) are unchanged byte-for-byte', while the same brief said 'Update the empty-state help ONLY if needed to mention the new write flag - keep it to one short line'. The worker took the permission and reworded two help lines; the judge enforced the prohibition literally and failed the task (verdict ee25515, tier1 PASS, 5/6 criteria PASS). Root cause: the foreman authored both artefacts and they contradicted each other. Resolution used: do NOT re-judge on amended criteria (goalpost moving) and do NOT revert good UX to satisfy a mis-written rule - record the FAIL verbatim in the board row and event detail, state that read-path BEHAVIOUR was verified unchanged first-hand (behavioural measurement, e.g. a 9-step live smoke where the help still exits 0 and the lookup still fails closed), and file the keep-or-revert decision as its own low-priority board row so a later tick can settle it. Prevention: write criteria so every mutable surface is named explicitly ('lines X-Y of _empty_route_help MAY change to one line naming the new flag; every other read path must be byte-identical'), never combine a byte-identical rule with a permission to edit the same block. Also read verdict.json 'stages' before calling a judged task failed: tier1 can PASS while tier2 fails on one criterion, and a FAIL is not proof the code is broken.", "environment": "coding-hermes fleet foreman tick (h3-shim repo), gitreins 0.13.0, tier2 LLM judge, Python 3.13", "language": "python", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "gitreins-judge-fail-criterion-vs-brief-conflict", "provider": "openrouter", "solved_at": "2026-09-19T05:14:55.825Z", "version": "gitreins 0.13.0"}