docs-battery-count-drift
Root cause. shim/test_battery.py is the count owner (44 tests after the suite grew), but four live docs still carried the pre-growth count 43 — root AGENTS.md (prose + compliance table), h3/AGENTS.md, README.md (5 spots: prose, pass-rate, compliance line, notes), and skills/SKILL.md (43/43). H3-GAP-016's PASS-grep only scanned the named files and missed README/SKILL.md (the #249 partial-fix lesson). The fix is a whole-repo sweep with three-way classification so era-correct artifacts and sibling-owned files are never blind-mutated.
Fix (fix_battery_count_drift.py, run with --root <repo>):
COUNT_RE = re.compile(r"(?<![\d.])\b43\b(?![\d.])") # bare 43 only: no 143 / 0.43
AUTH_RE = re.compile(r"^def (test_\w+)\s*\(", re.MULTILINE)
PROTECTED_DIRS = {"snapshots", "prd", "shim"}
def authoritative_count(test_file: Path) -> int: # count owned by the shim suite
return len(AUTH_RE.findall(test_file.read_text(encoding="utf-8")))
def classify(path: Path) -> str:
parts = {p.lower() for p in path.parts}
if parts & {"snapshots", "prd"}: return "snapshot" # era-correct by design
if "shim" in parts and path.name == "AGENTS.md": return "sibling" # cross-repo
return "live"
def sweep(root: Path, auth: int) -> Report:
for path in sorted(root.rglob("*")):
if not path.is_file() or path.suffix not in {".md", ".html", ".txt"}: continue
if path.name == "test_battery.py": continue # count owner, skip
text = path.read_text(encoding="utf-8")
if not COUNT_RE.findall(text): continue
kind = classify(path)
if kind == "sibling": # Rule 3: flag, never edit
print(DUCKBRAIN_FLAG, file=sys.stderr); report.flagged.append(str(path)); continue
if kind == "snapshot": # Rule 2: annotate, don't replace
path.write_text(text + "\n" + ANNOTATION.format(stamp=stamp, auth=auth) + "\n")
report.annotated.append(str(path)); continue
fixed = COUNT_RE.sub(str(auth), text) # Rule 1: live doc -> replace
if fixed != text:
path.write_text(fixed); report.changed.append(f"{path} ({len(hits)} spot(s))")
Key decisions, each mapped to the filing's lessons:
| Rule | Implementation |
|---|---|
| #249 partial-fix lesson | Sweep rglob("*") across the whole tree, not just the named files — README.md and skills/SKILL.md are caught by construction. |
| Snapshot annotation | snapshots/status-report.html (Generated: 2026-07-24) and frozen prd/ deliverables keep 43 but get the gate note: <!-- ANNOTATED (H3-GAP-016): era-correct snapshot of the 43-scenario suite as of 2026-07-24; current live count is 44 ... -- do not blind-replace (see tick #272) --> (same pattern as the P3-10 gate note from H3-GAP-013/#258, applied at every site to stop recurrence). |
| Sibling ownership | shim/AGENTS.md is never edited; a DUCKBRAIN-NOTE: is emitted to stderr flagging it for the shim team's tick. |
| False positives | Lookbehind/lookahead (?<![\d.]) ... (?![\d.]) rejects 143, 0.43, 43.0. |
| Idempotent + CI-friendly | Exit 2 = drift fixed, 0 = clean no-op (annotation already present → no duplicate). --test-file resolves relative to --root (bug caught by the harness). |
Fixture reproduced the exact filing state (root `AGENTS.md` 1 stale spot + prose/table, `README.md` 5, `SKILL.md` `43/43`, `h3/AGENTS.md` 1; snapshot `Generated: 2026-07-24`; `shim/test_battery.py` = 44 tests). Ran `test_fix.py` against a fresh copy each run — **22/22 pass, exit 0**: - **Drift reproduced:** authoritative count = 44; 10 stale `43` refs across the 4 live docs pre-fix. - **Post-fix live docs:** 0 stale `43` anywhere; table now `| battery count | 44 |`; `README` compliance line = 44; `SKILL.md` = `44/44`; `h3/AGENTS.md` clean. - **Protected files:** snapshot still `43 battery scenarios` with the `Generated: 2026-07-24` header preserved and one H3-GAP-016 annotation appended; PRD keeps frozen `43/43` + annotation; `shim/AGENTS.md` byte-for-byte untouched (still `43 scenarios`, no `44`), flagged via `DUCKBRAIN-NOTE` on stderr. - **Edge cases:** no `143`/`0.43` corruption; second run exits `0` with `changed (live docs): 0`; re-run appends no duplicate annotation; count is derived from the suite file, so the next suite growth (45) is caught by the same sweep instead of a hand-edited constant. - **Regression gap closed (the #249 lesson):** README.md + SKILL.md stale spots were fixed by this sweep — exactly what the original PASS-grep scope missed.
{"model": "deepseek-v4-flash", "problem_class": "docs-battery-count-drift", "result": "passed", "tests": 22}