docs-pm-gap-injection-verification

Answer

Root cause. The stand-in PM bypasses the foreman gates by writing GAP tasks straight into tasks.jsonl as spaced (pretty-printed) JSON, leaving them uncommitted and absent from board.db. That single injection breaks the pipeline in three concrete ways: (1) every line-based JSONL parser in the append scripts throws JSONDecodeError on multi-line rows; (2) dispatch against a board row that doesn't exist creates silent drift; (3) an uncommitted jsonl means the next run's state is not reproducible. The fix is an ingestion gate that makes the foreman flow the only legal path, plus hardened append scripts that refuse to operate on unverified state.

Fix 1 — Robust stream parser + one-time normalizer (foreman/jsonl_utils.py). Parse the JSONL stream with raw_decode instead of per-line json.loads, tolerate spaced rows, and compact them (idempotent → churn happens exactly once):

def iter_rows(text: str):
    dec = json.JSONDecoder()
    i, n = 0, len(text)
    while i < n:
        while i < n and text[i] in " \t\r\n":
            i += 1
        if i >= n: break
        if text[i] == "#":                       # comment lines allowed
            nl = text.find("\n", i); i = n if nl < 0 else nl + 1; continue
        try:
            obj, end = dec.raw_decode(text, i)   # reads ONE object wherever it starts
        except json.JSONDecodeError as exc:
            raise ValueError(f"malformed JSON near byte offset {i}: {exc.msg}") from exc
        yield Row(obj=obj, start=i, end=end, n_lines=text[i:end].count("\n") + 1, raw=text[i:end])
        i = end

def normalize_file(path, dry_run=False) -> int:   # returns churn count
    text = path.read_text(encoding="utf-8")
    new_text, churn = normalize_text(text)        # spaced rows -> compact, one per line
    if churn and not dry_run and new_text != text:
        path.write_text(new_text, encoding="utf-8")
    return churn

Fix 2 — Verify every cited fact pre-dispatch (foreman/verify_cited_facts.py). Doc facts grep the exact phrase in the cited doc; code consts grep NAME = VALUE in the cited source. This is the same grep the doc-only judge re-runs at completion (the judge exception):

if fact.get("type") == "doc":
    ok = phrase in (root / loc).read_text()
if fact.get("type") == "code":
    ok = bool(re.compile(r"^\s*%s\s*=\s*%s\s*$" % (const, value), re.M).search(text))

Fix 3 — Heal board.db before dispatch (foreman/sync_tasks_jsonl_to_db.py): insert any tasks.jsonl id missing from the board (status='injected'); idempotent, preserves dispatched/completed state.

Fix 4 — Guarded append scripts (append_board_task_dispatched.py / append_board_task_completed.py): dispatch takes one task id per call, enforces a shared worker pid, and REFUSES tasks absent from the board (run sync first), already completed, or not yet dispatched. Completion normalizes spaced rows (one-time churn) and re-runs the AC greps — for doc-only tasks the judge is the foreman re-running those greps, so a failed grep blocks completion with rc=2 and state is preserved.

Fix 5 — The orchestrated flow (foreman/foreman_gap_flow.py), gating each step:

1. GATE  rows parse + unique ids (spaced JSON tolerated)
2. GATE  tasks.jsonl committed (git status)        # blocks the uncommitted injection
3. NORM  compact spaced rows  -> one-time churn
4. GATE  verify_cited_facts.py (docs + code consts) # nothing dispatches until facts verify
5. SYNC  sync_tasks_jsonl_to_db.py heals board.db
6. DISP  append_board_task_dispatched.py per task, shared --pid
7. WORK  one worker session for all doc tasks, one git commit per task
8. COMP  append_board_task_completed.py per task (doc-only judge = foreman AC grep re-run)