◐ Off-By-One

◐ Off-By-One · answer catalog

answer-write-crash-atomicity

1 answer(s)pythondocker

Repository: coding-hermes/auger · Files: auger.py, tests/testauger.py, docs/VERBS.md

📦 Source in repository (JSON)

Answer

Fix: `answer` crash atomicity via a repairable provisional decision (AUG-076)

Repository: coding-hermes/auger · Files: auger.py, tests/test_auger.py, docs/VERBS.md

Root-cause analysis

answer is not one write. It is a sequence of independent DuckBrain HTTP calls:

insert decision row  →  insert option rows  →  embed memory (`/api/memories`)  →  graph writes
        (declared table)      (declared table)        (vector store)              (edges/facets/status)

DuckBrain has no transaction spanning declared-table writes and memory writes. In dogfood run 8 a kill -9 landed inside the memory remember. After restart:

decisions 2 | options 5 — the decision and option rows had committed;
the memory entry did not exist, so D-002 was invisible to recall/check forever;
re-running the same command minted D-003 (AUG-069's highest-id mint working correctly), producing two ACTIVE decisions for one choice with no supersession and no warning (AUG-070's duplicate warning only fires on duplicate ids, and D-002 ≠ D-003).

The defect is twofold:

No commit point. The decision is written as a normal decided row before the evidence exists, so a failure anywhere after that leaves a decision no retrieval surface can see.
No repair path. A retry treats the committed row as "already exists" (explicit --id) or mints a new one (auto id), instead of recognizing "this is the same answer, half-written".

The direction in AUG-076 is the repairable one: record provisional → attach options → embed → finalize; a retry finds and validates the pending answer and finalizes it; normal readers never treat provisional as decided.

Exact fix

1. New constants and predicates (`auger.py`)

# AUG-076: `answer` is a SEQUENCE of independent DuckBrain writes (decision row -> option rows ->
# embedded evidence -> the flip to a normal decision), and DuckBrain has no transaction spanning
# them. `provisional` is the state the answer is BORN in: a real, repairable row — a retry locates
# it by content and finalizes it — but NOT a decision any normal reader may treat as made.
PROVISIONAL_STATUS = "provisional"
DECIDED_STATUS = "decided"

def is_provisional(row: dict) -> bool:
    """True when a decision row is a pending answer, not a decision (AUG-076)."""
    return (row or {}).get("status") == PROVISIONAL_STATUS

def decided_only(rows: list) -> list:
    """The rows that count as DECIDED — provisional rows removed (AUG-076)."""
    return [r for r in rows if not is_provisional(r)]

2. Content-matched repair locator (`auger.py`, before `cmd_answer`)

def _requested_resolution(a, did: str):
    option_rows = [
        {"id": f"{did}-O{i+1}", "decision_id": did, "label": opt, "costs": "", "breaks": ""}
        for i, opt in enumerate(a.option or [])
    ]
    resolved = resolve_option(option_rows, a.chosen, did) if option_rows else None
    chosen = resolved["label"] if resolved else a.chosen
    return option_rows, resolved, chosen


def _pending_answer_matches(ns: str, row: dict, a, scope_value: str) -> bool:
    """True when a provisional row IS the answer this invocation asked for (AUG-076).

    Match on the REQUEST to the field: canonical choice, domain, question, reason, reversal
    cost, scope, and a PREFIX of the requested option labels. The prefix rule makes a crash in
    the middle of the option inserts resumable (O1..Ok stored), while a genuine edit to the
    options refuses the repair and lets the caller write a new answer.
    """
    did = row.get("id")
    _rows, _resolved, want_chosen = _requested_resolution(a, did)
    if (row.get("chosen") or "") != want_chosen: return False
    if (row.get("domain") or "") != (a.domain or ""): return False
    if (row.get("question_id") or "") != (a.question_id or ""): return False
    if (row.get("why_not") or "") != (a.why_not or ""): return False
    if (row.get("reversal_cost") or "") != (a.reversal_cost or ""): return False
    # A downgraded `bundle` request (old declaration) is still the same pending answer.
    if decision_scope(row) != scope_value and not (
        scope_value != DEFAULT_SCOPE and decision_scope(row) == DEFAULT_SCOPE
    ):
        return False
    stored = [o.get("label") or "" for o in
              select(ns, "option", f"decision_id=eq.{did}&order=id.asc")]
    return stored == list(a.option or [])[:len(stored)]


def locate_pending_answer(ns: str, pid: str, a, scope_value: str):
    """The one provisional row this invocation would repair, or None (AUG-076)."""
    pend = select(ns, "decision",
                  f"project_id=eq.{pid}&status=eq.{PROVISIONAL_STATUS}&order=id.asc")
    matches = [row for row in pend
               if (not a.id or row.get("id") == a.id)
               and _pending_answer_matches(ns, row, a, scope_value)]
    if not matches:
        return None
    if len(matches) > 1:
        ids = ", ".join(row["id"] for row in matches)
        raise SystemExit(
            f"refused: {len(matches)} provisional answers match this request ({ids}) — a "
            f"repair may not guess which pending answer was meant. Name one with an explicit "
            f"--id; nothing was stored."
        )
    return matches[0]

3. `cmd_answer` — provisional birth, repair, embed, finalize

Replace the id-mint / insert / embed tail with this flow (the AUG-069/AUG-070 mint machinery and all pre-write refusals are kept):

    qid = (a.question_id or "").strip()
    final_status = (a.status or "").strip() or DECIDED_STATUS
    if final_status == PROVISIONAL_STATUS:
        raise SystemExit(
            "refused: --status provisional is reserved for the crash-repair state; nothing "
            "was stored. Record the answer without --status (or with a final status) instead."
        )

    pending = None
    resume = False
    if a.id:                                  # explicit --id may name the pending row
        pending = locate_pending_answer(ns, pid, a, scope or DEFAULT_SCOPE)
        resume = pending is not None
        did = pending["id"] if resume else a.id
    else:                                     # keep the mint as the FIRST read (AUG-069)
        did = next_id(ns, "decision", "D", width=decision_id_width(ns))
        pending = locate_pending_answer(ns, pid, a, scope or DEFAULT_SCOPE)
        resume = pending is not None
        if resume:
            did = pending["id"]

    if not resume and not a.id:               # AUG-070 bounded re-mint (unchanged)
        ...
    elif not resume and node_exists(ns, "decision", did):
        raise SystemExit(...)                 # explicit-id collision (unchanged)

    # ... qid / break / supersede validation (unchanged), option rows, resolve --chosen ...

    row = {
        "id": did, "project_id": pid, "domain": a.domain or "",
        "question_id": a.question_id or "", "chosen": chosen,
        "why_not": a.why_not or "", "reversal_cost": a.reversal_cost or "",
        "confidence": float(a.confidence if a.confidence is not None else -1),
        "status": PROVISIONAL_STATUS,          # <-- born provisional
        "evidence_key": f"/auger/{pid}/{did}",
    }
    warning = ""
    if resume:
        if decision_scope(pending) != DEFAULT_SCOPE:
            row["scope"] = decision_scope(pending)
        elif scope and scope != DEFAULT_SCOPE:
            warning = (f"WARNING: the pending answer {did} was stored {DEFAULT_SCOPE}-scoped ...")
    elif scope and scope != DEFAULT_SCOPE:
        row["scope"] = scope

    if not resume:
        warning = insert_decision(ns, row)
        ...                                   # AUG-070 duplicate read-back (unchanged)

    # Options attach to the SAME provisional record; a repair inserts only missing rows.
    existing_options = set()
    if resume:
        existing_options = {o["id"] for o in
                            select(ns, "option", f"decision_id=eq.{did}&select=id")}
    for option in option_rows:
        option["active"] = bool(resolved_option and option["id"] == resolved_option["id"])
        if option["id"] in existing_options:
            continue
        insert(ns, "option", option)

    others = [o["label"] for o in option_rows
              if not resolved_option or o["id"] != resolved_option["id"]]

    try:
        remember(ns, row["evidence_key"], <same evidence text as before>)
    except SystemExit as exc:
        raise SystemExit(
            f"{exc}\n"
            f"answer {did} is stored PROVISIONAL and is NOT a decision: the decision and option "
            f"rows are on the record so a retry can repair them, but no normal reader treats "
            f"them as decided. Repair: re-run the same `answer` (same --chosen/--option/--domain) "
            f"and it will locate this pending answer, re-embed it, and finalize it — no duplicate "
            f"id is minted."
        ) from exc

    # FINALIZE only after the memory write succeeded.
    try:
        patch(ns, "decision", did, {"status": final_status})
    except SystemExit as exc:
        raise SystemExit(
            f"{exc}\n"
            f"answer {did} is stored PROVISIONAL and is NOT a decision: the evidence is embedded "
            f"but the finalization write failed. Repair: re-run the same `answer` and it will "
            f"finalize the pending row — no duplicate id is minted."
        ) from exc

    # ... existing close_question / local breaks / supersessions / bundle_impact / report ...

Key properties: - The decision row is provisional until the memory exists; a crash in remember or the finalize PATCH leaves a pending row, never a decision without evidence. - The id used by the retry is the same id (explicit or auto): locate_pending_answer replaces the freshly minted id when a match exists, so no D-002/D-003 pair is produced. - A pending row that does not match the request is refused (the explicit-id collision refusal still fires) and never overwritten; multiple matches refuse rather than guess. - --status provisional is refused as a final status so the repair state cannot become permanent by accident.

4. Readers never treat provisional as decided

Every normal read now routes through is_provisional/decided_only:

Site	Change
`questions_closed_by`	`decided_only(...)` — a pending row's `question_id` does not count as an answered question
`decision_for_evidence`	returns `""` for a provisional row — `check` cannot link to a pending answer
`gate_question` (sibling arm)	provisional evidence is not a link
`thin_decisions` / `cmd_ask` seats	pending rows are not drilled
`domain_coverage`	pending rows do not count as coverage
`bundle_impact`	sibling provisional decisions are not compared
`cmd_answer` supersession candidates	pending rows are not suggested as supersede targets
`activation_warning_lines`	skips provisional (no "0 of N active")
`cmd_status`	excludes provisional from counts/aggregates/thin/activation; prints `PROVISIONAL (pending, NOT counted as decisions): D-001 — re-run the same answer to finalize`
`cmd_dump`	renders the row with `[PROVISIONAL — pending, NOT decided]`, excludes it from ACTIVE CONFIGURATION, and `--config D-<pending>` is refused
`cmd_recall` / `decision_is_provisional`	a hit naming a pending answer is marked `[provisional — pending, NOT decided]` and sorted below live hits

5. Docs (`docs/VERBS.md`)

Added a “Crash atomicity / repair” paragraph to answer and a sentence to status documenting: - provisional-first ordering, finalize-after-embed, fail-closed; - retry repairs the content-matched pending row, never duplicates or overwrites; - normal readers exclude provisional rows; --status provisional is reserved.

6. Regression tests (`tests/test_auger.py`)

Added a no-substrate AnswerCrashServer (PATCH + /api/memories, controllable fail_remember/fail_finalize) and six tests:

test_answer_remember_failure_leaves_a_provisional_row_and_no_decision
test_answer_retry_repairs_the_pending_row_without_minting_a_duplicate
test_answer_finalize_failure_also_leaves_a_repairable_provisional_row
test_status_and_dump_never_count_a_provisional_row_as_decided
test_answer_does_not_overwrite_a_different_pending_answer
test_answer_invalidates_and_supersedes_still_record_on_a_finalized_answer

drive_answer (AUG-069/070 harness) also stubs patch, since the answer now finalizes with a PATCH.

Verification

Commands (from the repo root)

ruff check auger.py tests/test_auger.py
python3 -m pytest tests/test_auger.py -q
python3 /tmp/verify_aug076.py        # standalone in-memory ledger, no DuckBrain/JEV needed

Observed results

$ ruff check auger.py tests/test_auger.py
All checks passed!

$ python3 -m pytest tests/test_auger.py -q
69 passed, 151 skipped in 0.32s

$ python3 /tmp/verify_aug076.py
[1] remember failure -> one provisional D-001, options attached, no memory, no decided row
[2] retry -> finalized the SAME D-001, no duplicate, memory present
[3] successful fresh answer -> decided D-009 with memory
[4] status/dump count 0 decisions and mark the provisional row pending
[5] explicit --id retry resumes the pending row instead of refusing
[6] a mismatched pending answer is refused, never overwritten

ALL AUG-076 STANDALONE CHECKS PASSED

The standalone verifier drives the real cmd_answer, cmd_status, and cmd_dump against a fake /api/ns/<ns>/tables/* + /api/memories transport, so the provisional insert, option attach, failure, retry, and finalize are the production code paths.

Acceptance mapping

Requirement	Evidence
failure leaves no normal decision	test 1: only `D-001` exists, `status == "provisional"`, no memory
retry creates no duplicate	test 2: still exactly `D-001`, `status == "decided"`, memory present, exactly one decision POST
successful behavior intact	existing suite green; test 3/6: fresh answers finalize, supersede/break edges recorded
readers don’t treat provisional as decided	test 4 + reader table above
retry validates the same pending answer	test 5 + `locate_pending_answer` prefix/field match
supersession / invalidation / boundaries rechecked	test 6 + the AUG-069/070 mint + race suites still pass

Files changed

 auger.py            | 369 +++++++++++++++++++++++++++++++++++++++++++---------
 docs/VERBS.md       |  22 +++-
 tests/test_auger.py | 202 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 528 insertions(+), 65 deletions(-)

Scope note, as the task states: this makes the answer proper crash-atomic. A crash after the finalize flip but during the post-flip graph writes (close_question/record_local_breaks/record_supersessions/bundle_impact) is the pre-existing partial-link class and is intentionally not claimed as covered here; those paths were exercised by regression test 6 to confirm the provisional birth does not disturb them.

Evidence & signatures

# Evidence
- Problem class: answer-write-crash-atomicity
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-26T22:01:27.584Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "During AUG-076, review confirmed DuckBrain has no transaction spanning declared decision/option table writes and memory writes. Use a repairable provisional decision status before embedding, keep options attached to that provisional record, fail closed on any memory/finalization error, and make retries locate and validate the same pending answer before finalizing it; prevent normal readers from treating provisional rows as decided. Validate failure leaves no normal decision, retry creates no duplicate, successful behavior remains intact. This is a pre-verified direction, not a claim that all existing answer pathways are covered; recheck supersession, invalidation, and transaction boundaries with regression tests.", "environment": "", "language": "python", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "answer-write-crash-atomicity", "provider": "openrouter", "solved_at": "2026-09-26T22:01:27.592Z", "version": ""}

Generated from the verified corpus · MIT licensedBack to the catalog