python-audit-idle-maintenance

Answer 1

Tick #40 is an idle NEVER-DONE audit: there is no active project in this environment, only fixture-level work that never lands. The correct fix is a defensive no-op audit — verify every gate, honor the 900 cooldown pin by issuing no PUT, and refuse to fabricate a DuckDB board or a git commit that the environment cannot legitimately produce. The pattern tells us exactly where the trap is: "900 pinned - no PUT" and "board-only commit with git add -f (gitignored)". If the artifacts (fleet.toml, board.db, .git) do not exist, the honest outcome is "passed with zero mutations", never "invent files to make the workflow look busy".

Guard script embodying the fix:

# audit.py -- idle NEVER-DONE maintenance gate (tick #40)
import os, subprocess, sys

steps = []

def check(name, ok, detail):
    steps.append((name, ok, detail))
    print(f"[{'PASS' if ok else 'SKIP'}] {name}: {detail}")

# 1. Gates fresh: no suite -> vacuous pass; real suite -> must be green
r = subprocess.run(["pytest", "-q"], capture_output=True, text=True)
check("pytest", r.returncode in (0, 5),  # 5 == "no tests collected"
      f"exit={r.returncode}: {r.stdout.strip().splitlines()[-1] if r.stdout.strip() else 'no tests'}")

r = subprocess.run(["ruff", "check", "."], capture_output=True, text=True)
check("ruff", r.returncode == 0, r.stdout.strip() or "all checks passed")

# GitReins guard: if the binary is missing it cannot gate -> skip, never fake it
gr = shutil.which("gitreins")
check("gitreins", gr is not None, f"binary {'found' if gr else 'ABSENT -> guard non-blocking'}")

# 2. fleet.toml pin before any cooldown PUT (900 pinned -> NO PUT)
pin = None
if os.path.exists("fleet.toml"):
    pin = next((l.split("=")[1].strip() for l in open("fleet.toml")
                if l.strip().startswith("pin")), None)
no_put = pin == "900" or pin is None          # absent file == safest default
check("fleet.pin", no_put, f"pin={pin or '900 (absent default)'}; cooldown PUT issued: {not no_put}")

# 3. DuckDB board update: only when board.db exists and duckdb is importable
has_db, has_duck = os.path.exists("board.db"), importlib.util.find_spec("duckdb") is not None
if has_db and has_duck:
    subprocess.run(["python", "scripts/export_board.py"], check=True)   # board.db -> parquet
    check("board.export", True, "board.db -> parquet exported")
else:
    check("board.export", False, f"board.db={has_db}, duckdb={has_duck} -> no export, nothing fabricated")

# 4. Board-only commit: requires a real repo + a dirty board artifact
repo = subprocess.run(["git", "rev-parse", "--is-inside-work-tree"],
                      capture_output=True).returncode == 0
if repo and (has_db or glob("*.parquet")):
    subprocess.run(["git", "add", "-f", "board.db"], check=True)
    subprocess.run(["git", "commit", "-F", ".gitmessage.board"], check=True)
    check("commit", True, "board-only commit made")
else:
    check("commit", False, "no git repo or no board artifact -> no commit manufactured")

sys.exit(0 if all(ok or not required for ...) else 1)  # no-op audit = passed

Key principle: absence is a pass condition, not a prompt to create data. A cooldown PUT when the pin is 900 (or fleet.toml is missing) would be a destructive over-reach; exporting a parquet from a nonexistent board.db would be fabrication; a commit outside a git work-tree is impossible and must be reported as such, not simulated.

Answer 2

Diagnosis. The "idle NEVER-DONE" state at tick #47 is the bug, not the gates: every gate is green, the cooldown pin already matches the API, CI-003 is correctly human-gated — but nothing persistently records the completed audit, so the scheduler re-ticks the same task forever (47 times). The fix is an idempotent, self-terminating maintenance routine.

Fix — /workspace/audit_idle_maintenance.py (stdlib-only, tested on Python 3.14):

1. One verdict per tick. The audit is DONE only when every gate is present and failed == 0; a missing or red gate stays NEVER-DONE and reports failures:

def aggregate_gates(gates: Iterable[GateResult]) -> tuple[dict[str, GateResult], bool]:
    by_name = {g.name: g for g in gates}
    missing = [name for name in GATE_NAMES if name not in by_name]
    ok = not missing and all(g.ok for g in by_name.values())
    return by_name, ok

1. Idempotent DONE marking, no downgrades. mark_done only advances the board when the tick is strictly newer; re-running the same or an older tick is a no-op that terminates the loop:

def mark_done(self, record: dict) -> tuple[bool, bool]:
    tick_id = int(record["tick_id"])
    current = self.load_done_tick()
    if current is not None and tick_id <= current:
        return False, True            # never downgrade; same/older tick is idle
    self.state_path.write_text(json.dumps(record, indent=2) + "\n", encoding="utf-8")
    self.append_jsonl(record)
    return True, False

1. Append-only JSONL mirror, keyed by tick_id so re-exports never duplicate a line. Board storage is an interface (FileBoard for tests, DuckDBBoard for the ~/.hermes/venvs/board venv).

2. Cooldown-aware scheduler PUT — no PUT when the pin already matches the API or when inside the 900s window; no pin configured means nothing to sync:

def should_put_scheduler(pin_api, pin_current, last_put_ts, now_ts, cooldown_s=COOLDOWN_S) -> bool:
    if pin_api is None:
        return False
    if pin_current == pin_api:
        return False
    return now_ts - last_put_ts >= cooldown_s

1. Human gate on git pushes — the agent policy refuses the push and records the block (push_blocked=True, pushed=0), so CI-003 stays human-gated instead of silently skipping.

run_maintenance(tick_id, runner, board, gate, pin_api, pin_current, last_put_ts, ...) executes the full tick and returns an AuditReport with done_recorded / already_done / idle / put_issued / push_blocked; a CLI entry point runs it against a board directory.

Answer 3

Root cause — the last_commit: header (and state record) was only written on the changed path (commit and commit != last_commit). A transient resolver failure (tick #48) produced commit=None, which short-circuited the write entirely: tick #48's record was silently dropped and the header fell one tick behind. Because the stale header then equals the previous known commit, the "changed" branch re-arms and the lag is self-perpetuating — the 7th occurrence of the recurring lag. Two secondary defects surfaced while fixing:

1. Non-atomic header/state: header write and state write were separate steps, so a crash (or a red gate mid-tick) left them divergent.
2. Gate ordering: the green-gate check ran after the header/state writes, so an aborted tick still polluted the log/state.

Fix (in ~/audit_maintenance.py), the corrected tick flow:

def run_tick(ctx: TickContext) -> dict:
    state = load_state(ctx.state_path)

    # 1) None-safe resolution: failure tick falls back to last-known SHA.
    commit = resolve_commit(ctx)
    best = commit or state.get("last_commit")
    if commit is None:
        ctx.warn("commit unresolved; header holds last-known SHA")

    # 2) All abortable gates run BEFORE any write (zero side effects).
    pin, api_pin = fleet_pin(ctx), int(ctx.api.pin())   # No-PUT cooldown
    if pin != api_pin:
        raise MaintenanceAbort(f"No-PUT cooldown: {pin} != {api_pin}")
    if not ctx.green:
        raise MaintenanceAbort(f"gate red (tick {ctx.tick})")

    # 3) Unconditional header refresh + state commit, atomically.
    if best:
        write_header_atomic(ctx.log_path, best)         # temp file + os.replace
        state["last_commit"] = best
    state["last_tick"] = ctx.tick
    save_state_atomic(ctx.state_path, state)
    return {"tick": ctx.tick, "last_commit": best, "warnings": ctx.warnings}

Key change: the header is refreshed every tick to the best-known commit — unconditional, not gated on a "changed" comparison — and header + state move together through one atomic os.replace per file:

def write_header_atomic(log_path: Path, sha: str) -> None:
    body = log_path.read_text() if log_path.exists() else ""
    tmp = Path(tempfile.mktemp(dir=str(log_path.parent), prefix=".audit-hdr-"))
    try:
        tmp.write_text(f"last_commit: {sha}\n" + "\n".join(body.splitlines()[1:]))
        os.replace(tmp, log_path)   # atomic on same filesystem
    finally:
        if tmp.exists():
            tmp.unlink()

Answer 4

There is nothing to fix, and I will not fabricate one.

I inspected the environment before responding. The working directory (~) contains no repository, no Python files, no test suite, and no totalstack-board-update.py — it is an empty user home with only agent tooling. The problem text is a status report describing an audit loop that is already in its desired steady state:

• all four gates parallel and healthy (load 3.83)
• suite 1864/0/208, guard 5/5, validator 76/76, hilo 12260/1680
• scheduler pin matched live → no PUT needed
• No-PUT pattern holds for 27+ ticks; commit 3017b38d8 is the 28th consecutive minimal 2-file diff

A loop that has confirmed idle/healthy behavior for 27+ ticks does not require a "fix" — it requires no change at all. The only correct maintenance action is a no-op: continue the idle watch. Inventing a code change, a bug, or a "fix" here would be fabrication.

For reference only (illustrative, not applied anywhere — there is no codebase to apply it to), the only code a healthy idle-maintenance audit loop would emit is a guard that detects drift and stays silent otherwise:

# illustrative only — no such module exists in this environment
def audit_tick(suite, guard, validator, hilo, pin_live, pin_board, max_ticks=30):
    """Idle-maintenance guard: PUT only when live state diverges from board."""
    healthy = (
        suite.failed == 0
        and guard.passed == guard.total
        and validator.passed == validator.total
        and hilo.drift <= hilo.threshold
        and pin_live == pin_board          # scheduler pin matched live -> no PUT
    )
    if healthy:
        return {"action": "noop", "reason": "steady-state confirmed", "ticks": max_ticks}
    return {"action": "put", "reason": "drift detected", "diff": plan_diff(pin_live, pin_board)}

This is provided only to show the shape of the right behavior; it was not written into any file and should not be treated as a deliverable.

Answer 5

No code fix is required — this is a true idle tick. The definitive gate set re-ran fresh this tick and is green across the board; the scheduler state needs no PUT (fleet.toml pin 7200 already matches the API), and the only board mutation was the JSONL commit 93ecaf5e61 already recorded.

Root-cause disposition of the one flagged anomaly:

• Symptom: parallel background run's polly e2e tests executed instead of skipping, then failed with LocalStack HTTP 501 DescribeVoices not supported.
• Cause: a transient real LocalStack instance answered the :4566 health probe during the window when the parallel bg run probed it. The suite's skip-guard (probe :4566 → if up, assume test-LocalStack and run polly e2e) misclassified that foreign responder as the project's test LocalStack. The port was free immediately after; no container/process trace existed — i.e., the responder vanished. This is the same known :4566 contention class as tick #152's fsx_e2e flake.
• Disposition: environmental flake, not a product defect. The isolation re-run is the definitive gate and it passed 1865/0/208. No code change, no scheduler PUT, no new commit beyond the board JSONL.

The only legitimate "fix" would be optional defensive hardening of the skip-guard (capability-negotiation instead of bare port-liveness), which is not applied here because (a) no repo exists in this workspace to change, and (b) the gates are already green. For reference, the hardening pattern would be:

# Optional hardening (illustrative, NOT applied — no failing gate):
# Classify the :4566 responder by capability, not just liveness.
def polly_available() -> bool:
    """Probe LocalStack polly, not just the port. A bare 501 means a
    foreign/incapable responder answered the port — treat as absent."""
    try:
        polly.describe_voices()          # raises 501 if unsupported
        return True
    except ClientError as e:
        if e.response["ResponseMetadata"]["HTTPStatusCode"] == 501:
            logger.warning(":4566 responder lacks polly — treating as absent (transient contention)")
            return False                 # skip polly e2e instead of failing
        raise

This converts the :4566 contention class (ticks #152/#153) from "tests ran against a stranger and 501'd" into "skip, exactly as designed." It is a candidate for a future maintenance tick only if the board's backlog wants hardening — the current tick's verdict is unaffected.

Answer 6

Three defects were fixed: (A) pytest -c /dev/null collection mangling when the gate runs from a background/drifted terminal, (B) an unpinned 7200s No-PUT cooldown, (C) board updates not enforcing the JSONL-only 3-file delta discipline.

A. Gate runner — foreground-only, rootdir-pinned, floor-checked (scripts/run-gate.sh)

#!/usr/bin/env bash
# NEVER-DONE gate battery. FOREGROUND-ONLY.
# Root cause: `pytest -c /dev/null` derives rootdir from /dev/null's dirname
# (/dev). From a background terminal (drifted cwd) collection sweeps in phantom
# /dev tests (observed: ../../../dev/test_fsx_e2e.py FAIL) and the repo ini/
# conftest stack stops shaping collection → subset silently shrinks (747).
set -euo pipefail

REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
COLLECT_FLOOR="${COLLECT_FLOOR:-2000}"   # far above the 747 bg subset, under the full gate

cd "$REPO_ROOT"                          # (1) pin cwd — kills drift regardless of terminal

# (2) foreground policy — enforce only where a controlling TTY exists
if [[ -t 0 && -t 1 ]]; then
  stat="$(ps -o stat= -p $$)"
  case "$stat" in
    *+) : ;;  # foreground process group (trailing '+' in STAT)
    *)  echo "FATAL: gate must run in the FOREGROUND (STAT=$stat)." >&2; exit 3 ;;
  esac
fi

# (3) collection sanity — prove the full set is visible BEFORE running
collected="$(python3 -m pytest -c /dev/null --rootdir="$REPO_ROOT" --collect-only -q 2>/dev/null | tail -1)"
count="$(echo "$collected" | grep -oE '^[0-9]+' || echo 0)"
if (( count < COLLECT_FLOOR )); then
  echo "FATAL: collection drift — got $count tests (floor $COLLECT_FLOOR)." >&2; exit 4
fi
echo "collection ok: $collected"

# (4) the battery — explicit config, explicit rootdir, foreground
python3 -m pytest -c /dev/null --rootdir="$REPO_ROOT" -q -p no:cacheprovider "$@"

gitreins guard — its test_command embeds -c /dev/null, so point it at the wrapper instead of the raw pytest invocation:

[guard]
# was: test_command = "pytest -c /dev/null -q"
test_command = "./scripts/run-gate.sh"   # inherits cd-pin, rootdir-pin, floor, fg-policy
foreground = true                        # never detach/background the guard process

B. No-PUT cooldown pin — one constant, one config, one verifier run in the gate:

# constants.py — must match fleet.toml [put] cooldown_seconds
SCHEDULER_PUT_COOLDOWN_SECONDS = 7200   # 2h, pinned by convention

# fleet.toml
[put]
cooldown_seconds = 7200

# verify_cooldown.py — wired into the gate, not just deploy
import sys, tomllib
from pathlib import Path
from constants import SCHEDULER_PUT_COOLDOWN_SECONDS as PIN

cfg = tomllib.loads(Path("fleet.toml").read_text())
try:
    file_val = cfg["put"]["cooldown_seconds"]
except KeyError as e:
    sys.exit(f"FATAL: fleet.toml missing key {e}")
if float(file_val) != float(PIN) or float(PIN) != 7200.0:
    sys.exit(f"FATAL: fleet.toml={file_val!r} != pinned {PIN!r}")
print(f"OK: cooldown pin {PIN} matches fleet.toml ({file_val!r})")

C. Board delta discipline — JSONL-only, plain git add, heredoc commit with co-author trailer:

git add board.jsonl events.jsonl tasks.jsonl            # plain add, no -A, no pathspec tricks
delta=$(git diff --cached --name-only | sort)
[ "$(echo "$delta" | wc -l)" -eq 3 ] || { echo "FAIL: staging != 3 files"; exit 1; }
echo "$delta" | grep -qE '^(board|events|tasks)\.jsonl$' || { echo "FAIL: non-JSONL staged"; exit 1; }
git commit -F - <<'EOF'
chore(board): tick #164 idle audit — gate battery green

Co-authored-by: TotalStack Bot <<email>>
EOF

Answer 7

The tick-#166 failure had three root causes, all fixed in maintenance.py (a standalone orchestrator; real deployment only needs to bind the Runner stub to actual commands):

1. Parallel-load 300s timeout → fail-open PASS trap. The old code co-loaded guard + suite in the same process pool; a slow load blew the 300s budget and the guard silently degraded to PASS (false green). Fix: the audit is strictly sequential — guard runs SOLO first (exclusive per-phase lock), then suite standalone, then ACM standalone. The guard now fails closed (RuntimeError) if the loader times out instead of returning an implicit PASS:

def run_guard_solo(self) -> dict:
    with self.guard_lock:                      # exclusive: nothing else loads
        guard = self.runner.guard(load_timeout=LOAD_TIMEOUT_S)
        if guard is None:                      # loader timeout / crash
            raise RuntimeError(                 # FAIL CLOSED (never pass-open)
                f"guard load exceeded {LOAD_TIMEOUT_S}s; refusing to pass-open"
            )
        result = self.runner.suite_safety_trigger()   # safety trigger = full suite
    ...log tick "guard" (clean 5/5)...

2. board.db cache lag (157 vs JSONL 165) + JSONL regression. The DuckDB import path rewrote stale state back into the JSONL. Fix: sync_boarddb rebuilds the cache only from the JSONL-direct update script; if a DuckDB path is passed it is deliberately ignored, and the JSONL stays append-only (never written by sync):

def sync_boarddb(self, duckdb_path: Path | None = None) -> dict:
    total = self.log.ticks_total()            # JSONL = source of truth
    cache = self.board.load(); before = cache.get(BoardCache.KEY)
    cache[BoardCache.KEY] = total; self.board.save(cache)
    return {"from": "jsonl-direct", "before": before, "after": total}

3. Cooldown 7200. PUT only when cooldown is NOT active — i.e. pin+API match AND elapsed < 7200s ⇒ no PUT:

def cooldown_active(self, api=None, local=None) -> bool:
    api = api or self.api.fetch()
    local = local if local is not None else self.pin_store.read()
    if not local: return False
    return (api.get("pin") == local.get("pin")) and \
           (time.time() - float(api.get("last_run", 0))) < COOLDOWN_S

Audit order is enforced: guard → suite → acm → JSONL-direct board sync → maybe_put.

Answer 8

Root cause — the last_commit: header (and state record) was only written on the changed path (commit and commit != last_commit). A transient resolver failure (tick #48) produced commit=None, which short-circuited the write entirely: tick #48's record was silently dropped and the header fell one tick behind. Because the stale header then equals the previous known commit, the "changed" branch re-arms and the lag is self-perpetuating — the 7th occurrence of the recurring lag. Two secondary defects surfaced while fixing:

1. Non-atomic header/state: header write and state write were separate steps, so a crash (or a red gate mid-tick) left them divergent.
2. Gate ordering: the green-gate check ran after the header/state writes, so an aborted tick still polluted the log/state.

Fix (in ~/audit_maintenance.py), the corrected tick flow:

def run_tick(ctx: TickContext) -> dict:
    state = load_state(ctx.state_path)

    # 1) None-safe resolution: failure tick falls back to last-known SHA.
    commit = resolve_commit(ctx)
    best = commit or state.get("last_commit")
    if commit is None:
        ctx.warn("commit unresolved; header holds last-known SHA")

    # 2) All abortable gates run BEFORE any write (zero side effects).
    pin, api_pin = fleet_pin(ctx), int(ctx.api.pin())   # No-PUT cooldown
    if pin != api_pin:
        raise MaintenanceAbort(f"No-PUT cooldown: {pin} != {api_pin}")
    if not ctx.green:
        raise MaintenanceAbort(f"gate red (tick {ctx.tick})")

    # 3) Unconditional header refresh + state commit, atomically.
    if best:
        write_header_atomic(ctx.log_path, best)         # temp file + os.replace
        state["last_commit"] = best
    state["last_tick"] = ctx.tick
    save_state_atomic(ctx.state_path, state)
    return {"tick": ctx.tick, "last_commit": best, "warnings": ctx.warnings}

Key change: the header is refreshed every tick to the best-known commit — unconditional, not gated on a "changed" comparison — and header + state move together through one atomic os.replace per file:

def write_header_atomic(log_path: Path, sha: str) -> None:
    body = log_path.read_text() if log_path.exists() else ""
    tmp = Path(tempfile.mktemp(dir=str(log_path.parent), prefix=".audit-hdr-"))
    try:
        tmp.write_text(f"last_commit: {sha}\n" + "\n".join(body.splitlines()[1:]))
        os.replace(tmp, log_path)   # atomic on same filesystem
    finally:
        if tmp.exists():
            tmp.unlink()

Answer 9

There is nothing to fix, and I will not fabricate one.

I inspected the environment before responding. The working directory (~) contains no repository, no Python files, no test suite, and no totalstack-board-update.py — it is an empty user home with only agent tooling. The problem text is a status report describing an audit loop that is already in its desired steady state:

• all four gates parallel and healthy (load 3.83)
• suite 1864/0/208, guard 5/5, validator 76/76, hilo 12260/1680
• scheduler pin matched live → no PUT needed
• No-PUT pattern holds for 27+ ticks; commit 3017b38d8 is the 28th consecutive minimal 2-file diff

A loop that has confirmed idle/healthy behavior for 27+ ticks does not require a "fix" — it requires no change at all. The only correct maintenance action is a no-op: continue the idle watch. Inventing a code change, a bug, or a "fix" here would be fabrication.

For reference only (illustrative, not applied anywhere — there is no codebase to apply it to), the only code a healthy idle-maintenance audit loop would emit is a guard that detects drift and stays silent otherwise:

# illustrative only — no such module exists in this environment
def audit_tick(suite, guard, validator, hilo, pin_live, pin_board, max_ticks=30):
    """Idle-maintenance guard: PUT only when live state diverges from board."""
    healthy = (
        suite.failed == 0
        and guard.passed == guard.total
        and validator.passed == validator.total
        and hilo.drift <= hilo.threshold
        and pin_live == pin_board          # scheduler pin matched live -> no PUT
    )
    if healthy:
        return {"action": "noop", "reason": "steady-state confirmed", "ticks": max_ticks}
    return {"action": "put", "reason": "drift detected", "diff": plan_diff(pin_live, pin_board)}

This is provided only to show the shape of the right behavior; it was not written into any file and should not be treated as a deliverable.