board-migration-duckdb-cutover
I built the complete cutover tool (migrate.py) that implements every post-fix directive as a hard, tested invariant. Pipeline: tasks.md matrix → parse → normalize → validate (hard abort) → git-safe JSONL → archive .bak → PUT+GET-verify+pin → two-commit backfill → gitreins dual-track → audit event.
Directive 1 — namespace basename bug (home → project-key): the sentinel home (or empty) resolves to the basename of the fleet repo path, never "home" again:
NAMESPACE_BUG_SENTINEL = "home"
def _project_key(repo, fallback):
base = os.path.basename(repo.rstrip("/")) if repo else ""
return base[:-4] if base.endswith(".git") else (base or fallback)
if rec.ns == NAMESPACE_BUG_SENTINEL or not rec.ns:
rec.ns = _project_key(rec.repo, rec.key) # home -> alpha, never "home"
Directive 2 — cooldown_s=7200 (Bane 07-31), NOT 43200: constant pinned to 7200; the legacy 43200 is migrated in normalize() (logged to audit) and any non-7200 record fails validate():
BANE_31_JULY_COOLDOWN_S = 7200
LEGACY_WRONG_COOLDOWN_S = 43200
if rec.cooldown_s != BANE_31_JULY_COOLDOWN_S:
notes.append({"field": "cooldown_s", "from": rec.cooldown_s,
"to": 7200, "reason": "Bane 07-31 directive (7200, not 43200)"})
rec.cooldown_s = BANE_31_JULY_COOLDOWN_S
Directive 3 — DecayRate=0 rejected: hard validation before any write — no JSONL, no archive, no pin, no audit:
if not math.isfinite(rec.decay_rate) or rec.decay_rate <= 0:
errors.append(f"{rec.key}: DecayRate={rec.decay_rate!r} rejected (must be finite and > 0)")
Directive 4 — enforcer-set 900 policy-correct while BOARD-V2 pending: preserved verbatim — normalize() never touches it, the constant is documented, and a regression-guard test asserts no note/rewrite exists for the field:
ENFORCER_POLICY_WHILE_BOARD_V2_PENDING = 900 # policy-correct; keep verbatim
Directive 5 — durability: PUT → GET → field-level round-trip verify → atomic fleet.toml pin (os.replace), optionally git commit:
fleet_put_get_verify(store, records) # every record PUT then GET+field-checked
pin_fleet_toml(fleet_toml, jsonl_sha, len(records), snap_sha, cutover_ts)
Directive 6 — two-commit hash backfill: empty commits get a deterministic 12-char digest over HEAD and HEAD~1:
digest = hashlib.sha256(f"{head}:{head1}".encode()).hexdigest()[:12]
Directive 7 — gitreins dual-tracking: the task is registered in both gitreins/manifest.toml (tracked_in: ["gitreins-manifest","export-jsonl"]) and the JSONL (dual_tracked: true); completion criterion is file existence:
def is_complete(self): return os.path.isfile(self.rec.criterion_file)
Directive 8 — audit event instead of tasks.md append: tasks.md is archived to .bak (refusing to clobber) and made immutable; the cutover is appended to audit/events.jsonl (event: "board_v2_cutover" with JSONL sha, archive sha, records, migration notes) via append-only write. JSONL is git-safe: LF-only, trailing newline, sorted keys, sorted record order, no NaN/Inf, byte-reproducible, and verified by a real DuckDB load (verify_duckdb — row count, cooldown_s != 7200 count, ns='home' count, decay_rate<=0/NaN count all must be 0).
**Test suite: 43 passed, 0 failed** (`tests/test_migrate.py`, run with `duckdb 1.5.5`): | Class | What it proves | |---|---| | `TestParseMatrix` | header/aliases, CRLF+BOM, unknown columns, dup-key & missing-field rejection, empty matrix | | `TestNamespaceBasenameFix` | `home→alpha/beta`, `.git` stripped, explicit ns preserved, no `home` survives validation | | `TestCooldownDirective` | constant is 7200≠43200, 43200→7200 rewrite + audit note, raw 43200 rejected | | `TestDecayRateZero` | 0/NaN/Inf/-1 rejected; **abort leaves tasks.md, export, audit all untouched** | | `TestEnforcerPolicy` | 900 preserved, zero notes touch `enforcer` | | `TestJsonlGitSafe` | LF+trailing NL, sorted keys, byte-identical re-runs, non-finite rejected, CRLF caught, empty export valid, **real DuckDB load** | | `TestArchive` | rename preserves bytes, clobber refused | | `TestFleetDurability` | PUT/GET round-trip, mismatch detected (flaky GET), pin contents, pin sha == export sha | | `TestTwoCommitBackfill` | 2-commit repo backfills deterministically, existing hashes untouched, skipped without git | | `TestDualTrackedTask` | in both manifest+export, incomplete until file exists, then complete | | `TestAudit` | event recorded, `.bak` byte-identical to original, append-only | | `TestEndToEnd` | full cutover incl. DuckDB `"ok"`, backfill inside pipeline, dry-run touches nothing | **Live CLI runs:** - Sample matrix (4 rows): all `home→alpha/beta`, all `43200→7200`, enforcer `900` kept, export sha `5c96bb…` matched in `fleet.toml` pin and store snapshot, `duckdb verify: ok`, `tasks.md.bak` produced, audit event emitted (not a tasks.md append). - `DecayRate=0` row → exit 2, `tasks.md` intact, **no** `out.jsonl`, **no** audit dir. - Temp 2-commit git repo → `T-77 commits '' → '390c714c0926'` backfilled, `duckdb verify: ok`, and `git log` shows the durability commit containing `export.jsonl + fleet.toml + store/snapshot.jsonl + tasks.md.bak + audit/events.jsonl`. Edge cases covered: empty matrix (valid empty export), CRLF/BOM input, trailing `.git` in repo paths, duplicate keys, missing required fields, non-finite floats, existing `.bak` clobber refusal, flaky store GET mismatch, dry-run, idempotent manifest re-registration, abort-before-any-write atomicity.
{"model": "deepseek-v4-flash", "problem_class": "board-migration-duckdb-cutover", "result": "passed", "tests": 43}