board-storage-jsonl-canonical-migration
Implemented JSONL-NORM-001 in ~/board/ — a complete DuckDB-board → git-tracked JSONL canonical store migration. Artifacts: seed_board.py (legacy DuckDB board), migrate_to_jsonl.py (the migration), append_board_task_completed.py (canonical append), test_jsonl_norm_001.py (64-check suite).
1. The canonical writer — strict single-line, compact separators, raw UTF-8:
def canonical_dumps(obj):
"""Strict single-line canonical JSON: no whitespace padding, raw UTF-8."""
return json.dumps(obj, ensure_ascii=False, separators=(",", ":"), sort_keys=True)
def write_canonical_jsonl(path, records):
with open(path, "wb") as f: # one record per line, b'\n' LF-only
for rec in records:
f.write(canonical_dumps(rec).encode("utf-8")); f.write(b"\n")
2. Migration (migrate_to_jsonl.py) — extract from DuckDB → write tasks.jsonl/events.jsonl/fixtures.jsonl + board.jsonl (normalized header first line) → drop binary artifacts from the index, ignore them, commit:
git rm --cached --ignore-unmatch board.db parquet/*.parquet
printf '*.parquet\nboard.db\n' >> .gitignore
git add tasks.jsonl events.jsonl fixtures.jsonl board.jsonl .gitignore
git commit -m "JSONL-NORM-001: migrate DuckDB board to git-tracked JSONL canonical store"
Header written as a strict single line with the lag convention baked in: tick = commits+1, last_commit = commits (lags one tick).
3. Byte-preserving append (append_board_task_completed.py) — read files as raw bytes, split on \n, re-emit untouched lines verbatim, change only the touched task line + header tick, append one canonical event line:
def load_lines_bytes(path): # lines WITHOUT trailing \n
raw = open(path, "rb").read().split(b"\n")
return raw[:-1] if raw and raw[-1] == b"" else raw
def save_lines_bytes(path, lines):
open(path, "wb").write(b"\n".join(lines) + b"\n")
old_tick = header["tick"]; new_tick = old_tick + 1
event = {"event_id": next_event_id(events), "type": "task_completed",
"task_id": args.task_id, "ts": ts,
"last_commit": old_tick, # state committed in git BEFORE this append
"payload": {"note": args.note}}
new_events = events_lines + [canonical_dumps(event).encode("utf-8")] # events.jsonl: append-only
new_tasks[task_idx] = canonical_dumps(done_task).encode("utf-8") # tasks.jsonl: touch 1 line
new_header = dict(header, tick=new_tick) # board.jsonl: tick up, last_commit still lags
last_commit lags one tick by convention: the appended record carries the tick of the last state already committed in git (the commit that will contain this append cannot exist yet), so new_tick − event.last_commit == 1 always holds.
Suite `test_jsonl_norm_001.py` (rebuilds from scratch each run — deterministic, ran twice: **64/64 both times**):
- **Git tracking**: after `git rm --cached`, `git ls-files` shows the 4 canonical `.jsonl` files; `board.db` and `parquet/*.parquet` absent; `git check-ignore -v` reports both `.gitignore:3:board.db` and `.gitignore:2:*.parquet`.
- **Commit history**: `legacy: duckdb board with parquet exports` → `JSONL-NORM-001: migrate DuckDB board to git-tracked JSONL canonical store` → `board: complete T-002 via append_board_task_completed.py`; `git status` clean after each commit.
- **Header normalization**: `board.jsonl` line 1 is exactly `{"board":"kanban","last_commit":1,"normalized":true,"schema":1,"tick":3,"type":"board_header","updated_at":...}` — single line, no `", "`/`": "` padding (regex-verified), `normalized:true`.
- **Byte preservation**: `diff` of `events.jsonl` before/after the append shows exactly one added line (`4a5`); byte comparison confirms all 4 pre-existing lines are byte-identical and only the `E-005` line was appended. Same invariant holds for `tasks.jsonl` (exactly the `T-002` line changed) and `board.jsonl` (only header tick + `T-002` state line changed; all other lines byte-identical).
- **Raw UTF-8**: appended payload `{"note":"完成 東京ラベル — ünïcode ✓"}` stored unescaped (`ensure_ascii=False`); no `\uXXXX` escapes in any file (regex-verified on every line).
- **Lag convention**: pre-append header tick 2 → `E-005.last_commit == 2`, post-append header tick 3 → `3 − 2 == 1`. Same invariant holds when appending to an empty events stream.
- **Edge cases**: duplicate completion rejected ("already done", exit≠0), unknown `task_id` rejected, empty-stream append works and yields `E-001`, scratch-dir append leaves the repo store untouched, every line of every stream round-trips as valid JSON pre- and post-append.{"model": "deepseek-v4-flash", "problem_class": "board-storage-jsonl-canonical-migration", "result": "passed", "tests": 64}Implemented JSONL-NORM-001 in ~/board/ — a complete DuckDB-board → git-tracked JSONL canonical store migration. Artifacts: seed_board.py (legacy DuckDB board), migrate_to_jsonl.py (the migration), append_board_task_completed.py (canonical append), test_jsonl_norm_001.py (64-check suite).
1. The canonical writer — strict single-line, compact separators, raw UTF-8:
def canonical_dumps(obj):
"""Strict single-line canonical JSON: no whitespace padding, raw UTF-8."""
return json.dumps(obj, ensure_ascii=False, separators=(",", ":"), sort_keys=True)
def write_canonical_jsonl(path, records):
with open(path, "wb") as f: # one record per line, b'\n' LF-only
for rec in records:
f.write(canonical_dumps(rec).encode("utf-8")); f.write(b"\n")
2. Migration (migrate_to_jsonl.py) — extract from DuckDB → write tasks.jsonl/events.jsonl/fixtures.jsonl + board.jsonl (normalized header first line) → drop binary artifacts from the index, ignore them, commit:
git rm --cached --ignore-unmatch board.db parquet/*.parquet
printf '*.parquet\nboard.db\n' >> .gitignore
git add tasks.jsonl events.jsonl fixtures.jsonl board.jsonl .gitignore
git commit -m "JSONL-NORM-001: migrate DuckDB board to git-tracked JSONL canonical store"
Header written as a strict single line with the lag convention baked in: tick = commits+1, last_commit = commits (lags one tick).
3. Byte-preserving append (append_board_task_completed.py) — read files as raw bytes, split on \n, re-emit untouched lines verbatim, change only the touched task line + header tick, append one canonical event line:
def load_lines_bytes(path): # lines WITHOUT trailing \n
raw = open(path, "rb").read().split(b"\n")
return raw[:-1] if raw and raw[-1] == b"" else raw
def save_lines_bytes(path, lines):
open(path, "wb").write(b"\n".join(lines) + b"\n")
old_tick = header["tick"]; new_tick = old_tick + 1
event = {"event_id": next_event_id(events), "type": "task_completed",
"task_id": args.task_id, "ts": ts,
"last_commit": old_tick, # state committed in git BEFORE this append
"payload": {"note": args.note}}
new_events = events_lines + [canonical_dumps(event).encode("utf-8")] # events.jsonl: append-only
new_tasks[task_idx] = canonical_dumps(done_task).encode("utf-8") # tasks.jsonl: touch 1 line
new_header = dict(header, tick=new_tick) # board.jsonl: tick up, last_commit still lags
last_commit lags one tick by convention: the appended record carries the tick of the last state already committed in git (the commit that will contain this append cannot exist yet), so new_tick − event.last_commit == 1 always holds.
Suite `test_jsonl_norm_001.py` (rebuilds from scratch each run — deterministic, ran twice: **64/64 both times**):
- **Git tracking**: after `git rm --cached`, `git ls-files` shows the 4 canonical `.jsonl` files; `board.db` and `parquet/*.parquet` absent; `git check-ignore -v` reports both `.gitignore:3:board.db` and `.gitignore:2:*.parquet`.
- **Commit history**: `legacy: duckdb board with parquet exports` → `JSONL-NORM-001: migrate DuckDB board to git-tracked JSONL canonical store` → `board: complete T-002 via append_board_task_completed.py`; `git status` clean after each commit.
- **Header normalization**: `board.jsonl` line 1 is exactly `{"board":"kanban","last_commit":1,"normalized":true,"schema":1,"tick":3,"type":"board_header","updated_at":...}` — single line, no `", "`/`": "` padding (regex-verified), `normalized:true`.
- **Byte preservation**: `diff` of `events.jsonl` before/after the append shows exactly one added line (`4a5`); byte comparison confirms all 4 pre-existing lines are byte-identical and only the `E-005` line was appended. Same invariant holds for `tasks.jsonl` (exactly the `T-002` line changed) and `board.jsonl` (only header tick + `T-002` state line changed; all other lines byte-identical).
- **Raw UTF-8**: appended payload `{"note":"完成 東京ラベル — ünïcode ✓"}` stored unescaped (`ensure_ascii=False`); no `\uXXXX` escapes in any file (regex-verified on every line).
- **Lag convention**: pre-append header tick 2 → `E-005.last_commit == 2`, post-append header tick 3 → `3 − 2 == 1`. Same invariant holds when appending to an empty events stream.
- **Edge cases**: duplicate completion rejected ("already done", exit≠0), unknown `task_id` rejected, empty-stream append works and yields `E-001`, scratch-dir append leaves the repo store untouched, every line of every stream round-trips as valid JSON pre- and post-append.{"model": "deepseek-v4-flash", "problem_class": "board-storage-jsonl-canonical-migration", "result": "passed", "tests": 64}