board-jsonl-blank-line-separator-preservation
Diagnosed and verified. The referenced repo isn't present in this environment, so I reconstructed the exact scenario (/tmp/boardrepro) and confirmed the root cause, the minimal fix, and a byte-preserving robust fix empirically.
Root cause: The board files are blank-line-separated JSONL (one object per line, each followed by an empty line). The rewrite joined records with a single \n, so git saw every blank separator as a deletion and every following JSON line as shifted — inflating a 2-record edit into a near-full-file diff. No records were lost; untouched rows were byte-identical because dicts were mutated in place (stable json.dumps key order).
Verified results (35-row repro, T7 updated, T99 appended):
| Rewrite | git diff --numstat |
lost | added | changed |
|---|---|---|---|---|
broken "\n".join |
2 35 |
[] |
T99 |
T7 |
fixed "\n\n".join |
3 1 |
[] |
T99 |
T7 |
| surgical raw-line reuse | 3 1 |
[] |
T99 |
T7 |
Content comparison is identical across all three — proving the big diff was pure separator noise. jq -c . file \| wc -l still parses all 36 records after the fix.
Exact fix: write "\n\n".join(json.dumps(r, ensure_ascii=False) for r in records) + "\n" — still valid JSONL since per-line parsers skip blanks. Best practice is the surgical writer that reuses raw source lines for untouched records to avoid key-order/separator drift.
Detect before rewriting: compare parsed id -> row dicts old-vs-new (lost == []), and profile blank vs nonblank line counts on the HEAD version. Never infer data loss from a line diff.
The full self-contained write-up (root cause, detection, fix, verification, drop-in board_edit.py helper, and a guard checklist) is at:
~/board-jsonl-separator-preservation.md
# Evidence - Problem class: board-jsonl-blank-line-separator-preservation - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-13T01:05:33.801Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Rewriting a JSONL board file in Python to update one row destroyed the file's inter-record formatting and produced a 194-line diff for a 1-row change. SYMPTOM: after updating one task row and appending a new one, `git diff --numstat .coding-hermes/board/tasks.jsonl` showed '3 insertions / 33 deletions' and events.jsonl showed ~170 changed lines, even though a content-level comparison proved ZERO records were lost and only 2 records actually changed. ROOT CAUSE: the on-disk board was written as blank-line-separated JSONL - one JSON object per line, each followed by an EMPTY line (tasks.jsonl: 35 records over 66 lines = 32 blanks; events.jsonl: 163 records over 317 lines). My rewrite emitted records as one JSON per line with no blanks, so git rendered every removed separator as a deletion. The records themselves were byte-identical for unchanged rows (same json.dumps key order because dicts were mutated in place, not rebuilt), which is why a naive 'diff looks huge' read suggests data loss when there is none. DETECTION: (a) count nonblank vs blank lines in the HEAD version before rewriting - a large blank count means separators are part of the convention; (b) after a rewrite, compare parsed content sets old-vs-new (id -> row dict equality) instead of trusting line diffs; ids lost == [] and only intended ids changed == data-safe. FIX: preserve the writer's record separator - write '\\n\\n'.join(json.dumps(r, ensure_ascii=False) for r in records) + '\\n'. The result is still valid JSONL because every per-line parser (jq, python per-line loop, Go bufio line scan) tolerates and skips blank lines; only whole-file json.loads() would break, and that is never used for JSONL boards. VERIFY: git diff --numstat then shows only the genuinely changed records (tasks.jsonl 7/2, events.jsonl 22/6 in the fixed run). Also relevant: the same class of churn happens from json.dumps separator drift (', ' vs ','), so re-serializing ANY hand-maintained JSONL should either preserve separators byte-for-byte or be validated with the parsed-content comparison above before commit. Repo: ~/my-project (.coding-hermes/board JSONL canonical store, commit b09d27a / 5403827).", "environment": "local linux, python3 stdlib, git; board = blank-line-separated JSONL (per-line parsers)", "language": "python", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "board-jsonl-blank-line-separator-preservation", "provider": "openrouter", "solved_at": "2026-09-13T01:05:33.802Z", "version": ""}Diagnosed and verified. The referenced repo isn't present in this environment, so I reconstructed the exact scenario (/tmp/boardrepro) and confirmed the root cause, the minimal fix, and a byte-preserving robust fix empirically.
Root cause: The board files are blank-line-separated JSONL (one object per line, each followed by an empty line). The rewrite joined records with a single \n, so git saw every blank separator as a deletion and every following JSON line as shifted — inflating a 2-record edit into a near-full-file diff. No records were lost; untouched rows were byte-identical because dicts were mutated in place (stable json.dumps key order).
Verified results (35-row repro, T7 updated, T99 appended):
| Rewrite | git diff --numstat |
lost | added | changed |
|---|---|---|---|---|
broken "\n".join |
2 35 |
[] |
T99 |
T7 |
fixed "\n\n".join |
3 1 |
[] |
T99 |
T7 |
| surgical raw-line reuse | 3 1 |
[] |
T99 |
T7 |
Content comparison is identical across all three — proving the big diff was pure separator noise. jq -c . file \| wc -l still parses all 36 records after the fix.
Exact fix: write "\n\n".join(json.dumps(r, ensure_ascii=False) for r in records) + "\n" — still valid JSONL since per-line parsers skip blanks. Best practice is the surgical writer that reuses raw source lines for untouched records to avoid key-order/separator drift.
Detect before rewriting: compare parsed id -> row dicts old-vs-new (lost == []), and profile blank vs nonblank line counts on the HEAD version. Never infer data loss from a line diff.
The full self-contained write-up (root cause, detection, fix, verification, drop-in board_edit.py helper, and a guard checklist) is at:
~/board-jsonl-separator-preservation.md
# Evidence - Problem class: board-jsonl-blank-line-separator-preservation - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-13T01:05:33.801Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Rewriting a JSONL board file in Python to update one row destroyed the file's inter-record formatting and produced a 194-line diff for a 1-row change. SYMPTOM: after updating one task row and appending a new one, `git diff --numstat .coding-hermes/board/tasks.jsonl` showed '3 insertions / 33 deletions' and events.jsonl showed ~170 changed lines, even though a content-level comparison proved ZERO records were lost and only 2 records actually changed. ROOT CAUSE: the on-disk board was written as blank-line-separated JSONL - one JSON object per line, each followed by an EMPTY line (tasks.jsonl: 35 records over 66 lines = 32 blanks; events.jsonl: 163 records over 317 lines). My rewrite emitted records as one JSON per line with no blanks, so git rendered every removed separator as a deletion. The records themselves were byte-identical for unchanged rows (same json.dumps key order because dicts were mutated in place, not rebuilt), which is why a naive 'diff looks huge' read suggests data loss when there is none. DETECTION: (a) count nonblank vs blank lines in the HEAD version before rewriting - a large blank count means separators are part of the convention; (b) after a rewrite, compare parsed content sets old-vs-new (id -> row dict equality) instead of trusting line diffs; ids lost == [] and only intended ids changed == data-safe. FIX: preserve the writer's record separator - write '\\n\\n'.join(json.dumps(r, ensure_ascii=False) for r in records) + '\\n'. The result is still valid JSONL because every per-line parser (jq, python per-line loop, Go bufio line scan) tolerates and skips blank lines; only whole-file json.loads() would break, and that is never used for JSONL boards. VERIFY: git diff --numstat then shows only the genuinely changed records (tasks.jsonl 7/2, events.jsonl 22/6 in the fixed run). Also relevant: the same class of churn happens from json.dumps separator drift (', ' vs ','), so re-serializing ANY hand-maintained JSONL should either preserve separators byte-for-byte or be validated with the parsed-content comparison above before commit. Repo: ~/my-project (.coding-hermes/board JSONL canonical store, commit b09d27a / 5403827).", "environment": "local linux, python3 stdlib, git; board = blank-line-separated JSONL (per-line parsers)", "language": "python", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "board-jsonl-blank-line-separator-preservation", "provider": "openrouter", "solved_at": "2026-09-13T01:05:33.802Z", "version": ""}