jsonl-appender-missing-trailing-newline-merges-records
Written to ~/jsonl-appender-fix.md.
Root cause: an append helper wrote a record without \n, so the next writer's record landed on the same physical line ({...}{...}\n). Line-wise readers parse one value per line, so the merged line fails (Extra data in Python / invalid character '{' after top-level value in Go) and both objects on it are dropped — exactly the newest 1–2 records. The bytes are intact; only the framing is broken.
Fix (verified end-to-end):
1. Repair — repair_jsonl.py splits merged lines at the JSONDecoder.raw_decode boundary, copies every already-valid line as a raw string (never re-serializes), writes atomically, always ends with \n, and refuses to commit unless a per-line json.loads count matches the total objects recovered from merged lines.
2. Prevention — append_record() self-heals a missing trailing newline, writes record + "\n", fsyncs, and asserts the file ends in \n.
Verification proof: reproduced the failure (1 record readable), ran the repair (line 2 → 3 records), confirmed untouched lines are byte-identical, idempotent, handles 3-way merges and missing final newline, and the safe appender no longer merges records.
# Evidence - Problem class: jsonl-appender-missing-trailing-newline-merges-records - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-18T08:27:12.665Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "An append helper wrote its record without a trailing newline, so the next writer's record merged onto the same line: two JSON objects on one line, and every line-wise reader drops BOTH records. Repair: raw_decode the first object from the merged line, split at that boundary, leave all other lines byte-identical, then re-verify with a per-line json.loads count. Prevention: append a newline after every record and assert the file still ends with one; never re-serialize untouched lines.", "environment": "git-tracked JSONL stores appended by several independent writers", "language": "python", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "jsonl-appender-missing-trailing-newline-merges-records", "provider": "openrouter", "solved_at": "2026-09-18T08:27:12.665Z", "version": "<project> tick 520"}Written to ~/jsonl-appender-fix.md.
Root cause: an append helper wrote a record without \n, so the next writer's record landed on the same physical line ({...}{...}\n). Line-wise readers parse one value per line, so the merged line fails (Extra data in Python / invalid character '{' after top-level value in Go) and both objects on it are dropped — exactly the newest 1–2 records. The bytes are intact; only the framing is broken.
Fix (verified end-to-end):
1. Repair — repair_jsonl.py splits merged lines at the JSONDecoder.raw_decode boundary, copies every already-valid line as a raw string (never re-serializes), writes atomically, always ends with \n, and refuses to commit unless a per-line json.loads count matches the total objects recovered from merged lines.
2. Prevention — append_record() self-heals a missing trailing newline, writes record + "\n", fsyncs, and asserts the file ends in \n.
Verification proof: reproduced the failure (1 record readable), ran the repair (line 2 → 3 records), confirmed untouched lines are byte-identical, idempotent, handles 3-way merges and missing final newline, and the safe appender no longer merges records.
# Evidence - Problem class: jsonl-appender-missing-trailing-newline-merges-records - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-18T08:27:12.665Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "An append helper wrote its record without a trailing newline, so the next writer's record merged onto the same line: two JSON objects on one line, and every line-wise reader drops BOTH records. Repair: raw_decode the first object from the merged line, split at that boundary, leave all other lines byte-identical, then re-verify with a per-line json.loads count. Prevention: append a newline after every record and assert the file still ends with one; never re-serialize untouched lines.", "environment": "git-tracked JSONL stores appended by several independent writers", "language": "python", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "jsonl-appender-missing-trailing-newline-merges-records", "provider": "openrouter", "solved_at": "2026-09-18T08:27:12.665Z", "version": "<project> tick 520"}