File: .coding-hermes/board/events.jsonl Repo: hilo/<project> Tool: appendboardtaskcompleted.py
I diagnosed, reproduced the exact failure, implemented a repair, and verified prevention. Files written: ~/solution.md, ~/repair_board_jsonl.py, ~/verify_repair.py.
board-jsonl-torn-append-no-trailing-newline)File: .coding-hermes/board/events.jsonl Repo: hilo/<project> Tool: append_board_task_completed.py
Symptom: json.decoder.JSONDecodeError: Extra data: line 1 column 1679 (char 1678)
A concurrent board lane appended an event object without a trailing newline. The next appender wrote its object immediately after the first, so two complete JSON objects ended up on a single physical line:
{"id":41,...,"note":"xxx..."}{"id":42,...,"note":"xxx..."}
json.loads() parses the first object, then finds the second { at the same line and raises Extra data at the offset where the first object ends. No data was corrupted — both objects are intact; only the line separator is missing, so ids remain monotonic once split. The cause is a non-atomic append race: writers did open(path,"a"); f.write(json.dumps(ev)) with no terminating "\n" and no lock.
repair_board_jsonl.py splits every physical line into one or more JSON values, validates each with json.loads, checks ids stay monotonic, and rewrites atomically.
def split_concatenated_objects(line: str) -> list[str]:
parts, depth, in_str, esc, start = [], 0, False, False, None
for i, ch in enumerate(line):
if in_str:
if esc: esc = False
elif ch == "\\": esc = True
elif ch == '"': in_str = False
continue
if ch == '"':
in_str = True
if start is None: start = i
elif ch in "{[": # OPEN
if start is None: start = i
depth += 1
elif ch in "}]": # CLOSE
depth -= 1
if depth < 0:
raise ValueError(f"unbalanced {ch!r} at column {i}")
if depth == 0 and start is not None:
parts.append(line[start:i + 1].strip())
start = None
if in_str or depth != 0:
raise ValueError(f"truncated/unbalanced JSON: {line[:80]!r}...")
return parts
The script also validates each piece, checks monotonic ids, and writes with os.replace(tmp, path) so readers never see a half-written file. Object bytes are preserved verbatim — nothing is lost.
python3 repair_board_jsonl.py .coding-hermes/board/events.jsonl # dry report
python3 repair_board_jsonl.py .coding-hermes/board/events.jsonl --write # atomic repair
import fcntl, json, os
def append_event(path: str, event: dict) -> None:
line = json.dumps(event, separators=(",", ":"), ensure_ascii=False) + "\n"
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o644)
try:
fcntl.flock(fd, fcntl.LOCK_EX) # serialize lanes
os.write(fd, line.encode("utf-8")) # single write of a complete line
os.fsync(fd)
finally:
fcntl.flock(fd, fcntl.LOCK_UN)
os.close(fd)
O_APPEND + a single os.write of the full ...\n makes the append atomic on POSIX, and flock removes cross-process races.
python3 - <<'PY'
import json, sys
path = ".coding-hermes/board/events.jsonl"
bad = 0
for n, line in enumerate(open(path, encoding="utf-8"), 1):
if line.strip():
try:
json.loads(line)
except json.JSONDecodeError as e:
print(f"{path}:{n}: {e}"); bad += 1
sys.exit(1 if bad else 0)
PY
Wire this into CI / a post-merge hook so a torn line is caught the moment a foreign lane writes it.
verify_repair.py builds a first object exactly 1678 bytes long and appends a second with no newline, reproducing the reported error byte-for-byte, then repairs and re-validates:
$ python3 verify_repair.py
[repro] JSONDecodeError: Extra data: line 1 column 1679 (char 1678)
[unit] brace/quote-in-string handling OK
[repair] split torn lines=1, ids=[40, 41, 42] (monotonic), all lines valid
[cli] lines in: 2 torn lines: 1 events out: 3
rewrote /tmp/tmp.../events.jsonl
[cli] 3 newline-terminated valid JSON lines, no content lost
ALL CHECKS PASSED
Additionally, the prevention appender was stress-tested with 8 threads × 100 concurrent appends:
[prevent] 8 threads x100 concurrent appends -> 800 valid JSON lines, 800 unique ids, no torn lines
Verified: exact fault reproduced, string-aware splitting ({"a": "}{ \" }"}{"b": 1} → two values), no events lost ([40,41,42], monotonic), every line strict-parseable, file newline-terminated, and concurrent locked appends never tear.
# Evidence - Problem class: board-jsonl-torn-append-no-trailing-newline - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-23T03:20:53.274Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Concurrent board lane appends an event object without a trailing newline; the next append concatenates both objects on ONE events.jsonl line and every strict per-line JSON parser dies with JSONDecodeError Extra data. Repair: brace-depth scanner splits the line; no content lost; ids stay monotonic. Prevention: appenders must write newline-terminated lines; run a json-per-line check after foreign lanes touch the board.", "environment": "", "language": "python", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "board-jsonl-torn-append-no-trailing-newline", "provider": "openrouter", "solved_at": "2026-09-23T03:20:53.275Z", "version": ""}I diagnosed, reproduced the exact failure, implemented a repair, and verified prevention. Files written: ~/solution.md, ~/repair_board_jsonl.py, ~/verify_repair.py.
board-jsonl-torn-append-no-trailing-newline)File: .coding-hermes/board/events.jsonl Repo: hilo/<project> Tool: append_board_task_completed.py
Symptom: json.decoder.JSONDecodeError: Extra data: line 1 column 1679 (char 1678)
A concurrent board lane appended an event object without a trailing newline. The next appender wrote its object immediately after the first, so two complete JSON objects ended up on a single physical line:
{"id":41,...,"note":"xxx..."}{"id":42,...,"note":"xxx..."}
json.loads() parses the first object, then finds the second { at the same line and raises Extra data at the offset where the first object ends. No data was corrupted — both objects are intact; only the line separator is missing, so ids remain monotonic once split. The cause is a non-atomic append race: writers did open(path,"a"); f.write(json.dumps(ev)) with no terminating "\n" and no lock.
repair_board_jsonl.py splits every physical line into one or more JSON values, validates each with json.loads, checks ids stay monotonic, and rewrites atomically.
def split_concatenated_objects(line: str) -> list[str]:
parts, depth, in_str, esc, start = [], 0, False, False, None
for i, ch in enumerate(line):
if in_str:
if esc: esc = False
elif ch == "\\": esc = True
elif ch == '"': in_str = False
continue
if ch == '"':
in_str = True
if start is None: start = i
elif ch in "{[": # OPEN
if start is None: start = i
depth += 1
elif ch in "}]": # CLOSE
depth -= 1
if depth < 0:
raise ValueError(f"unbalanced {ch!r} at column {i}")
if depth == 0 and start is not None:
parts.append(line[start:i + 1].strip())
start = None
if in_str or depth != 0:
raise ValueError(f"truncated/unbalanced JSON: {line[:80]!r}...")
return parts
The script also validates each piece, checks monotonic ids, and writes with os.replace(tmp, path) so readers never see a half-written file. Object bytes are preserved verbatim — nothing is lost.
python3 repair_board_jsonl.py .coding-hermes/board/events.jsonl # dry report
python3 repair_board_jsonl.py .coding-hermes/board/events.jsonl --write # atomic repair
import fcntl, json, os
def append_event(path: str, event: dict) -> None:
line = json.dumps(event, separators=(",", ":"), ensure_ascii=False) + "\n"
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o644)
try:
fcntl.flock(fd, fcntl.LOCK_EX) # serialize lanes
os.write(fd, line.encode("utf-8")) # single write of a complete line
os.fsync(fd)
finally:
fcntl.flock(fd, fcntl.LOCK_UN)
os.close(fd)
O_APPEND + a single os.write of the full ...\n makes the append atomic on POSIX, and flock removes cross-process races.
python3 - <<'PY'
import json, sys
path = ".coding-hermes/board/events.jsonl"
bad = 0
for n, line in enumerate(open(path, encoding="utf-8"), 1):
if line.strip():
try:
json.loads(line)
except json.JSONDecodeError as e:
print(f"{path}:{n}: {e}"); bad += 1
sys.exit(1 if bad else 0)
PY
Wire this into CI / a post-merge hook so a torn line is caught the moment a foreign lane writes it.
verify_repair.py builds a first object exactly 1678 bytes long and appends a second with no newline, reproducing the reported error byte-for-byte, then repairs and re-validates:
$ python3 verify_repair.py
[repro] JSONDecodeError: Extra data: line 1 column 1679 (char 1678)
[unit] brace/quote-in-string handling OK
[repair] split torn lines=1, ids=[40, 41, 42] (monotonic), all lines valid
[cli] lines in: 2 torn lines: 1 events out: 3
rewrote /tmp/tmp.../events.jsonl
[cli] 3 newline-terminated valid JSON lines, no content lost
ALL CHECKS PASSED
Additionally, the prevention appender was stress-tested with 8 threads × 100 concurrent appends:
[prevent] 8 threads x100 concurrent appends -> 800 valid JSON lines, 800 unique ids, no torn lines
Verified: exact fault reproduced, string-aware splitting ({"a": "}{ \" }"}{"b": 1} → two values), no events lost ([40,41,42], monotonic), every line strict-parseable, file newline-terminated, and concurrent locked appends never tear.
# Evidence - Problem class: board-jsonl-torn-append-no-trailing-newline - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-23T03:20:53.274Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Concurrent board lane appends an event object without a trailing newline; the next append concatenates both objects on ONE events.jsonl line and every strict per-line JSON parser dies with JSONDecodeError Extra data. Repair: brace-depth scanner splits the line; no content lost; ids stay monotonic. Prevention: appenders must write newline-terminated lines; run a json-per-line check after foreign lanes touch the board.", "environment": "", "language": "python", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "board-jsonl-torn-append-no-trailing-newline", "provider": "openrouter", "solved_at": "2026-09-23T03:20:53.275Z", "version": ""}