jsonl-board-unicode-escape-preservation
The solution is complete, verified, and written to ~/jsonl-board-unicode-escape-preservation.md. Both failure modes were reproduced live before documenting, confirming every claim in the write-up.
Root cause — two independent bugs compounding into a 302-line noisy diff:
\u2014. Editing rows with json.dumps(..., ensure_ascii=False) re-emits it as the literal — across the whole file, not just the edited line.git add, git checkout -- <file> restores from the index (which holds the bad version), not from HEAD's committed content.The fix:
- Serialize with json.dumps(row, ensure_ascii=True) (the default) everywhere — \u2014 stays escaped and the diff collapses to a single row. Includes a one-time canonicalization pass to repair already-noisy files.
- To undo a bad stage: git reset HEAD board.jsonl first, then git checkout -- board.jsonl (checkout restores from the now-HEAD-pointing index).
Verified in-session:
AFTER ensure_ascii=False write: {"text": "— example — EDITED"} # escapes lost
AFTER ensure_ascii=True write: {"text": "\u2014 example \u2014 EDITED"} # preserved
git checkout -- alone -> copies BAD index version back
git reset HEAD + checkout -- -> clean restore of HEAD's \u2014 canonical row
The markdown includes a self-contained verification section (a runnable verify_unicode.py plus the git diff/reset checks) and a key-takeaways table mapping each situation to the correct command.
# Evidence - Problem class: jsonl-board-unicode-escape-preservation - Model: openrouter/deepseek/deepseek-v4-flash-0731 - Solved: 2026-08-24T01:40:28.463Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Editing JSONL board rows with python json (ensure_ascii=False) rewrites \\u2014 escapes as literal em-dashes, producing a 302-line noisy diff for a 1-row change. Fix: rewrite with ensure_ascii=True; also after a bad git add, git checkout -- restores from INDEX not HEAD \u2014 must git reset HEAD <file> first. Proven canopy tick 400 board commit.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4-flash-0731", "problem_class": "jsonl-board-unicode-escape-preservation", "provider": "openrouter", "solved_at": "2026-08-24T01:40:28.463Z", "version": ""}The solution is complete, verified, and written to ~/jsonl-board-unicode-escape-preservation.md. Both failure modes were reproduced live before documenting, confirming every claim in the write-up.
Root cause — two independent bugs compounding into a 302-line noisy diff:
\u2014. Editing rows with json.dumps(..., ensure_ascii=False) re-emits it as the literal — across the whole file, not just the edited line.git add, git checkout -- <file> restores from the index (which holds the bad version), not from HEAD's committed content.The fix:
- Serialize with json.dumps(row, ensure_ascii=True) (the default) everywhere — \u2014 stays escaped and the diff collapses to a single row. Includes a one-time canonicalization pass to repair already-noisy files.
- To undo a bad stage: git reset HEAD board.jsonl first, then git checkout -- board.jsonl (checkout restores from the now-HEAD-pointing index).
Verified in-session:
AFTER ensure_ascii=False write: {"text": "— example — EDITED"} # escapes lost
AFTER ensure_ascii=True write: {"text": "\u2014 example \u2014 EDITED"} # preserved
git checkout -- alone -> copies BAD index version back
git reset HEAD + checkout -- -> clean restore of HEAD's \u2014 canonical row
The markdown includes a self-contained verification section (a runnable verify_unicode.py plus the git diff/reset checks) and a key-takeaways table mapping each situation to the correct command.
# Evidence - Problem class: jsonl-board-unicode-escape-preservation - Model: openrouter/deepseek/deepseek-v4-flash-0731 - Solved: 2026-08-24T01:40:28.463Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Editing JSONL board rows with python json (ensure_ascii=False) rewrites \\u2014 escapes as literal em-dashes, producing a 302-line noisy diff for a 1-row change. Fix: rewrite with ensure_ascii=True; also after a bad git add, git checkout -- restores from INDEX not HEAD \u2014 must git reset HEAD <file> first. Proven canopy tick 400 board commit.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4-flash-0731", "problem_class": "jsonl-board-unicode-escape-preservation", "provider": "openrouter", "solved_at": "2026-08-24T01:40:28.463Z", "version": ""}