telegram-thread-fallback-precedence
Solution written and verified.
Deliverables
- ~/solution/SOLUTION.md — full self-contained write-up (title, root cause, exact fix, verification).
- ~/solution/test_fallback_precedence.py — executable proof; ALL TESTS PASSED.
What I confirmed
- The sandbox had none of the live paths (~/.hermes, my-project), so I couldn't inspect the real log counts or the original source. Instead I reconstructed the code path from the spec and proved the logic with a mock Telegram harness.
- The pre-fix branch shape raises message thread not found on the double fault (parse error → plain-text resend keeps the dead thread id) — reproduced.
- The loop-based fix recovers: attempt 1 parse error → drop parse_mode; attempt 2 thread not found → drop message_thread_id; attempt 3 succeeds. Also verified a lone dead thread still works and unrelated errors (403) still raise.
Notable finding during verification: the reported MarkdownV2 error can't parse entities: Can't find end of the entity… is only matched by parse because of the can't parse entities: prefix. I kept the real signature markers in the fix and added can't find end of / unsupported start tag defensively.
Fix summary
- C (code): replace the two mutually-exclusive except branches in _telegram_send_text_chunk with a single retry loop tracking thread_id/mode, max_attempts=4; preserve the exact warning strings so existing greps still work. Minimal one-line variant given too.
- B (config): telegram:<chat> instead of telegram:<chat>:<thread> — a reader-visible no-op since the fallback already lands in General. Must be changed in both fleet.toml and the scheduler DB (regeneration can overwrite TOML-only edits).
- A: operator-supplied live topic id; PASS signature is zero thread-fallback warnings.
Live verification is specified as re-runnable commands with explicit PASS signatures (recover on double-fault reports; canary warning count stops growing).
# Evidence - Problem class: telegram-thread-fallback-precedence - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-15T18:17:13.156Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "SYMPTOM (what a tick sees): a project's report leg looks 'intermittent' \u2014 scheduler.log shows 155 ok / 37 fail over 192 deliveries, with ~1 failure/day and long ok runs in between, so every previous investigation closed it as 'transient Telegram flake, NOT a wrong thread id' (the same disposition was applied FOUR times: GAP-013, GAP-014, GAP-018, GAP-043; one of those closures argued '3 ok deliveries prove the thread works again').\n\nROOT CAUSE, part 1 \u2014 the thread id is DEAD and the success is a masked fallback. hermes-agent's standalone Telegram sender retries once WITHOUT message_thread_id on a 'thread not found' Bad Request (tools/send_message_senders.py, _telegram_send_text_chunk), so a dead topic id does NOT break delivery \u2014 the message simply lands in the forum's General topic instead of the project topic. The proof is a 1:1 log pairing, not an inference: `grep -h 'retrying without message_thread_id' ~/.hermes/logs/agent.log*` returns exactly 13 lines, and those 13 timestamps (09-11 06:26 / 18:49 / 21:20, 09-12 07:22 / 13:29 / 19:44 / 22:43, 09-13 07:33 / 14:34 / 23:30, 09-14 08:44 / 15:14 / 23:48) map to the 13 successful my-project deliveries in the same retention window and to ZERO of the failures. Every single delivery attempt fails the thread lookup. Without this pairing, the ok/fail ratio reads as flakiness and the target never gets fixed.\n\nROOT CAUSE, part 2 \u2014 the parse-mode fallback preempts the thread fallback, which is where the hard failures come from. _telegram_send_text_chunk's except block branches: (a) thread-not-found AND message_thread_id still set -> pop the thread id and resend; (b) else, if the error text contains parse/markdown/html -> resend the chunk as plain text. Branch (b) does NOT pop message_thread_id, and a thread-not-found raised by branch (b)'s send is not caught by branch (a) (they are mutually exclusive, one except clause, no loop). So a report that trips BOTH \u2014 a MarkdownV2/HTML parse error AND the dead thread \u2014 fails outright: the plain-text retry keeps the dead id, Telegram answers 'message thread not found', and it propagates out of _send_telegram as 'Telegram send failed: ...'. Every hard failure has exactly that shape: a 'Parse mode MarkdownV2/HTML failed in _send_telegram, falling back to plain text' warning within ~1s of the failed DELIVER line, and NO 'retrying without message_thread_id' line at that timestamp. Examples: 09-15 06:39:33 (MarkdownV2, 'can't find end of code entity at byte offset 3824'), 09-14 21:45:51 (HTML, 'unsupported start tag \"org\"'), 09-13 20:40:51 (HTML). Reports that contain an unbalanced code entity / stray tag are the 1-in-5 that dies.\n\nDIAGNOSIS RECIPE (re-runnable, no Telegram API access needed): (1) count ok vs fail DELIVER lines for the project in scheduler.log (deliver.go:102 = ok, :99 = fail); (2) count 'retrying without message_thread_id' warnings in agent.log* and match them to the ok DELIVER timestamps \u2014 a near-1:1 match means the configured thread id is dead and every delivery is being silently redirected; (3) for each failed delivery, grep the same second in agent.log for a 'Parse mode ... failed in _send_telegram' line; if present, the failure is the fallback-precedence bug, not Telegram flakiness. NOTE: the Bot API has no topic-list method, so you cannot enumerate valid topic ids; getUpdates is also unavailable while the gateway is polling (409 Conflict), so the correct topic id must come from the operator or from createForumTopic.\n\nFIX OPTIONS: (A) operator-supplied live topic id for the project (then a delivery succeeds with NO thread-fallback warning at all \u2014 that is the PASS signature); (B) point the target at the chat with no thread (`telegram:<chat>` instead of `telegram:<chat>:<thread>`) \u2014 deterministic 100% delivery, and behaviorally a no-op for the reader because the fallback already delivers to that same General topic; (C) fix hermes-agent so the parse-mode branch also drops message_thread_id (or wraps both retries in a loop), which removes the hard-failure class even when a dead id remains. Watch out for regeneration: the deliver target can live in both fleet.toml and the scheduler DB, so editing only the TOML may be overwritten by the next config regeneration.\n\nLESSON for future ticks: 'intermittent' delivery failures with a low but nonzero failure rate are usually a masked permanent configuration error plus a second, mutually-exclusive fallback. Pair the sender's WARNING lines against the delivery OK/FAIL lines before believing a flake; a target that only ever succeeds through a fallback is not working, it is degrading quietly. Also: the canary's 2-consecutive-failure threshold (GAP-014) is what eventually forced this diagnosis after four 'transient' dispositions \u2014 a masked-degradation signal (fallback-warning count > 0) would have surfaced it months earlier.\n", "environment": "Linux host running the Hermes gateway + the coding-hermes scheduler; scheduler delivers per-project tick reports with `hermes send telegram:<chat>:<thread>`; logs: ~/.hermes/coding-hermes/scheduler.log (deliver.go:102 ok / :99 fail) and ~/.hermes/logs/agent.log* (tools.send_message_tool warnings).", "language": "shell", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "telegram-thread-fallback-precedence", "provider": "openrouter", "solved_at": "2026-09-15T18:17:13.156Z", "version": "hermes-agent (tools/send_message_senders.py, standalone _send_telegram path); gitreins 0.12.1"}Solution written and verified.
Deliverables
- ~/solution/SOLUTION.md — full self-contained write-up (title, root cause, exact fix, verification).
- ~/solution/test_fallback_precedence.py — executable proof; ALL TESTS PASSED.
What I confirmed
- The sandbox had none of the live paths (~/.hermes, my-project), so I couldn't inspect the real log counts or the original source. Instead I reconstructed the code path from the spec and proved the logic with a mock Telegram harness.
- The pre-fix branch shape raises message thread not found on the double fault (parse error → plain-text resend keeps the dead thread id) — reproduced.
- The loop-based fix recovers: attempt 1 parse error → drop parse_mode; attempt 2 thread not found → drop message_thread_id; attempt 3 succeeds. Also verified a lone dead thread still works and unrelated errors (403) still raise.
Notable finding during verification: the reported MarkdownV2 error can't parse entities: Can't find end of the entity… is only matched by parse because of the can't parse entities: prefix. I kept the real signature markers in the fix and added can't find end of / unsupported start tag defensively.
Fix summary
- C (code): replace the two mutually-exclusive except branches in _telegram_send_text_chunk with a single retry loop tracking thread_id/mode, max_attempts=4; preserve the exact warning strings so existing greps still work. Minimal one-line variant given too.
- B (config): telegram:<chat> instead of telegram:<chat>:<thread> — a reader-visible no-op since the fallback already lands in General. Must be changed in both fleet.toml and the scheduler DB (regeneration can overwrite TOML-only edits).
- A: operator-supplied live topic id; PASS signature is zero thread-fallback warnings.
Live verification is specified as re-runnable commands with explicit PASS signatures (recover on double-fault reports; canary warning count stops growing).
# Evidence - Problem class: telegram-thread-fallback-precedence - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-15T18:17:13.156Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "SYMPTOM (what a tick sees): a project's report leg looks 'intermittent' \u2014 scheduler.log shows 155 ok / 37 fail over 192 deliveries, with ~1 failure/day and long ok runs in between, so every previous investigation closed it as 'transient Telegram flake, NOT a wrong thread id' (the same disposition was applied FOUR times: GAP-013, GAP-014, GAP-018, GAP-043; one of those closures argued '3 ok deliveries prove the thread works again').\n\nROOT CAUSE, part 1 \u2014 the thread id is DEAD and the success is a masked fallback. hermes-agent's standalone Telegram sender retries once WITHOUT message_thread_id on a 'thread not found' Bad Request (tools/send_message_senders.py, _telegram_send_text_chunk), so a dead topic id does NOT break delivery \u2014 the message simply lands in the forum's General topic instead of the project topic. The proof is a 1:1 log pairing, not an inference: `grep -h 'retrying without message_thread_id' ~/.hermes/logs/agent.log*` returns exactly 13 lines, and those 13 timestamps (09-11 06:26 / 18:49 / 21:20, 09-12 07:22 / 13:29 / 19:44 / 22:43, 09-13 07:33 / 14:34 / 23:30, 09-14 08:44 / 15:14 / 23:48) map to the 13 successful my-project deliveries in the same retention window and to ZERO of the failures. Every single delivery attempt fails the thread lookup. Without this pairing, the ok/fail ratio reads as flakiness and the target never gets fixed.\n\nROOT CAUSE, part 2 \u2014 the parse-mode fallback preempts the thread fallback, which is where the hard failures come from. _telegram_send_text_chunk's except block branches: (a) thread-not-found AND message_thread_id still set -> pop the thread id and resend; (b) else, if the error text contains parse/markdown/html -> resend the chunk as plain text. Branch (b) does NOT pop message_thread_id, and a thread-not-found raised by branch (b)'s send is not caught by branch (a) (they are mutually exclusive, one except clause, no loop). So a report that trips BOTH \u2014 a MarkdownV2/HTML parse error AND the dead thread \u2014 fails outright: the plain-text retry keeps the dead id, Telegram answers 'message thread not found', and it propagates out of _send_telegram as 'Telegram send failed: ...'. Every hard failure has exactly that shape: a 'Parse mode MarkdownV2/HTML failed in _send_telegram, falling back to plain text' warning within ~1s of the failed DELIVER line, and NO 'retrying without message_thread_id' line at that timestamp. Examples: 09-15 06:39:33 (MarkdownV2, 'can't find end of code entity at byte offset 3824'), 09-14 21:45:51 (HTML, 'unsupported start tag \"org\"'), 09-13 20:40:51 (HTML). Reports that contain an unbalanced code entity / stray tag are the 1-in-5 that dies.\n\nDIAGNOSIS RECIPE (re-runnable, no Telegram API access needed): (1) count ok vs fail DELIVER lines for the project in scheduler.log (deliver.go:102 = ok, :99 = fail); (2) count 'retrying without message_thread_id' warnings in agent.log* and match them to the ok DELIVER timestamps \u2014 a near-1:1 match means the configured thread id is dead and every delivery is being silently redirected; (3) for each failed delivery, grep the same second in agent.log for a 'Parse mode ... failed in _send_telegram' line; if present, the failure is the fallback-precedence bug, not Telegram flakiness. NOTE: the Bot API has no topic-list method, so you cannot enumerate valid topic ids; getUpdates is also unavailable while the gateway is polling (409 Conflict), so the correct topic id must come from the operator or from createForumTopic.\n\nFIX OPTIONS: (A) operator-supplied live topic id for the project (then a delivery succeeds with NO thread-fallback warning at all \u2014 that is the PASS signature); (B) point the target at the chat with no thread (`telegram:<chat>` instead of `telegram:<chat>:<thread>`) \u2014 deterministic 100% delivery, and behaviorally a no-op for the reader because the fallback already delivers to that same General topic; (C) fix hermes-agent so the parse-mode branch also drops message_thread_id (or wraps both retries in a loop), which removes the hard-failure class even when a dead id remains. Watch out for regeneration: the deliver target can live in both fleet.toml and the scheduler DB, so editing only the TOML may be overwritten by the next config regeneration.\n\nLESSON for future ticks: 'intermittent' delivery failures with a low but nonzero failure rate are usually a masked permanent configuration error plus a second, mutually-exclusive fallback. Pair the sender's WARNING lines against the delivery OK/FAIL lines before believing a flake; a target that only ever succeeds through a fallback is not working, it is degrading quietly. Also: the canary's 2-consecutive-failure threshold (GAP-014) is what eventually forced this diagnosis after four 'transient' dispositions \u2014 a masked-degradation signal (fallback-warning count > 0) would have surfaced it months earlier.\n", "environment": "Linux host running the Hermes gateway + the coding-hermes scheduler; scheduler delivers per-project tick reports with `hermes send telegram:<chat>:<thread>`; logs: ~/.hermes/coding-hermes/scheduler.log (deliver.go:102 ok / :99 fail) and ~/.hermes/logs/agent.log* (tools.send_message_tool warnings).", "language": "shell", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "telegram-thread-fallback-precedence", "provider": "openrouter", "solved_at": "2026-09-15T18:17:13.156Z", "version": "hermes-agent (tools/send_message_senders.py, standalone _send_telegram path); gitreins 0.12.1"}