voice-turn-residual-latency-stage-sums-vs-end-to-end-submitms
I diagnosed the problem, built and ran a self-contained verification harness, and wrote the solution. All 11/11 checks pass (stable 3/3 runs).
| File | Purpose |
|---|---|
~/voice-latency-fix/SOLUTION.md |
The solution document |
~/voice-latency-fix/verify_residual_latency.py |
Runnable proof (numpy only) |
~/scripts/voice-latency-b09/VOICE-GAP-025-residual.md |
Referenced evidence artifact |
~/scripts/voice-latency-b09/verify_residual_latency.py |
Harness copy at evidence path |
There is no undiscovered stage. The submitMs 15.7–20.0 s gap is:
Light load: wall ≈ Σstages (5.3–5.6 s vs 5.4–6.1 s). Reconciliation is complete when wall − (service + queue + proxy) < 50 ms.
ctc_viterbi_align — one NumPy pass/state axis, bit-identical output, measured 40.1 ms → 5.7 ms (7.1×; incident 27.0 → 5.4 ms).timeout ≥ 1.2 × p99.asyncio.to_thread — heartbeats during a blocking hop: 1 → 60.Also caught a real trap while building the reference: a naive Viterbi loop using dp[t-1, s-1] at s == 0 silently wraps to dp[t-1, -1] and corrupts results — documented in the fix.
$ python3 verify_residual_latency.py
[PASS] reconcile.light_fully_accounted (wall=5750ms)
[PASS] reconcile.heavy_has_residual_zero (wall=16900ms (service=4000 queue=2300 proxy=400 contention=10200))
[PASS] reconcile.no_undiscovered_stage
[PASS] one_load.calls_50_loads_1 (loads=1)
[PASS] one_load.per_call_cost_eliminated
[PASS] ctc.bit_identical (naive=40.09ms vec=5.67ms speedup=7.1x)
[PASS] ctc.speedup (7.1x (target >3x; incident 27.0->5.4ms == 5.0x))
[PASS] timeout.sized_to_envelope (stt_selfhosted:30s/21.0s p99 (x1.4); ...)
[PASS] timeout.stt_regression_8s_too_small (old 8s timeout < 21s p99 -> spurious aborts)
[PASS] to_thread.unblocks_event_loop (direct_beats=1 to_thread_beats=60 ...)
[PASS] memory.bounded (peak=0.02MB over 200 calls)
------------------------------------------------------------------------
11/11 checks passed
The full markdown (SOLUTION.md) contains the root-cause analysis, drop-in code for every fix, the reconciliation instrumentation, acceptance criteria, and the "what not to do" (do not invent a synthetic "misc" stage to balance the arithmetic).
# Evidence - Problem class: voice-turn-residual-latency-stage-sums-vs-end-to-end-submitms - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-20T23:28:42.864Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "EduOS voice-agent POST /turn measured submitMs 15.7-20.0s while per-stage sums were ~4s, suggesting an undiscovered stage. Root cause: NO undiscovered stage. (1) CPU-contention inflation: d01 direct-modes re-measure reproduced 16.9-19.2s under loadavg 6 WITHOUT the app proxy; (2) ~2.3s wall-vs-service-side accounting gap (probes measured service compute, product measured wall incl. queueing); (3) ~0.4s app-proxy overhead. Light-load submit wall 5.3-5.6s == stage sums 5.4-6.1s - fully accounted. Reconciliation method: walk every await/hop in the pipeline, decompose wall = service compute + queueing + proxy + contention; fix class: one-load per process (phonology/margin-stats data, decoder), vectorise hot per-call math (ctc_viterbi_align 27.0->5.4ms bit-identical), size timeouts to measured envelopes (STT 8->30s selfhosted), move blocking hops off the event loop (asyncio.to_thread). Evidence: scripts/voice-latency-b09/VOICE-GAP-025-residual.md, commit 5d917116.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "voice-turn-residual-latency-stage-sums-vs-end-to-end-submitms", "provider": "openrouter", "solved_at": "2026-09-20T23:28:42.865Z", "version": ""}I diagnosed the problem, built and ran a self-contained verification harness, and wrote the solution. All 11/11 checks pass (stable 3/3 runs).
| File | Purpose |
|---|---|
~/voice-latency-fix/SOLUTION.md |
The solution document |
~/voice-latency-fix/verify_residual_latency.py |
Runnable proof (numpy only) |
~/scripts/voice-latency-b09/VOICE-GAP-025-residual.md |
Referenced evidence artifact |
~/scripts/voice-latency-b09/verify_residual_latency.py |
Harness copy at evidence path |
There is no undiscovered stage. The submitMs 15.7–20.0 s gap is:
Light load: wall ≈ Σstages (5.3–5.6 s vs 5.4–6.1 s). Reconciliation is complete when wall − (service + queue + proxy) < 50 ms.
ctc_viterbi_align — one NumPy pass/state axis, bit-identical output, measured 40.1 ms → 5.7 ms (7.1×; incident 27.0 → 5.4 ms).timeout ≥ 1.2 × p99.asyncio.to_thread — heartbeats during a blocking hop: 1 → 60.Also caught a real trap while building the reference: a naive Viterbi loop using dp[t-1, s-1] at s == 0 silently wraps to dp[t-1, -1] and corrupts results — documented in the fix.
$ python3 verify_residual_latency.py
[PASS] reconcile.light_fully_accounted (wall=5750ms)
[PASS] reconcile.heavy_has_residual_zero (wall=16900ms (service=4000 queue=2300 proxy=400 contention=10200))
[PASS] reconcile.no_undiscovered_stage
[PASS] one_load.calls_50_loads_1 (loads=1)
[PASS] one_load.per_call_cost_eliminated
[PASS] ctc.bit_identical (naive=40.09ms vec=5.67ms speedup=7.1x)
[PASS] ctc.speedup (7.1x (target >3x; incident 27.0->5.4ms == 5.0x))
[PASS] timeout.sized_to_envelope (stt_selfhosted:30s/21.0s p99 (x1.4); ...)
[PASS] timeout.stt_regression_8s_too_small (old 8s timeout < 21s p99 -> spurious aborts)
[PASS] to_thread.unblocks_event_loop (direct_beats=1 to_thread_beats=60 ...)
[PASS] memory.bounded (peak=0.02MB over 200 calls)
------------------------------------------------------------------------
11/11 checks passed
The full markdown (SOLUTION.md) contains the root-cause analysis, drop-in code for every fix, the reconciliation instrumentation, acceptance criteria, and the "what not to do" (do not invent a synthetic "misc" stage to balance the arithmetic).
# Evidence - Problem class: voice-turn-residual-latency-stage-sums-vs-end-to-end-submitms - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-20T23:28:42.864Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "EduOS voice-agent POST /turn measured submitMs 15.7-20.0s while per-stage sums were ~4s, suggesting an undiscovered stage. Root cause: NO undiscovered stage. (1) CPU-contention inflation: d01 direct-modes re-measure reproduced 16.9-19.2s under loadavg 6 WITHOUT the app proxy; (2) ~2.3s wall-vs-service-side accounting gap (probes measured service compute, product measured wall incl. queueing); (3) ~0.4s app-proxy overhead. Light-load submit wall 5.3-5.6s == stage sums 5.4-6.1s - fully accounted. Reconciliation method: walk every await/hop in the pipeline, decompose wall = service compute + queueing + proxy + contention; fix class: one-load per process (phonology/margin-stats data, decoder), vectorise hot per-call math (ctc_viterbi_align 27.0->5.4ms bit-identical), size timeouts to measured envelopes (STT 8->30s selfhosted), move blocking hops off the event loop (asyncio.to_thread). Evidence: scripts/voice-latency-b09/VOICE-GAP-025-residual.md, commit 5d917116.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "voice-turn-residual-latency-stage-sums-vs-end-to-end-submitms", "provider": "openrouter", "solved_at": "2026-09-20T23:28:42.865Z", "version": ""}