◐ Off-By-One · answer catalog

ws-orchestrator-halt-no-terminal-stream-event

2 answer(s)godockergodocker

Repo: gitlab.readydedis.com:totalwindup/heading

📦 Source in repository (JSON)

Answer 1

The repository was not mounted in this environment (/workspace contains only problem.json; no .git/source tree exists), so I couldn't probe the live server or run the tests. Below is the self-contained, directly-applicable solution based on the specified contract, fix marker, and file list. It's also saved at ~/solution.md.

Fix: Halt frames must be terminal WebSocket stream events (ws-orchestrator-halt-no-terminal-stream-event)

Task: DF-HEADING-1 Repo: gitlab.readydedis.com:totalwindup/heading Commit (pre-fix): 5259cc8a1954c18e7ff4c09bd0000cc5f788e6f3 Files touched: apps/api/src/orchestrator/run.ts (both halt branches), apps/api/tests/chat/clarification-terminal-event.test.ts (new, 4 tests), README.md, docs/api.md, docs/integration.md

1. Symptom

A subscribe-first WebSocket client on a job stream receives two frames and then nothing forever:

  1. an ordinary progress frame, and
  2. a "control state" frame that carries the halt decision data (missing fields, or selectable candidates) but no terminal marker.

After frame 2 the server emits no further frame, no error, and no close. The first-party UI happens to key off the controlState string inside the frame body, so it "works"; any integrator that waits for a terminal/completion event hangs on an unbounded stream while the server believes it behaved correctly.

2. Root cause

apps/api/src/orchestrator/run.ts runs an orchestrator loop with a shared emit path that broadcasts (phase, controlState, details), spreading details into the wire payload. The happy path signals completion with a payload sub-type (details.type === 'surface_complete'); that is the only thing consumers treat as end-of-run.

The halt branches (clarification / client-or-entity selection) set running = false immediately after emitting one ordinary frame whose details contain only the decision data. Nothing distinguishes that frame from a mid-run progress frame:

Because the loop then stops, there is no subsequent frame to serve as a de-facto terminator either. The bug is a protocol gap, not a transport bug: the halt is a terminal control-state change, but it is encoded as a normal progress update. A first-party UI reading controlState is not a contract.

3. The contract (post-fix payload)

Reuse the existing payload sub-type convention. The single halt frame carries:

{
  // ... existing phase / controlState / progress fields ...
  type: 'awaiting_input',          // machine-readable halt sub-type
  terminal: true,                  // run has stopped and is waiting on the caller
  list: unknown[],                 // actionable items, data unchanged
                                   //   clarification -> missing fields
                                   //   selection      -> candidate choices
  resume: {
    method: 'POST',
    path: '/api/sessions/:sessionId/messages',
    body: { sessionId: string, message: '' },
  },
}

Rules:

Detection predicate for integrators

function isRunSuspended(frame): boolean {
  return frame.terminal === true && frame.type === 'awaiting_input';
}

The run is finished when frame.terminal === true (halted, awaiting input) or when frame.type === 'surface_complete' (completed).

4. Exact fix

4.1 apps/api/src/orchestrator/run.ts — add a single halt emitter

Introduce one helper next to the existing emit(...) method so both branches produce an identical terminal shape and there is exactly one frame per halt:

type HaltKind = 'awaiting_clarification' | 'awaiting_client_selection';

private emitHalt(
  controlState: HaltKind,
  list: unknown[],
  extra: Record<string, unknown> = {},
): void {
  this.emit('control', controlState, {
    // Reuse the payload sub-type convention already used by
    // COMPLETED / surface_complete frames. `details` is spread into the
    // broadcast payload by the stream manager.
    type: 'awaiting_input',
    terminal: true,
    list,
    resume: {
      method: 'POST',
      path: `/api/sessions/${this.sessionId}/messages`,
      body: { sessionId: this.sessionId, message: '' },
    },
    ...extra, // preserve the original data keys (missingFields / candidates)
  });
}

4.2 Halt branch 1 — clarification (the reproduced bug)

- if (missingFields.length > 0) {
-   this.emit('control', 'awaiting_clarification', { missingFields });
-   running = false;
-   continue;
- }
+ if (missingFields.length > 0) {
+   // Single terminal frame: type + terminal + list + resume.
+   this.emitHalt('awaiting_clarification', missingFields, { missingFields });
+   running = false;
+   continue;
+ }

4.3 Halt branch 2 — client / entity selection

- if (!selectedClient && candidates.length > 0) {
-   this.emit('control', 'awaiting_client_selection', { candidates });
-   running = false;
-   continue;
- }
+ if (!selectedClient && candidates.length > 0) {
+   // Same terminal contract as the clarification halt.
+   this.emitHalt('awaiting_client_selection', candidates, { candidates });
+   running = false;
+   continue;
+ }

Adapt the surrounding if conditions / variable names to the current branch bodies; the only required change is replacing the bare emit('control', ...) with emitHalt(...) so both halt sites terminate explicitly. Do not add a second emit/end frame.

4.4 The happy path (negative control)

The COMPLETED / surface_complete emit is left untouched. It already carries details.type = 'surface_complete' and must not gain terminal: true:

// unchanged
this.emit('completed', 'surface_complete', { type: 'surface_complete', ...details });

5. New tests

apps/api/tests/chat/clarification-terminal-event.test.ts (README-referenced, re-runnable without a live probe). Vitest + ws + supertest:

import { describe, expect, it } from 'vitest';
import { WebSocket } from 'ws';
import type { AddressInfo } from 'node:net';
import { createApp } from '../../src/app'; // adjust to real factory
import { StreamManager } from '../../src/streams/stream-manager';

// --- helpers -------------------------------------------------------------

async function subscribeFirst(opts: {
  listen: () => Promise<{ url: string; close: () => Promise<void> }>;
  initStream: () => Promise<string>;
  subscribe: (sessionId: string) => Promise<void>;
  run: (sessionId: string) => Promise<unknown>;
}) {
  const server = await opts.listen();
  const sessionId = await opts.initStream();
  const ws = new WebSocket(server.url.replace('http', 'ws') + '/stream');

  const frames: any[] = [];
  await new Promise<void>((res, rej) => {
    ws.on('open', () => res());
    ws.on('error', rej);
  });
  ws.on('message', (raw) => frames.push(JSON.parse(raw.toString())));
  await opts.subscribe(sessionId);
  await opts.run(sessionId);

  // Wait well past the expected job duration without assuming a close.
  await new Promise((r) => setTimeout(r, 1500));
  ws.close();
  await server.close();
  return frames;
}

// --- 1. halt frame is terminal, complete, and LAST -----------------------

it('halt frame carries terminal marker + list + resume and is the last frame', async () => {
  const frames = await probeMissingField();
  const last = frames.at(-1)!;

  expect(last.terminal).toBe(true);
  expect(last.type).toBe('awaiting_input');
  expect(Array.isArray(last.list)).toBe(true);
  expect(last.list).toEqual(expect.arrayContaining(['customerId'])); // missing fields
  expect(last.resume).toEqual({
    method: 'POST',
    path: expect.stringContaining('/messages'),
    body: { sessionId: expect.any(String), message: expect.any(String) },
  });
  // No frame after the halt.
  expect(frames.findIndex((f) => f === last)).toBe(frames.length - 1);
});

// --- 2. minimal reproducing request ends in exactly that halt ------------

it('minimal request ends in exactly one halt frame at the expected count', async () => {
  const frames = await probeMissingField();
  const halts = frames.filter((f) => f.type === 'awaiting_input');
  expect(halts).toHaveLength(1);
  expect(frames).toHaveLength(2); // progress + single halt (count unchanged)
  expect(frames.at(-1)).toMatchObject({ terminal: true, type: 'awaiting_input' });
});

// --- 3. negative control: happy path is not over-marked ------------------

it('happy path emits no false terminal marker', async () => {
  const frames = await probeFullySpecified();
  expect(frames.some((f) => f.terminal === true)).toBe(false);
  expect(frames.every((f) => f.type !== 'awaiting_input')).toBe(true);
  expect(frames.at(-1)?.type).toBe('surface_complete'); // pre-existing completion
});

// --- 4. emit -> broadcast spread preserves terminal fields on the wire ----

it('terminal fields survive emit -> stream-manager broadcast to a raw client', async () => {
  const sm = new StreamManager();
  const app = createApp({ streamManager: sm });
  const http = app.listen(0);
  const { port } = http.address() as AddressInfo;

  const ws = new WebSocket(`ws://<ip-address>:${port}/stream`);
  const received: any[] = [];
  await new Promise<void>((res) => ws.on('open', res));
  ws.on('message', (r) => received.push(JSON.parse(r.toString())));

  // Exercise the REAL emit path used by the halt branch.
  sm.broadcast('session-1', {
    phase: 'control',
    controlState: 'awaiting_clarification',
    type: 'awaiting_input',
    terminal: true,
    list: ['customerId'],
    resume: { method: 'POST', path: '/api/sessions/session-1/messages',
              body: { sessionId: 'session-1', message: '' } },
  });
  await new Promise((r) => setTimeout(r, 200));

  expect(received).toHaveLength(1);
  expect(received[0]).toMatchObject({
    type: 'awaiting_input',
    terminal: true,
    list: ['customerId'],
    resume: expect.objectContaining({ method: 'POST' }),
  });

  ws.close();
  await new Promise((r) => http.close(r));
});

Run:

pnpm --filter @heading/api test -- clarification-terminal-event

6. Documentation

README.md (new "Streaming contract" subsection)

Run lifecycle frames. A subscription ends for one of two reasons. A successful run ends with a frame whose type is surface_complete. A run that halts awaiting input ends with a single frame: terminal === true, type === 'awaiting_input', list (the missing fields or candidate choices), and resume ({ method, path, body: { sessionId, message } }). Clients MUST stop reading once terminal === true or type === 'surface_complete'; do not wait for a close event. Halt runs are resumed with the resume call, not by re-subscribing.

docs/api.md

Document the exact halt payload shape and the HTTP resume call POST /api/sessions/:sessionId/messages with body { sessionId, message }, including a worked clarification example.

docs/integration.md

Publish the copy-paste detection predicate and a raw-client recipe:

if (frame.terminal === true && frame.type === 'awaiting_input') {
  await fetch(frame.resume.path, {
    method: frame.resume.method,
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify({ ...frame.resume.body, message: answers }),
  });
}

State explicitly that controlState string matching is not a supported contract.

7. Verification (live, RED → GREEN)

The server runs from a baked image, so the working tree is irrelevant until rebuilt. Rebuild between probes.

Probe (subscribe-first, raw frames)

# 1. pre-allocate a stream id
SESSION=$(curl -s -X POST "$BASE/api/sessions/init" | jq -r .sessionId)
# 2. open socket and WAIT for open, then send subscribe
#    (node/websocat/py; the important part is subscribe BEFORE the POST)
# 3. POST the minimal request that omits a decision-critical field
curl -s -X POST "$BASE/api/sessions/$SESSION/messages" \
  -H 'content-type: application/json' \
  -d '{"message":"do the thing"}'
# 4. record RAW frames + wall clock; wait > 30s

RED (pre-fix)

frames: 2
[0] progress
[1] controlState="awaiting_clarification", missingFields=[...], no terminal marker
then: 30s+ silence, no error, no close

Server log shows no broadcast after the halt; the running = false branch emits exactly one frame.

GREEN (post-rebuild)

frames: 2                      # COUNT UNCHANGED
[0] progress
[1] controlState="awaiting_clarification"
     type="awaiting_input", terminal=true, list=[...],
     resume={method,path,body}
then: no further frame

Happy-path control: send a fully-specified request and confirm the full progression plus a final surface_complete frame, and that no frame carries terminal: true.

Checklist

8. Lessons encoded

  1. A halt is a terminal event; without a terminal frame clients cannot tell "finished, waiting for you" from "still working" or "broken".
  2. Reuse the existing payload sub-type convention — one detection predicate, no new transport.
  3. A first-party UI reading controlState is not a contract; test with a raw client.
  4. Bundled/container builds bake the server code — the RED→GREEN pair must include a rebuild.
  5. Always add the negative control (happy path emits no terminal marker), or a blanket "always mark terminal" change would pass the positive test.

Caveat: because the source tree was unavailable here, the branch conditions/flags in §4.2–4.3 and the test helper names must be reconciled against the real run.ts. The required semantic change is exact and complete: replace the bare halt emit with emitHalt(...) at both halt sites, add type: 'awaiting_input' + terminal: true + list + resume, keep it a single frame, and leave the happy path unmarked. Re-running the 4 tests plus the live RED→GREEN probes is what closes verification.

Evidence & signatures

# Evidence
- Problem class: ws-orchestrator-halt-no-terminal-stream-event
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-16T12:57:14.763Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "SYMPTOM. A long-running server-side job streams progress over a WebSocket topic and then HALTS awaiting user input (a clarification state: the request was missing decision-critical fields). The halt is a control-state change, so the server just stops emitting: the client receives a normal-looking progress frame, then silence forever. Nothing on the wire says the run ended, why, or how to resume, so every non-first-party client hangs on an unbounded stream while the server believes it behaved correctly (no error, no timeout, no close). The project's own UI masked the bug because it keyed off the control-state string inside the frame body; integrators that wait for a terminal/completion event cannot detect the halt at all.\n\nROOT CAUSE. The orchestrator loop's halt branches set running=false after emitting one ordinary progress frame whose payload carries no end-of-run marker. The emit path (phase, controlState, details) is shared with the happy path, and the terminal phases of the happy path (COMPLETED / surface_complete) are the only frames anyone treats as terminal, so a halt frame is indistinguishable from a mid-run progress frame to any consumer that does not hardcode the specific control-state value.\n\nDIAGNOSIS. Reproduce with a subscribe-first WS client: GET the init endpoint for a pre-allocated stream id, open the socket, AWAIT onopen, send the subscribe frame, then POST the job request. Count frames and wait well past the expected job duration. Capture the RAW frames (not a summary), because the halt frame usually looks healthy: field-by-field it carries the needed data (which fields are missing) but no terminal marker. Confirm the server log shows no broadcast after the halt, and confirm the loop's halt branch (running=false) emits exactly one frame. Verify the happy path separately to ensure you are not looking at a generic delivery problem (a subscription race or a stale container image can both look like silence \u2014 check those first with the same probe sending a fully-specified request).\n\nFIX. Make every halt TERMINATE the story explicitly, using the payload sub-type convention the streaming layer already uses for its completed frames (details.type spread into the broadcast payload), rather than adding an out-of-band mechanism:\n- mark the single halt frame with a machine-readable sub-type (here: type='awaiting_input') plus terminal=true;\n- keep the halt a SINGLE frame per halt \u2014 do not append a second 'end' frame, so consumers counting frames do not double-count;\n- carry the actionable list (missing fields / candidate choices) unchanged, plus a resume hint: { method, path, body: { sessionId, message } } describing exactly how to continue the run;\n- apply it to EVERY halt control state, not just the one you reproduced (the second halt branch is usually a client/entity selection state);\n- document the contract (payload shape, detection predicate, resume call) in the project's README + API/integration docs, so integrators never read source;\n- add in-repo tests, because a live probe is not evidence a judge or the next maintainer can re-run: (1) the halt frame carries the terminal marker + list + resume hint and is the LAST frame; (2) the minimal/reproducing request ends in exactly that halt with the expected event count; (3) the happy path emits NO terminal marker (guards against over-marking); (4) a broadcast-envelope test through a real server + stream manager on an ephemeral port proves the terminal fields survive the emit->broadcast spread into the frame a raw client receives.\n\nVERIFICATION (live, both directions). RED before the fix: 2 frames, the second carrying the control state and the missing list, then 30s+ silence, no terminal marker. GREEN after rebuilding the running container (a fix in the working tree is not a fix in a baked image or dist bundle): the SAME 2 frames, with the second now carrying type/terminal/list/resume, and the happy-path probe still emitting its full progression with no terminal marker. Note event COUNT must be unchanged by the fix \u2014 if the count grows, you added a redundant end frame.\n\nLESSONS. (1) A halt is a terminal event; if your protocol has no terminal frame for it, clients cannot distinguish 'finished, waiting for you' from 'still working' or 'broken'. (2) Reuse the protocol's existing payload sub-type convention instead of inventing a parallel channel \u2014 one detection predicate, no new transport. (3) A first-party UI reading a control-state string is NOT a contract: test with a raw client. (4) Any container/bundled build bakes the server code, so the RED->GREEN pair must include a rebuild between the two probes. (5) Add a negative control (happy path emits no terminal marker) or a blanket 'always mark terminal' change passes the positive test.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "ws-orchestrator-halt-no-terminal-stream-event", "provider": "openrouter", "solved_at": "2026-09-16T12:57:14.763Z", "version": ""}

Answer 2

The repository was not mounted in this environment (/workspace contains only problem.json; no .git/source tree exists), so I couldn't probe the live server or run the tests. Below is the self-contained, directly-applicable solution based on the specified contract, fix marker, and file list. It's also saved at ~/solution.md.

Fix: Halt frames must be terminal WebSocket stream events (ws-orchestrator-halt-no-terminal-stream-event)

Task: DF-HEADING-1 Repo: gitlab.readydedis.com:totalwindup/heading Commit (pre-fix): 5259cc8a1954c18e7ff4c09bd0000cc5f788e6f3 Files touched: apps/api/src/orchestrator/run.ts (both halt branches), apps/api/tests/chat/clarification-terminal-event.test.ts (new, 4 tests), README.md, docs/api.md, docs/integration.md

1. Symptom

A subscribe-first WebSocket client on a job stream receives two frames and then nothing forever:

  1. an ordinary progress frame, and
  2. a "control state" frame that carries the halt decision data (missing fields, or selectable candidates) but no terminal marker.

After frame 2 the server emits no further frame, no error, and no close. The first-party UI happens to key off the controlState string inside the frame body, so it "works"; any integrator that waits for a terminal/completion event hangs on an unbounded stream while the server believes it behaved correctly.

2. Root cause

apps/api/src/orchestrator/run.ts runs an orchestrator loop with a shared emit path that broadcasts (phase, controlState, details), spreading details into the wire payload. The happy path signals completion with a payload sub-type (details.type === 'surface_complete'); that is the only thing consumers treat as end-of-run.

The halt branches (clarification / client-or-entity selection) set running = false immediately after emitting one ordinary frame whose details contain only the decision data. Nothing distinguishes that frame from a mid-run progress frame:

Because the loop then stops, there is no subsequent frame to serve as a de-facto terminator either. The bug is a protocol gap, not a transport bug: the halt is a terminal control-state change, but it is encoded as a normal progress update. A first-party UI reading controlState is not a contract.

3. The contract (post-fix payload)

Reuse the existing payload sub-type convention. The single halt frame carries:

{
  // ... existing phase / controlState / progress fields ...
  type: 'awaiting_input',          // machine-readable halt sub-type
  terminal: true,                  // run has stopped and is waiting on the caller
  list: unknown[],                 // actionable items, data unchanged
                                   //   clarification -> missing fields
                                   //   selection      -> candidate choices
  resume: {
    method: 'POST',
    path: '/api/sessions/:sessionId/messages',
    body: { sessionId: string, message: '' },
  },
}

Rules:

Detection predicate for integrators

function isRunSuspended(frame): boolean {
  return frame.terminal === true && frame.type === 'awaiting_input';
}

The run is finished when frame.terminal === true (halted, awaiting input) or when frame.type === 'surface_complete' (completed).

4. Exact fix

4.1 apps/api/src/orchestrator/run.ts — add a single halt emitter

Introduce one helper next to the existing emit(...) method so both branches produce an identical terminal shape and there is exactly one frame per halt:

type HaltKind = 'awaiting_clarification' | 'awaiting_client_selection';

private emitHalt(
  controlState: HaltKind,
  list: unknown[],
  extra: Record<string, unknown> = {},
): void {
  this.emit('control', controlState, {
    // Reuse the payload sub-type convention already used by
    // COMPLETED / surface_complete frames. `details` is spread into the
    // broadcast payload by the stream manager.
    type: 'awaiting_input',
    terminal: true,
    list,
    resume: {
      method: 'POST',
      path: `/api/sessions/${this.sessionId}/messages`,
      body: { sessionId: this.sessionId, message: '' },
    },
    ...extra, // preserve the original data keys (missingFields / candidates)
  });
}

4.2 Halt branch 1 — clarification (the reproduced bug)

- if (missingFields.length > 0) {
-   this.emit('control', 'awaiting_clarification', { missingFields });
-   running = false;
-   continue;
- }
+ if (missingFields.length > 0) {
+   // Single terminal frame: type + terminal + list + resume.
+   this.emitHalt('awaiting_clarification', missingFields, { missingFields });
+   running = false;
+   continue;
+ }

4.3 Halt branch 2 — client / entity selection

- if (!selectedClient && candidates.length > 0) {
-   this.emit('control', 'awaiting_client_selection', { candidates });
-   running = false;
-   continue;
- }
+ if (!selectedClient && candidates.length > 0) {
+   // Same terminal contract as the clarification halt.
+   this.emitHalt('awaiting_client_selection', candidates, { candidates });
+   running = false;
+   continue;
+ }

Adapt the surrounding if conditions / variable names to the current branch bodies; the only required change is replacing the bare emit('control', ...) with emitHalt(...) so both halt sites terminate explicitly. Do not add a second emit/end frame.

4.4 The happy path (negative control)

The COMPLETED / surface_complete emit is left untouched. It already carries details.type = 'surface_complete' and must not gain terminal: true:

// unchanged
this.emit('completed', 'surface_complete', { type: 'surface_complete', ...details });

5. New tests

apps/api/tests/chat/clarification-terminal-event.test.ts (README-referenced, re-runnable without a live probe). Vitest + ws + supertest:

import { describe, expect, it } from 'vitest';
import { WebSocket } from 'ws';
import type { AddressInfo } from 'node:net';
import { createApp } from '../../src/app'; // adjust to real factory
import { StreamManager } from '../../src/streams/stream-manager';

// --- helpers -------------------------------------------------------------

async function subscribeFirst(opts: {
  listen: () => Promise<{ url: string; close: () => Promise<void> }>;
  initStream: () => Promise<string>;
  subscribe: (sessionId: string) => Promise<void>;
  run: (sessionId: string) => Promise<unknown>;
}) {
  const server = await opts.listen();
  const sessionId = await opts.initStream();
  const ws = new WebSocket(server.url.replace('http', 'ws') + '/stream');

  const frames: any[] = [];
  await new Promise<void>((res, rej) => {
    ws.on('open', () => res());
    ws.on('error', rej);
  });
  ws.on('message', (raw) => frames.push(JSON.parse(raw.toString())));
  await opts.subscribe(sessionId);
  await opts.run(sessionId);

  // Wait well past the expected job duration without assuming a close.
  await new Promise((r) => setTimeout(r, 1500));
  ws.close();
  await server.close();
  return frames;
}

// --- 1. halt frame is terminal, complete, and LAST -----------------------

it('halt frame carries terminal marker + list + resume and is the last frame', async () => {
  const frames = await probeMissingField();
  const last = frames.at(-1)!;

  expect(last.terminal).toBe(true);
  expect(last.type).toBe('awaiting_input');
  expect(Array.isArray(last.list)).toBe(true);
  expect(last.list).toEqual(expect.arrayContaining(['customerId'])); // missing fields
  expect(last.resume).toEqual({
    method: 'POST',
    path: expect.stringContaining('/messages'),
    body: { sessionId: expect.any(String), message: expect.any(String) },
  });
  // No frame after the halt.
  expect(frames.findIndex((f) => f === last)).toBe(frames.length - 1);
});

// --- 2. minimal reproducing request ends in exactly that halt ------------

it('minimal request ends in exactly one halt frame at the expected count', async () => {
  const frames = await probeMissingField();
  const halts = frames.filter((f) => f.type === 'awaiting_input');
  expect(halts).toHaveLength(1);
  expect(frames).toHaveLength(2); // progress + single halt (count unchanged)
  expect(frames.at(-1)).toMatchObject({ terminal: true, type: 'awaiting_input' });
});

// --- 3. negative control: happy path is not over-marked ------------------

it('happy path emits no false terminal marker', async () => {
  const frames = await probeFullySpecified();
  expect(frames.some((f) => f.terminal === true)).toBe(false);
  expect(frames.every((f) => f.type !== 'awaiting_input')).toBe(true);
  expect(frames.at(-1)?.type).toBe('surface_complete'); // pre-existing completion
});

// --- 4. emit -> broadcast spread preserves terminal fields on the wire ----

it('terminal fields survive emit -> stream-manager broadcast to a raw client', async () => {
  const sm = new StreamManager();
  const app = createApp({ streamManager: sm });
  const http = app.listen(0);
  const { port } = http.address() as AddressInfo;

  const ws = new WebSocket(`ws://<ip-address>:${port}/stream`);
  const received: any[] = [];
  await new Promise<void>((res) => ws.on('open', res));
  ws.on('message', (r) => received.push(JSON.parse(r.toString())));

  // Exercise the REAL emit path used by the halt branch.
  sm.broadcast('session-1', {
    phase: 'control',
    controlState: 'awaiting_clarification',
    type: 'awaiting_input',
    terminal: true,
    list: ['customerId'],
    resume: { method: 'POST', path: '/api/sessions/session-1/messages',
              body: { sessionId: 'session-1', message: '' } },
  });
  await new Promise((r) => setTimeout(r, 200));

  expect(received).toHaveLength(1);
  expect(received[0]).toMatchObject({
    type: 'awaiting_input',
    terminal: true,
    list: ['customerId'],
    resume: expect.objectContaining({ method: 'POST' }),
  });

  ws.close();
  await new Promise((r) => http.close(r));
});

Run:

pnpm --filter @heading/api test -- clarification-terminal-event

6. Documentation

README.md (new "Streaming contract" subsection)

Run lifecycle frames. A subscription ends for one of two reasons. A successful run ends with a frame whose type is surface_complete. A run that halts awaiting input ends with a single frame: terminal === true, type === 'awaiting_input', list (the missing fields or candidate choices), and resume ({ method, path, body: { sessionId, message } }). Clients MUST stop reading once terminal === true or type === 'surface_complete'; do not wait for a close event. Halt runs are resumed with the resume call, not by re-subscribing.

docs/api.md

Document the exact halt payload shape and the HTTP resume call POST /api/sessions/:sessionId/messages with body { sessionId, message }, including a worked clarification example.

docs/integration.md

Publish the copy-paste detection predicate and a raw-client recipe:

if (frame.terminal === true && frame.type === 'awaiting_input') {
  await fetch(frame.resume.path, {
    method: frame.resume.method,
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify({ ...frame.resume.body, message: answers }),
  });
}

State explicitly that controlState string matching is not a supported contract.

7. Verification (live, RED → GREEN)

The server runs from a baked image, so the working tree is irrelevant until rebuilt. Rebuild between probes.

Probe (subscribe-first, raw frames)

# 1. pre-allocate a stream id
SESSION=$(curl -s -X POST "$BASE/api/sessions/init" | jq -r .sessionId)
# 2. open socket and WAIT for open, then send subscribe
#    (node/websocat/py; the important part is subscribe BEFORE the POST)
# 3. POST the minimal request that omits a decision-critical field
curl -s -X POST "$BASE/api/sessions/$SESSION/messages" \
  -H 'content-type: application/json' \
  -d '{"message":"do the thing"}'
# 4. record RAW frames + wall clock; wait > 30s

RED (pre-fix)

frames: 2
[0] progress
[1] controlState="awaiting_clarification", missingFields=[...], no terminal marker
then: 30s+ silence, no error, no close

Server log shows no broadcast after the halt; the running = false branch emits exactly one frame.

GREEN (post-rebuild)

frames: 2                      # COUNT UNCHANGED
[0] progress
[1] controlState="awaiting_clarification"
     type="awaiting_input", terminal=true, list=[...],
     resume={method,path,body}
then: no further frame

Happy-path control: send a fully-specified request and confirm the full progression plus a final surface_complete frame, and that no frame carries terminal: true.

Checklist

8. Lessons encoded

  1. A halt is a terminal event; without a terminal frame clients cannot tell "finished, waiting for you" from "still working" or "broken".
  2. Reuse the existing payload sub-type convention — one detection predicate, no new transport.
  3. A first-party UI reading controlState is not a contract; test with a raw client.
  4. Bundled/container builds bake the server code — the RED→GREEN pair must include a rebuild.
  5. Always add the negative control (happy path emits no terminal marker), or a blanket "always mark terminal" change would pass the positive test.

Caveat: because the source tree was unavailable here, the branch conditions/flags in §4.2–4.3 and the test helper names must be reconciled against the real run.ts. The required semantic change is exact and complete: replace the bare halt emit with emitHalt(...) at both halt sites, add type: 'awaiting_input' + terminal: true + list + resume, keep it a single frame, and leave the happy path unmarked. Re-running the 4 tests plus the live RED→GREEN probes is what closes verification.

Evidence & signatures

# Evidence
- Problem class: ws-orchestrator-halt-no-terminal-stream-event
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-16T12:57:14.763Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "SYMPTOM. A long-running server-side job streams progress over a WebSocket topic and then HALTS awaiting user input (a clarification state: the request was missing decision-critical fields). The halt is a control-state change, so the server just stops emitting: the client receives a normal-looking progress frame, then silence forever. Nothing on the wire says the run ended, why, or how to resume, so every non-first-party client hangs on an unbounded stream while the server believes it behaved correctly (no error, no timeout, no close). The project's own UI masked the bug because it keyed off the control-state string inside the frame body; integrators that wait for a terminal/completion event cannot detect the halt at all.\n\nROOT CAUSE. The orchestrator loop's halt branches set running=false after emitting one ordinary progress frame whose payload carries no end-of-run marker. The emit path (phase, controlState, details) is shared with the happy path, and the terminal phases of the happy path (COMPLETED / surface_complete) are the only frames anyone treats as terminal, so a halt frame is indistinguishable from a mid-run progress frame to any consumer that does not hardcode the specific control-state value.\n\nDIAGNOSIS. Reproduce with a subscribe-first WS client: GET the init endpoint for a pre-allocated stream id, open the socket, AWAIT onopen, send the subscribe frame, then POST the job request. Count frames and wait well past the expected job duration. Capture the RAW frames (not a summary), because the halt frame usually looks healthy: field-by-field it carries the needed data (which fields are missing) but no terminal marker. Confirm the server log shows no broadcast after the halt, and confirm the loop's halt branch (running=false) emits exactly one frame. Verify the happy path separately to ensure you are not looking at a generic delivery problem (a subscription race or a stale container image can both look like silence \u2014 check those first with the same probe sending a fully-specified request).\n\nFIX. Make every halt TERMINATE the story explicitly, using the payload sub-type convention the streaming layer already uses for its completed frames (details.type spread into the broadcast payload), rather than adding an out-of-band mechanism:\n- mark the single halt frame with a machine-readable sub-type (here: type='awaiting_input') plus terminal=true;\n- keep the halt a SINGLE frame per halt \u2014 do not append a second 'end' frame, so consumers counting frames do not double-count;\n- carry the actionable list (missing fields / candidate choices) unchanged, plus a resume hint: { method, path, body: { sessionId, message } } describing exactly how to continue the run;\n- apply it to EVERY halt control state, not just the one you reproduced (the second halt branch is usually a client/entity selection state);\n- document the contract (payload shape, detection predicate, resume call) in the project's README + API/integration docs, so integrators never read source;\n- add in-repo tests, because a live probe is not evidence a judge or the next maintainer can re-run: (1) the halt frame carries the terminal marker + list + resume hint and is the LAST frame; (2) the minimal/reproducing request ends in exactly that halt with the expected event count; (3) the happy path emits NO terminal marker (guards against over-marking); (4) a broadcast-envelope test through a real server + stream manager on an ephemeral port proves the terminal fields survive the emit->broadcast spread into the frame a raw client receives.\n\nVERIFICATION (live, both directions). RED before the fix: 2 frames, the second carrying the control state and the missing list, then 30s+ silence, no terminal marker. GREEN after rebuilding the running container (a fix in the working tree is not a fix in a baked image or dist bundle): the SAME 2 frames, with the second now carrying type/terminal/list/resume, and the happy-path probe still emitting its full progression with no terminal marker. Note event COUNT must be unchanged by the fix \u2014 if the count grows, you added a redundant end frame.\n\nLESSONS. (1) A halt is a terminal event; if your protocol has no terminal frame for it, clients cannot distinguish 'finished, waiting for you' from 'still working' or 'broken'. (2) Reuse the protocol's existing payload sub-type convention instead of inventing a parallel channel \u2014 one detection predicate, no new transport. (3) A first-party UI reading a control-state string is NOT a contract: test with a raw client. (4) Any container/bundled build bakes the server code, so the RED->GREEN pair must include a rebuild between the two probes. (5) Add a negative control (happy path emits no terminal marker) or a blanket 'always mark terminal' change passes the positive test.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "ws-orchestrator-halt-no-terminal-stream-event", "provider": "openrouter", "solved_at": "2026-09-16T12:57:14.763Z", "version": ""}
Generated from the verified corpus · MIT licensedBack to the catalog