go-cli-stub-mode-surfacing

Answer

Problem: When the server had no real model configured it silently served canned answers (logging falling back to stub server-side), and the Go CLI rendered them as if they were real model output — misleading users into thinking live inference was happening.

Fix, in three parts:

1. types.ChatResponse gains a Stub flag

internal/types/chat.go:

package types

// ChatResponse is the wire format for POST /api/chat.
type ChatResponse struct {
    ID      string `json:"id,omitempty"`
    Message string `json:"message"`
    // Stub is true when the server fell back to a canned stub answer
    // because no real model is configured. Omitted from JSON when false
    // so existing clients see an unchanged payload.
    Stub bool `json:"stub,omitempty"`
}

omitempty guarantees the flag never pollutes the response in the normal (non-stub) path — a pre-stub client or a diff of old vs. new JSON output is byte-identical when Stub is false.

2. The express handler flags the response

The server selects a model once at startup; the express HTTP handler type-asserts the concrete model to detect the stub:

internal/express/handler.go:

func (s *Server) handleChat(w http.ResponseWriter, r *http.Request) {
    var req types.ChatRequest
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        writeError(w, http.StatusBadRequest, err)
        return
    }

    msg, err := s.model.Chat(r.Context(), req.Prompt)
    if err != nil {
        writeError(w, http.StatusInternalServerError, err)
        return
    }

    resp := types.ChatResponse{ID: newID(), Message: msg}

    // Surface stub mode to the client instead of hiding it.
    if _, ok := s.model.(*stubChatModel); ok {
        resp.Stub = true
    }

    writeJSON(w, resp)
}

And the fallback decision point (where falling back to stub was previously logged and forgotten):

// model.go
func selectModel(cfg Config) ChatModel {
    switch {
    case cfg.Provider == "":
        log.Println("falling back to stub: RABBITHOLE_CHAT_MODEL_PROVIDER not set")
        return &stubChatModel{}
    case cfg.Name == "":
        log.Println("falling back to stub: RABBITHOLE_CHAT_MODEL_NAME not set")
        return &stubChatModel{}
    case cfg.APIKey == "":
        log.Println("falling back to stub: RABBITHOLE_CHAT_MODEL_API_KEY not set")
        return &stubChatModel{}
    default:
        return newProviderModel(cfg)
    }
}

Using a type assertion (rather than an extra config boolean) keeps the stub status derived from the single source of truth — the actual model in use — so it can't drift from what the handler really executed.

3. CLI warns on stderr, keeps --json stdout pure

cmd/chat/main.go:

const stubWarning = "warning: server returned a canned stub answer (stub mode). " +
    "Set RABBITHOLE_CHAT_MODEL_PROVIDER, RABBITHOLE_CHAT_MODEL_NAME, and " +
    "RABBITHOLE_CHAT_MODEL_API_KEY to use a real model."

func run(args []string, stdout, stderr io.Writer) int {
    flags := flag.NewFlagSet("chat", flag.ContinueOnError)
    jsonOut := flags.Bool("json", false, "emit response as JSON")
    // ...parse args...

    resp, err := api.Chat(server, prompt) // decodes types.ChatResponse
    if err != nil {
        fmt.Fprintf(stderr, "chat: %v\n", err)
        return 1
    }

    if resp.Stub {
        fmt.Fprintln(stderr, stubWarning) // always stderr, never stdout
    }

    if *jsonOut {
        enc := json.NewEncoder(stdout)
        enc.SetIndent("", "  ")
        if err := enc.Encode(resp); err != nil {
            fmt.Fprintf(stderr, "chat: %v\n", err)
            return 1
        }
        return 0
    }

    fmt.Fprintln(stdout, resp.Message)
    return 0
}

Key decision: the warning goes to stderr in both modes. In --json mode stdout carries only the JSON document (ready for jq/piping), and the warning is visible on the terminal or captured separately by the caller. Non-JSON mode is unchanged except for the extra warning line on stderr.

4. Paired docs task

• Integration guide (docs/integration.md): documents the stub field in ChatResponse, the three RABBITHOLE_CHAT_MODEL_* env vars, and how to detect stub mode programmatically.
• README: added the OpenAPI link so consumers can see the new field in the schema.

---

**Verification performed:**

1. **Unit test — JSON shape of `ChatResponse`:** `TestChatResponseMarshal` asserts `{"message":"hi"}` when `Stub` is false (field omitted, payload unchanged from before the fix) and `{"message":"hi","stub":true}` when true. Round-trip via `json.Unmarshal` reproduces the flag.
2. **Handler test — stub vs. real model:** with `selectModel` returning `&stubChatModel{}`, `POST /api/chat` response body contains `"stub":true`; with a fake real model, the body contains no `stub` key at all.
3. **CLI test — output-channel separation:** runs `run()` with buffers, asserts:
   - stub + `--json` → stdout parses as pure JSON with `stub:true`, and the warning text appears **only** in the stderr buffer (grep on stdout buffer finds zero occurrences of "warning");
   - stub + text mode → message on stdout, warning on stderr;
   - non-stub + `--json` → stdout identical to pre-fix output (golden fixture), stderr empty.
4. **End-to-end manual check:** started the server with all three `RABBITHOLE_CHAT_MODEL_*` vars unset, ran `chat --json` — observed the warning on stderr and clean JSON on stdout; `chat | jq .` worked unbroken. Re-ran with vars set (mock provider) — no warning, no `stub` key.
5. **Regression:** existing tests for the API client and non-chat routes untouched and green.

**Edge cases tested:**
- Stub flag absent when a real model is configured (no false positives, `omitempty` keeps old clients working).
- Only *some* of the three env vars set — still falls back to stub and warns (each var independently required).
- `--json` piping: `chat --json | jq '.message'` — warning cannot corrupt stdout since it's on stderr.
- Server restart between env changes — flag recomputed via the type assertion, no stale state.

**Automated gates:** guard suite **4/4** passed (build, `go vet`, `gofmt -l` clean, `go test ./...`); judge suites **PASS 4/4 + 3/3** (11 checks total: 4 handler/type checks, 3 CLI-channel checks, plus the 4 guards).

---

{"model": "deepseek-v4-flash", "problem_class": "go-cli-stub-mode-surfacing", "result": "passed", "tests": 11}