◐ Off-By-One

go-llm-routing-provider-selection

2 answer(s)godockergodocker

Answer 1

Built a greenfield internal/ai package in ~ containing a deterministic LLM provider router plus a capability side-table. The pipeline runs in exactly the stated order — cost filter → type filter → complexity escalation → priority tie-break → fallback chain → default — and never mutates ProviderConfig, so existing NewClient callers are untouched.

internal/ai/router.go — the router, with each stage as an independently testable method:

// Primary runs stages 1-4 and returns the ordered candidate pool.
func (r *Router) Primary(opts RouteOptions) []Provider {
    if opts.Complexity == 0 {
        opts.Complexity = ComplexityLow
    }
    pool := make([]Provider, 0, len(r.order))
    for _, p := range r.order {
        if _, ok := r.cap(p); ok { // side-table is the source of routing truth
            pool = append(pool, p)
        }
    }
    pool = r.costFilter(pool, opts.MaxCost)           // 1. cost ceiling
    pool = r.typeFilter(pool, opts.RequiredType)      // 2. model type
    pool = r.complexityEscalate(pool, opts.Complexity) // 3. tier escalation
    return r.priorityTieBreak(pool)                    // 4. deterministic sort
}

// priorityTieBreak: (Priority asc, CostPer1K asc, registration index) — the
// registration-index term makes even full ties deterministic.
func (r *Router) priorityTieBreak(providers []Provider) []Provider {
    out := append([]Provider(nil), providers...)
    sort.SliceStable(out, func(i, j int) bool {
        ci, _ := r.cap(out[i])
        cj, _ := r.cap(out[j])
        if ci.Priority != cj.Priority { return ci.Priority < cj.Priority }
        if ci.CostPer1K != cj.CostPer1K { return ci.CostPer1K < cj.CostPer1K }
        return r.idx[out[i]] < r.idx[out[j]]
    })
    return out
}

// Candidates: primary pool (1-4) -> fallback chain (5) -> default (6).
func (r *Router) Candidates(opts RouteOptions) []Provider { /* dedup chain + default */ }

// Route: head of Primary, else Default, else error.
func (r *Router) Route(opts RouteOptions) (Provider, error) { ... }

// RouteWithFallback: tries the full plan at call time; attempt returns nil on success.
func (r *Router) RouteWithFallback(opts RouteOptions, attempt func(Provider) error) (Provider, error) { ... }

internal/ai/capability.go — the side-table keyed by provider, deliberately separate from config:

type ProviderCapability struct {
    Provider      Provider
    Type          ModelType
    CostPer1K     float64
    MaxComplexity Complexity
    Priority      int
}
type capabilityTable map[Provider]ProviderCapability
func (r *Router) RegisterCapability(c ProviderCapability) { r.add(c.Provider); r.caps.set(c) }

internal/ai/client.go — the pre-existing surface, unchanged in shape: ProviderConfig{Name, APIKey, BaseURL}, NewClient(cfg) *Client. The router only ever reads the side-table; ProviderConfig is never mutated.

internal/ai/router_test.go — same-package, table-driven, 21 cases covering every stage plus the fallback/default/error paths.

Evidence & signatures

`go vet ./...` clean; `gofmt -l` clean; full run:

```
--- PASS: TestRouterTable (0.00s)          # 21/21 subtests PASS
    cost_filter_disabled_keeps_all_chat_providers_eligible ... PASS
    cost_filter_keeps_providers_at_or_under_ceiling ........... PASS
    cost_filter_drops_expensive_providers ..................... PASS
    cost_filter_with_empty_pool_falls_back_to_default ......... PASS
    type_filter_routes_chat / _embedding / _image ............. PASS
    type_filter_with_unknown_type_falls_back_to_default ....... PASS
    low_complexity_stays_on_the_cheap_provider ................ PASS
    medium_complexity_escalates_past_low-capability_provider .. PASS
    high_complexity_escalates_to_high-capability_provider ..... PASS
    critical_complexity_escalates_to_top_tier ................. PASS
    zero_complexity_is_treated_as_low ......................... PASS
    escalation_plus_cost_filter_empties_the_pool .............. PASS
    priority_tie-break_prefers_lower_value .................... PASS
    equal_priority_broken_by_cost ............................. PASS
    full_tie_broken_by_registration_order ..................... PASS
    fallback_chain_used_after_primary_pool_fails .............. PASS
    fallback_chain_exhausted_falls_to_default ................. PASS
    all_candidates_fail_returns_error ......................... PASS
    empty_router_with_no_default_returns_error ................ PASS
--- PASS: TestDeterminismAndNewClientUntouched (0.00s)
PASS   ok  llmrouter/internal/ai   0.002s   (coverage: 95.8%)
```

Edge cases verified:
- **Escalation**: low stays on `cheap-chat` ($0.10); medium/high escalate to `mid-chat` (cap High); critical requires the only Critical-capable provider `big-chat`; `Complexity: 0` normalizes to low.
- **Filters**: `MaxCost <= 0` disables the cost filter; a type filter with no match (e.g., `"speech"`) and a cost ceiling nobody meets (`$0.01`) both fall through to the default; escalation + cost together can empty the pool.
- **Tie-breaks**: lower priority value wins; equal priority falls to cheaper cost; identical priority **and** cost falls to registration order — fully deterministic (repeated `Route` calls return the same provider).
- **Fallback/default**: providers without a capability record are excluded from the primary pool but reachable via the chain (`rescue`) or default (`emergency`); unregistered chain members are skipped; if every candidate's `attempt` fails, an error is returned.
- **Side-table isolation**: `TestDeterminismAndNewClientUntouched` asserts a caller's `ProviderConfig` is byte-identical after `NewClient`, even with capabilities registered.

{"model": "deepseek-v4-flash", "problem_class": "go-llm-routing-provider-selection", "result": "passed", "tests": 21}

Answer 2

internal/ai/router.go — the router, with each stage as an independently testable method:

// Primary runs stages 1-4 and returns the ordered candidate pool.
func (r *Router) Primary(opts RouteOptions) []Provider {
    if opts.Complexity == 0 {
        opts.Complexity = ComplexityLow
    }
    pool := make([]Provider, 0, len(r.order))
    for _, p := range r.order {
        if _, ok := r.cap(p); ok { // side-table is the source of routing truth
            pool = append(pool, p)
        }
    }
    pool = r.costFilter(pool, opts.MaxCost)           // 1. cost ceiling
    pool = r.typeFilter(pool, opts.RequiredType)      // 2. model type
    pool = r.complexityEscalate(pool, opts.Complexity) // 3. tier escalation
    return r.priorityTieBreak(pool)                    // 4. deterministic sort
}

// priorityTieBreak: (Priority asc, CostPer1K asc, registration index) — the
// registration-index term makes even full ties deterministic.
func (r *Router) priorityTieBreak(providers []Provider) []Provider {
    out := append([]Provider(nil), providers...)
    sort.SliceStable(out, func(i, j int) bool {
        ci, _ := r.cap(out[i])
        cj, _ := r.cap(out[j])
        if ci.Priority != cj.Priority { return ci.Priority < cj.Priority }
        if ci.CostPer1K != cj.CostPer1K { return ci.CostPer1K < cj.CostPer1K }
        return r.idx[out[i]] < r.idx[out[j]]
    })
    return out
}

// Candidates: primary pool (1-4) -> fallback chain (5) -> default (6).
func (r *Router) Candidates(opts RouteOptions) []Provider { /* dedup chain + default */ }

// Route: head of Primary, else Default, else error.
func (r *Router) Route(opts RouteOptions) (Provider, error) { ... }

// RouteWithFallback: tries the full plan at call time; attempt returns nil on success.
func (r *Router) RouteWithFallback(opts RouteOptions, attempt func(Provider) error) (Provider, error) { ... }

internal/ai/capability.go — the side-table keyed by provider, deliberately separate from config:

type ProviderCapability struct {
    Provider      Provider
    Type          ModelType
    CostPer1K     float64
    MaxComplexity Complexity
    Priority      int
}
type capabilityTable map[Provider]ProviderCapability
func (r *Router) RegisterCapability(c ProviderCapability) { r.add(c.Provider); r.caps.set(c) }

internal/ai/router_test.go — same-package, table-driven, 21 cases covering every stage plus the fallback/default/error paths.

Evidence & signatures

`go vet ./...` clean; `gofmt -l` clean; full run:

```
--- PASS: TestRouterTable (0.00s)          # 21/21 subtests PASS
    cost_filter_disabled_keeps_all_chat_providers_eligible ... PASS
    cost_filter_keeps_providers_at_or_under_ceiling ........... PASS
    cost_filter_drops_expensive_providers ..................... PASS
    cost_filter_with_empty_pool_falls_back_to_default ......... PASS
    type_filter_routes_chat / _embedding / _image ............. PASS
    type_filter_with_unknown_type_falls_back_to_default ....... PASS
    low_complexity_stays_on_the_cheap_provider ................ PASS
    medium_complexity_escalates_past_low-capability_provider .. PASS
    high_complexity_escalates_to_high-capability_provider ..... PASS
    critical_complexity_escalates_to_top_tier ................. PASS
    zero_complexity_is_treated_as_low ......................... PASS
    escalation_plus_cost_filter_empties_the_pool .............. PASS
    priority_tie-break_prefers_lower_value .................... PASS
    equal_priority_broken_by_cost ............................. PASS
    full_tie_broken_by_registration_order ..................... PASS
    fallback_chain_used_after_primary_pool_fails .............. PASS
    fallback_chain_exhausted_falls_to_default ................. PASS
    all_candidates_fail_returns_error ......................... PASS
    empty_router_with_no_default_returns_error ................ PASS
--- PASS: TestDeterminismAndNewClientUntouched (0.00s)
PASS   ok  llmrouter/internal/ai   0.002s   (coverage: 95.8%)
```

Edge cases verified:
- **Escalation**: low stays on `cheap-chat` ($0.10); medium/high escalate to `mid-chat` (cap High); critical requires the only Critical-capable provider `big-chat`; `Complexity: 0` normalizes to low.
- **Filters**: `MaxCost <= 0` disables the cost filter; a type filter with no match (e.g., `"speech"`) and a cost ceiling nobody meets (`$0.01`) both fall through to the default; escalation + cost together can empty the pool.
- **Tie-breaks**: lower priority value wins; equal priority falls to cheaper cost; identical priority **and** cost falls to registration order — fully deterministic (repeated `Route` calls return the same provider).
- **Fallback/default**: providers without a capability record are excluded from the primary pool but reachable via the chain (`rescue`) or default (`emergency`); unregistered chain members are skipped; if every candidate's `attempt` fails, an error is returned.
- **Side-table isolation**: `TestDeterminismAndNewClientUntouched` asserts a caller's `ProviderConfig` is byte-identical after `NewClient`, even with capabilities registered.

{"model": "deepseek-v4-flash", "problem_class": "go-llm-routing-provider-selection", "result": "passed", "tests": 21}

Generated from the verified corpus · MIT licensedBack to the catalog