Answer

Diagnosed, fixed, and verified with a runnable Go reproduction. Solution written to ~/df-ui-retry/SOLUTION.md (code in retry.go, tests in retry_test.go; go test ./... passes and the buggy path reproduces the exact error). Here is the self-contained solution:

---

Fix: HTTP client retry storm without a shared deadline budget

Symptom

context deadline exceeded (Client.Timeout exceeded while awaiting headers) x N retries

A request that should fail fast instead consumes its full per-attempt timeout on every retry. When the upstream is slow, the client amplifies load, each call takes attempts × timeout + backoff, and logs fill with the same timeout error.

Root cause

Two compounding mistakes; both must be fixed.

1. http.Client.Timeout is a per-attempt cap, not a request budget. It starts a timer for one Do call. A retry loop reusing the same client resets that timer each iteration, so worst-case latency is MaxAttempts × Client.Timeout. Go appends (Client.Timeout exceeded while awaiting headers) in net/http/client.go only when the client's own timer fired — which is why the error repeats once per retry.
2. No single shared deadline covers the whole operation. If each attempt gets a fresh context.WithTimeout (or only Client.Timeout), nothing stops the loop when the caller's real budget is gone. Backoff sleeps also ignore cancellation.

The fix is one deadline for the entire operation, with every attempt and every backoff derived from and bounded by it.

The fix

Set http.Client.Timeout = 0 and drive cancellation entirely from the request context. Put the total budget on that context once, at the call site.

1. Drop-in deadline-aware retry helper

package retry

import (
    "context"
    "errors"
    "fmt"
    "io"
    "net/http"
    "time"
)

// ErrBudgetExhausted is returned when the shared deadline (or the caller's
// context) stopped the retry loop before an attempt succeeded.
var ErrBudgetExhausted = errors.New("retry budget exhausted")

// RetryPolicy describes a deadline-aware retry loop.
type RetryPolicy struct {
    MaxAttempts int           // total attempts, including the first (>= 1)
    PerAttempt  time.Duration // cap for a single attempt; 0 = no extra cap
    BaseBackoff time.Duration // first backoff; doubles up to MaxBackoff
    MaxBackoff  time.Duration // backoff ceiling
}

// DoWithBudget performs req under a single caller-supplied budget, retrying
// transient failures without ever letting the *sum* of attempts exceed it.
//
// Key rules:
//   - client.Timeout must be 0; the request context is the only timer.
//   - every attempt context is derived from ctx, so it cannot outlive it.
//   - backoff is cancellable and is skipped if it would cross the deadline.
func (p RetryPolicy) DoWithBudget(ctx context.Context, client *http.Client, req *http.Request) (*http.Response, error) {
    if p.MaxAttempts < 1 {
        p.MaxAttempts = 1
    }
    if p.MaxBackoff <= 0 {
        p.MaxBackoff = p.BaseBackoff
    }

    var lastErr error
    backoff := p.BaseBackoff

    for attempt := 1; attempt <= p.MaxAttempts; attempt++ {
        if err := ctx.Err(); err != nil {
            return nil, budgetError(lastErr, err)
        }

        attemptCtx, cancel := ctx, context.CancelFunc(func() {})
        if p.PerAttempt > 0 {
            attemptCtx, cancel = context.WithTimeout(ctx, p.PerAttempt)
        }
        resp, err := client.Do(req.Clone(attemptCtx))
        cancel()

        if err == nil {
            if !retryableStatus(resp.StatusCode) || attempt == p.MaxAttempts {
                return resp, nil
            }
            io.Copy(io.Discard, io.LimitReader(resp.Body, 1<<12))
            resp.Body.Close()
            lastErr = fmt.Errorf("retryable status %d", resp.StatusCode)
        } else {
            lastErr = err
        }

        if attempt == p.MaxAttempts {
            break
        }
        // Don't start a backoff we cannot finish inside the budget.
        if deadline, ok := ctx.Deadline(); ok && time.Now().Add(backoff).After(deadline) {
            return nil, budgetError(lastErr, context.DeadlineExceeded)
        }
        t := time.NewTimer(backoff)
        select {
        case <-ctx.Done():
            t.Stop()
            return nil, budgetError(lastErr, ctx.Err())
        case <-t.C:
        }
        if backoff > 0 {
            backoff *= 2
            if backoff > p.MaxBackoff {
                backoff = p.MaxBackoff
            }
        }
    }
    return nil, lastErr
}

func budgetError(last, cause error) error {
    if last == nil {
        last = cause
    }
    return fmt.Errorf("%w: %v", ErrBudgetExhausted, last)
}

func retryableStatus(code int) bool {
    switch code {
    case http.StatusTooManyRequests,
        http.StatusBadGateway,
        http.StatusServiceUnavailable,
        http.StatusGatewayTimeout:
        return true
    default:
        return false
    }
}

2. Call site — one budget for the whole operation

// BEFORE (bug: per-attempt timer, no total budget)
client := &http.Client{Timeout: 2 * time.Second}
for i := 0; i < 4; i++ {
    resp, err := client.Get(url) // may take 4 × 2s = 8s
    ...
}

// AFTER (one shared budget; client has no independent timer)
client := &http.Client{Timeout: 0} // cancellation lives in the context
ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
defer cancel()

req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
    return err
}

policy := retry.RetryPolicy{
    MaxAttempts: 4,
    PerAttempt:  800 * time.Millisecond, // optional single-attempt cap
    BaseBackoff: 50 * time.Millisecond,
    MaxBackoff:  400 * time.Millisecond,
}
resp, err := policy.DoWithBudget(ctx, client, req)
if errors.Is(err, retry.ErrBudgetExhausted) {
    // total budget spent — fail fast, do not retry again
}

3. Find every offender in the codebase

# Clients with a hard per-attempt timeout that are reused in a retry loop:
rg -n 'http\.Client\s*\{[^}]*Timeout' --glob '*.go'
# Retry loops around Do/Get/Post:
rg -n 'for .*attempt|retr(y|ies)|backoff' --glob '*.go' -i

Every hit where a Client.Timeout is reused across retries must be migrated to the budgeted context pattern above.

Migration checklist

• [ ] http.Client.Timeout reduced to 0 on retried clients.
• [ ] One context.WithTimeout per logical operation (from the inbound request context so client disconnects propagate).
• [ ] req.Clone(ctx) (or NewRequestWithContext) per attempt.
• [ ] Backoff uses select { case <-ctx.Done(): ... case <-timer.C: }.
• [ ] Retry loop re-checks ctx.Err() before each attempt.
• [ ] Only idempotent requests and retryable statuses (429/502/503/504) retry.
• [ ] Response bodies drained/closed before a retry to reuse connections.

Verification

cd ~/df-ui-retry
go vet ./...
go test -v -count=1 ./...

Verified output:

=== RUN   TestBuggyBlowsPastBudget
    retry_test.go:41: buggy: elapsed=1.004098084s
        err=Get "http://<ip-address>:39803": context deadline exceeded
        (Client.Timeout exceeded while awaiting headers)
--- PASS: TestBuggyBlowsPastBudget (1.00s)
=== RUN   TestFixedHonorsSharedBudget
    retry_test.go:76: fixed: elapsed=351.516335ms
        err=retry budget exhausted: Get "http://<ip-address>:37891": context deadline exceeded
--- PASS: TestFixedHonorsSharedBudget (0.35s)
=== RUN   TestFixedSucceedsWithinBudget
--- PASS: TestFixedSucceedsWithinBudget (0.10s)
PASS
ok      example.com/retryfix    1.460s

Test	Before	After
TestBuggyBlowsPastBudget	4 retries × 200 ms Client.Timeout = 1.00 s, exact Client.Timeout exceeded error	—
TestFixedHonorsSharedBudget	—	351 ms, bounded by the 400 ms shared budget, returns ErrBudgetExhausted
TestFixedSucceedsWithinBudget	—	200 for a fast server, no false timeout

The package-level budget assertion elapsed <= budget + 75ms makes the regression impossible to reintroduce: the sum of all attempts and backoffs can never exceed the caller's deadline.

---

Bottom line: remove http.Client.Timeout from retried clients, put a single context.WithTimeout around the whole operation, derive each attempt from it, and make backoff cancellable/deadline-aware. That converts N × timeout worst-case latency into a hard budget upper bound.

# Evidence
- Problem class: df-ui-golang-http-client-timeout-retry-budget-20261013
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-10-13T06:03:57.480Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "UI dogfood: HTTP client retry storm without a shared deadline budget causes cascading timeouts.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "df-ui-golang-http-client-timeout-retry-budget-20261013", "provider": "openrouter", "solved_at": "2026-10-13T06:03:57.480Z", "version": ""}