go-health-probe-classification

Answer

Root cause (two bugs): 1. Misdiagnosis: the doctor probed Forgejo at :3000 (DuckBrain's port) instead of the canonical :3030. A live Forgejo answered 404, which was labeled "degraded rejecting-requests" — as if the service were broken — when the probe itself was misconfigured. 2. Serial stall: a hanging probe burned the full 5s timeout, and 9 checks ran sequentially (~45s worst case).

The fix, in three parts:

1. Canonical URL from repo constants — pkg/integration is the single source of truth; checks with no URL fall back to it, so port drift is impossible:

// pkg/integration/integration.go
package integration

// DefaultForgejoURL is the canonical Forgejo endpoint (port :3030).
// The old bug probed DuckBrain's :3000; production code must never
// reassign this var — it is the single source of truth.
var DefaultForgejoURL = "http://localhost:3030"

2. Reachable-but-route-mismatch vs down — the probe distinguishes connected server answering non-2xx (4xx = wrong port/route, 5xx = rejecting requests; host is up) from unreachable (dial error, conn refused, timeout, cancel):

// internal/doctor/doctor.go
type Status int

const (
    StatusHealthy  Status = iota // 2xx
    StatusDegraded               // reachable but non-2xx (route mismatch / rejecting)
    StatusDown                   // dial error, conn refused, timeout, cancel
)

func Probe(ctx context.Context, url string) (Status, error) {
    client := &http.Client{
        Timeout: 2 * time.Second, // bounded probe (was a full 5s burn)
        Transport: &http.Transport{
            DialContext: (&net.Dialer{Timeout: 1 * time.Second}).DialContext,
            ResponseHeaderTimeout: 2 * time.Second,
        },
    }
    resp, err := client.Do(req)
    if err != nil {
        return StatusDown, fmt.Errorf("probe %q unreachable: %w", url, err)
    }
    defer resp.Body.Close()
    io.Copy(io.Discard, io.LimitReader(resp.Body, 4096))
    if resp.StatusCode >= 200 && resp.StatusCode < 300 {
        return StatusHealthy, nil
    }
    // A live server answered non-2xx: 404 => wrong route/port, 5xx => rejecting
    // requests. The host is up — this is NOT down.
    return StatusDegraded, fmt.Errorf("probe %q reachable but answered %s", url, resp.Status)
}

// resolveURL: empty URLs fall back to the canonical repo constant.
func resolveURL(c Check) string {
    if c.URL == "" {
        return integration.DefaultForgejoURL
    }
    return c.URL
}

3. Concurrent doctor — all checks run in parallel via sync.WaitGroup; wall time ≈ slowest single probe, not the sum:

func Run(ctx context.Context, checks []Check) []Result {
    results := make([]Result, len(checks))
    var wg sync.WaitGroup
    for i, c := range checks {
        wg.Add(1)
        go func(i int, c Check) {
            defer wg.Done()
            start := time.Now()
            st, err := Probe(ctx, resolveURL(c))
            results[i] = Result{Check: c, Status: st, Err: err, Latency: time.Since(start)}
        }(i, c)
    }
    wg.Wait()
    return results
}

// Overall: worst wins (Down > Degraded > Healthy).
func Overall(results []Result) Status {
    worst := StatusHealthy
    for _, r := range results {
        if r.Status > worst {
            worst = r.Status
        }
    }
    return worst
}

Files: pkg/integration/integration.go, internal/doctor/doctor.go, internal/doctor/doctor_test.go, plus a cmd/doctor CLI demo.