go-e2e-battery-harness-grep

Answer

Root cause. The battery harness grepped the live webhook body for a hand-written literal (bytes.Contains(body, []byte("42"))). The flow's own source is print(1), so the runtime's byte-exact response is {"result":"1"} with HTTP 200. The FAIL was a harness bug (wrong fixed literal + substring grep), not an app regression.

Fix (3 rules).

1. Derive the expectation from the flow's own source — parse the flow's print(...) and evaluate it with the same formatting rules the runtime uses. Never a fixed literal, never a Want field in the test spec.
2. Assert byte-exact equality — bytes.Equal over raw io.ReadAll bytes. No substring grep, no JSON unmarshal/remarshal (reformatting would break byte-exactness), and reuse the server's exact string serializer so the two sides can't drift.
3. Re-verify before counting — every FAIL is re-checked against the flow's own derived output; a FAIL that matches its own flow's output is a stale-literal harness bug and is reclassified PASS.

// Step 1: evaluate the flow's own print(...). Ambiguous sources (zero or
// multiple prints) are an error, never a guess — a guess reintroduces the trap.
var printRe = regexp.MustCompile(`print\(\s*(.*?)\s*\)`)

func printedValue(src string) (string, error) {
    matches := printRe.FindAllStringSubmatch(src, -1)
    if len(matches) != 1 {
        return "", fmt.Errorf("flow source must contain exactly one print(...): %q", src)
    }
    arg := strings.TrimSpace(matches[0][1])
    if s, err := strconv.Unquote(arg); err == nil {
        return s, nil // print("hi") -> hi
    }
    if f, err := strconv.ParseFloat(arg, 64); err == nil {
        return strconv.FormatFloat(f, 'f', -1, 64), nil // print(1.50) -> "1.5"
    }
    return arg, nil // raw token, visible on mismatch instead of guessed
}

// Step 2: byte-exact body from the flow's own source. The same jsonQuoted
// serializer the server uses — single source of truth, so they cannot diverge.
func flowExpectation(f Flow) ([]byte, error) {
    v, err := printedValue(f.Source)
    if err != nil {
        return nil, err
    }
    return []byte(`{"result":` + jsonQuoted(v) + `}`), nil
}

// Step 3: assert raw bytes, check status, no reformatting.
func assertFlow(baseURL string, f Flow) error {
    want, err := flowExpectation(f)
    if err != nil {
        return err
    }
    resp, err := http.Post(baseURL+"/hooks/"+f.ID, "application/json", strings.NewReader("{}"))
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    got, err := io.ReadAll(resp.Body) // raw bytes only
    if err != nil {
        return err
    }
    if resp.StatusCode != http.StatusOK {
        return fmt.Errorf("status %d, want 200", resp.StatusCode)
    }
    if !bytes.Equal(got, want) {
        return fmt.Errorf("byte-exact mismatch: want %q (from source %q) got %q", want, f.Source, got)
    }
    return nil
}

// Battery rule: gate every FAIL through this before counting it.
func reVerify(baseURL string, f Flow) (bool, string) {
    if err := assertFlow(baseURL, f); err != nil {
        return false, "genuine failure: " + err.Error()
    }
    return true, "re-verified PASS: output matches the flow's own source-derived expectation"
}

Structural guard: test specs carry no Want/Expected literal field — TestCase{Flow} only, so the harness and the app share one contract. print(1) print(2) or x = 1 fail fast as ambiguous rather than being guessed.