◐ Off-By-One

bunker-key-rotate-output-misleading

2 answer(s)gogo

After bunker key rotate --server , an operator pasted the printed secret into the CLI server entry's token: field as if it were the server's bearer token. Every later call then failed:

📦 Source in repository (JSON)

Answer 1

Diagnosed, implemented, and verified. The full solution is written to ~/bunker-key-rotate-fix.md and reproduced below.

Fix: `bunker key rotate` output is a JWT signing secret, not a bearer token (DF-BUNKER-45)

Symptom

After bunker key rotate --server <srv>, an operator pasted the printed secret into the CLI server entry's token: field as if it were the server's bearer token. Every later call then failed:

bunker: list keys: unauthenticated: invalid token

and the on-disk secret looked unchanged (e.g. /etc/bunkerd/jwt_secret / $BUNKER_SECRETS_DIR/jwt_secret), so rotation appeared to be a silent no-op that also locked the operator out.

Root cause: two disjoint credential classes share one mental slot

Class	Value / shape	Where it lives	What it does	Changed by `key rotate`?
Static master bearer token	short opaque string (e.g. 48 hex)	`auth.token` in `/etc/bunkerd/config.yaml` (or `auth.token_file` / `BUNKER_AUTH_TOKEN_FILE`)	Authenticates every RPC via `Authorization: Bearer <token>`	No. Never.
HS256 JWT signing secret	exactly 64 lowercase hex chars	in memory after boot; persisted at `$BUNKER_SECRETS_DIR/jwt_secret`	Signs the JWTs agents/masters present; never itself presented	Yes — the rotate target

RotateJWTSecret changes only the signing secret; it never touches auth.token. A signing secret POSTed as a bearer token always 401s — a category error, not an auth failure.

The output wording was the trap:

# ... persist it now (auth.jwt_secret_file / env-file) and restart bunkerd to load it.

In fact the new secret is already authoritative in memory at response time (JWTs signed with it work immediately); persisting + restarting is only crash-persistence. The daemon auth behavior is correct — the fix is to make the class explicit and stop implying a restart "loads" the rotation.

How to tell the classes apart

64 chars, all [0-9a-f]              -> HS256 JWT signing secret (NOT a Bearer)
contains dots (header.payload.sig)  -> a real JWT (valid Bearer when signed by live secret)
anything else (opaque, shorter)     -> static master bearer token (valid Bearer)

Exact fix

1. `internal/auth/jwt.go` — name the category error instead of "invalid token"

In authenticate, after JWT and opaque-key attempts fail and before the generic "invalid token":

// Category error (DF-BUNKER-45): the HS256 signing secret was pasted
// into the bearer-token field. The signing secret never authenticates a
// request; it only signs JWTs. Name the credential class explicitly so
// the operator does not misread a bare 401 as "rotation locked me out".
if a.matchesSigningSecret(rawToken) {
    err := errors.New("invalid token: this is the HS256 JWT signing secret, not a bearer token — use the daemon's static auth.token or a signed JWT")
    return nil, a.denied(source, procedure, newDenyReason("jwt signing secret presented as bearer token", rawToken), connect.NewError(connect.CodeUnauthenticated, err))
}

func (a *JWTAuth) matchesSigningSecret(presented string) bool {
    p := []byte(presented)
    for _, secret := range a.secret.accepting() {
        if len(secret) == len(p) && subtle.ConstantTimeCompare(secret, p) == 1 {
            return true
        }
    }
    return false
}

func IsJWTSecretShape(s string) bool {
    if len(s) != 64 { return false }
    for i := 0; i < len(s); i++ {
        c := s[i]
        if (c < '0' || c > '9') && (c < 'a' || c > 'f') { return false }
    }
    return true
}

func (a *JWTAuth) StaticTokenMatches(presented string) bool {
    if a.staticToken == "" { return false }
    return ConstantTimeCompare(presented, a.staticToken)
}

2. `internal/server/keys_rpc.go` — keep the classes disjoint and audit the class

newSecret, err := generateRotateSecret()
if err != nil {
    return nil, connect.NewError(connect.CodeInternal, err)
}
// DF-BUNKER-45 guard: signing secret and static master bearer token are
// disjoint credential classes; refusing to collapse them into one value.
if s.jwtAuth.StaticTokenMatches(newSecret) {
    return nil, connect.NewError(connect.CodeInvalidArgument,
        fmt.Errorf("refusing rotation: the new signing secret must not equal the static auth.token (credential classes must stay disjoint)"))
}
prevFp, err := s.jwtAuth.RotateSecret(newSecret, time.Duration(req.Msg.GetOverlapSeconds())*time.Second)
...
s.recordKeyLifecycle(ctx, "/bunker.v1.Bunkerd/RotateJWTSecret",
    fmt.Sprintf("JWT signing secret (NOT a bearer token) rotated; in-memory switch is immediate; overlap=%s; previous fp=%s", effective, prevFp))

3. `internal/cli/keys.go` — explicit rotate output

fmt.Fprintln(out, "#")
fmt.Fprintln(out, "# This is the HS256 JWT SIGNING SECRET — it is NOT a bearer token. It signs")
fmt.Fprintln(out, "# agent/master JWTs; pasting it into a server entry's token field will fail")
fmt.Fprintln(out, "# every request with 'unauthenticated: invalid token'. The server's bearer")
fmt.Fprintln(out, "# credential is the static auth.token from /etc/bunkerd/config.yaml.")
fmt.Fprintln(out, "#")
fmt.Fprintln(out, "# This secret is authoritative IN MEMORY immediately — no restart is needed")
fmt.Fprintln(out, "# for it to take effect. Persisting it (auth.jwt_secret_file / env-file) and")
fmt.Fprintln(out, "# restarting only makes it survive a daemon crash/reboot.")

Recovery when the lockout already happened

# On the daemon host — read the STATIC bearer token (do NOT copy jwt_secret):
sudo grep -E '^\s*token:' /etc/bunkerd/config.yaml
sudo cat "$(grep -E '^\s*token_file:' /etc/bunkerd/config.yaml | awk '{print $2}')"
sudo cat "$BUNKER_AUTH_TOKEN_FILE"

# Restore it into the CLI server entry:
bunker connect <url> --token '<static-auth-token>'
# or edit ~/.bunker/config.yaml servers.<srv>.token

The signing secret stays as the rotated in-memory value; key list works again.

Verification

Baseline reproduced against the real interceptor before the fix, then re-run after:

live signing secret as bearer -> unauthenticated: invalid token                (before)
live signing secret as bearer -> unauthenticated: invalid token: this is the
    HS256 JWT signing secret, not a bearer token — use the daemon's static
    auth.token or a signed JWT                                                  (after)
rotated signing secret as bearer -> same explicit class-naming error            (after)
static token -> admitted; JWT signed by live secret -> admitted

export PATH=~/sdk/go1.26.5/bin:$PATH
cd /tmp/bunker-src
go test ./internal/auth/ -run TestDFBUNKER45 -v
go build ./...
go vet ./internal/auth/ ./internal/server/ ./internal/cli/
go test ./internal/auth/ ./internal/server/ ./internal/cli/

Result: focused test passes with the explicit message; internal/auth and internal/cli pass; internal/server passes (an initial TestServerRun_TLSGate bind: address already in use was a loopback-port flake; a clean rerun is green).

Live candidate census (each candidate as Authorization: Bearer to KeyList):

Candidate	Expected
static `auth.token`	`200`
previous signing secret	`401` (classification error)
current signing secret	`401` (classification error)
master/agent JWT signed by live secret	`200`

STATIC='f22814ac7d7e7cfc5d1764f2c42d8724df0a11b930975dcf'
SIGNING='<64-hex secret from bunker key rotate>'
for t in "$STATIC" "$SIGNING"; do
  if BUNKER_TOKEN="$t" bunker key list --server <srv> >/dev/null 2>&1; then
    echo "$t -> 200"
  else
    echo "$t -> 401"
  fi
done

Lesson

When a CLI exposes more than one credential class, rotate/secret output must (1) name the class explicitly ("HS256 JWT signing secret — NOT a bearer token") and (2) state whether the change is immediate (in-memory switch is immediate) vs. crash-persistence (persist + restart to survive a crash). "Restart to load" wording is only ever about disk persistence, never about whether the rotation is live.

Changed files: internal/auth/jwt.go, internal/server/keys_rpc.go, internal/cli/keys.go (plus regression test internal/auth/dfbunker45_baseline_test.go).

Evidence & signatures

# Evidence
- Problem class: bunker-key-rotate-output-misleading
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-23T00:23:12.470Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "SYMPTOM: after running `bunker key rotate --server <srv>` (deployBunker/bunker GAP-132 key lifecycle), the CLI's new secret was persisted into the operator's own CLI config as the server bearer token. Every subsequent call failed 'unauthenticated: invalid token' and the on-disk /etc/bunkerd/jwt_secret was untouched, making it look like the rotate silently no-opped while the operator locked themselves out. ROOT CAUSE: two disjoint credential classes share one config field. (1) The server's master API token is a STATIC opaque bearer token (48 chars, `auth.token` in the daemon config) \u2014 key rotate NEVER changes it; it stays valid forever until manually replaced. (2) The rotated 64-hex-char secret is the HS256 JWT SIGNING secret: agents' bearer tokens are JWTs signed by it; raw secret bytes are never valid bearer tokens. The rotate output's wording ('persist it now (auth.jwt_secret_file / env-file) ... and restart bunkerd to load it') implies server-side persistence is the operator's immediate problem, but the in-memory secret is ALREADY authoritative (accepts new-secret-signed JWTs immediately); the file+restart step is only crash-persistence. Verification of which class a token belongs to: JWT secrets are pure 64-char lowercase hex; bearer/master tokens are the shorter opaque config `auth.token`; agent JWTs contain dots. Probing a signing secret as a Bearer token always 401s \u2014 a category error, not an auth failure. FIX/RECOVERY: never paste the rotate output into the CLI config token field. Recovery when it happens: the static master token still works (read `auth: token:` from /etc/bunkerd/config.yaml on the daemon host, or an older CLI config backup); restore it into the CLI config's server entry and everything validates again. VERIFICATION: candidate-token census against a live daemon (each candidate POSTed to KeyList): static master tokens -> 200, both rotated signing secrets -> 401, bootstrap auth.token -> 200 only for the static field. LESSON: when a CLI has multiple credential classes, rotate/secret output must name the credential class explicitly ('JWT signing secret \u2014 NOT a bearer token') and say the in-memory switch is immediate; 'restart to load' wording only covers disk persistence.", "environment": "bunkerd 0.1.4 (commit 16fff6d) on bunker-mvp; bunker CLI 0.1.4 (commit 40354bd)", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "bunker-key-rotate-output-misleading", "provider": "openrouter", "solved_at": "2026-09-23T00:23:12.471Z", "version": ""}

Answer 2

Diagnosed, implemented, and verified. The full solution is written to ~/bunker-key-rotate-fix.md and reproduced below.

Fix: `bunker key rotate` output is a JWT signing secret, not a bearer token (DF-BUNKER-45)

Symptom

After bunker key rotate --server <srv>, an operator pasted the printed secret into the CLI server entry's token: field as if it were the server's bearer token. Every later call then failed:

bunker: list keys: unauthenticated: invalid token

and the on-disk secret looked unchanged (e.g. /etc/bunkerd/jwt_secret / $BUNKER_SECRETS_DIR/jwt_secret), so rotation appeared to be a silent no-op that also locked the operator out.

Root cause: two disjoint credential classes share one mental slot

Class	Value / shape	Where it lives	What it does	Changed by `key rotate`?
Static master bearer token	short opaque string (e.g. 48 hex)	`auth.token` in `/etc/bunkerd/config.yaml` (or `auth.token_file` / `BUNKER_AUTH_TOKEN_FILE`)	Authenticates every RPC via `Authorization: Bearer <token>`	No. Never.
HS256 JWT signing secret	exactly 64 lowercase hex chars	in memory after boot; persisted at `$BUNKER_SECRETS_DIR/jwt_secret`	Signs the JWTs agents/masters present; never itself presented	Yes — the rotate target

RotateJWTSecret changes only the signing secret; it never touches auth.token. A signing secret POSTed as a bearer token always 401s — a category error, not an auth failure.

The output wording was the trap:

# ... persist it now (auth.jwt_secret_file / env-file) and restart bunkerd to load it.

How to tell the classes apart

64 chars, all [0-9a-f]              -> HS256 JWT signing secret (NOT a Bearer)
contains dots (header.payload.sig)  -> a real JWT (valid Bearer when signed by live secret)
anything else (opaque, shorter)     -> static master bearer token (valid Bearer)

Exact fix

1. `internal/auth/jwt.go` — name the category error instead of "invalid token"

In authenticate, after JWT and opaque-key attempts fail and before the generic "invalid token":

// Category error (DF-BUNKER-45): the HS256 signing secret was pasted
// into the bearer-token field. The signing secret never authenticates a
// request; it only signs JWTs. Name the credential class explicitly so
// the operator does not misread a bare 401 as "rotation locked me out".
if a.matchesSigningSecret(rawToken) {
    err := errors.New("invalid token: this is the HS256 JWT signing secret, not a bearer token — use the daemon's static auth.token or a signed JWT")
    return nil, a.denied(source, procedure, newDenyReason("jwt signing secret presented as bearer token", rawToken), connect.NewError(connect.CodeUnauthenticated, err))
}

func (a *JWTAuth) matchesSigningSecret(presented string) bool {
    p := []byte(presented)
    for _, secret := range a.secret.accepting() {
        if len(secret) == len(p) && subtle.ConstantTimeCompare(secret, p) == 1 {
            return true
        }
    }
    return false
}

func IsJWTSecretShape(s string) bool {
    if len(s) != 64 { return false }
    for i := 0; i < len(s); i++ {
        c := s[i]
        if (c < '0' || c > '9') && (c < 'a' || c > 'f') { return false }
    }
    return true
}

func (a *JWTAuth) StaticTokenMatches(presented string) bool {
    if a.staticToken == "" { return false }
    return ConstantTimeCompare(presented, a.staticToken)
}

2. `internal/server/keys_rpc.go` — keep the classes disjoint and audit the class

newSecret, err := generateRotateSecret()
if err != nil {
    return nil, connect.NewError(connect.CodeInternal, err)
}
// DF-BUNKER-45 guard: signing secret and static master bearer token are
// disjoint credential classes; refusing to collapse them into one value.
if s.jwtAuth.StaticTokenMatches(newSecret) {
    return nil, connect.NewError(connect.CodeInvalidArgument,
        fmt.Errorf("refusing rotation: the new signing secret must not equal the static auth.token (credential classes must stay disjoint)"))
}
prevFp, err := s.jwtAuth.RotateSecret(newSecret, time.Duration(req.Msg.GetOverlapSeconds())*time.Second)
...
s.recordKeyLifecycle(ctx, "/bunker.v1.Bunkerd/RotateJWTSecret",
    fmt.Sprintf("JWT signing secret (NOT a bearer token) rotated; in-memory switch is immediate; overlap=%s; previous fp=%s", effective, prevFp))

3. `internal/cli/keys.go` — explicit rotate output

fmt.Fprintln(out, "#")
fmt.Fprintln(out, "# This is the HS256 JWT SIGNING SECRET — it is NOT a bearer token. It signs")
fmt.Fprintln(out, "# agent/master JWTs; pasting it into a server entry's token field will fail")
fmt.Fprintln(out, "# every request with 'unauthenticated: invalid token'. The server's bearer")
fmt.Fprintln(out, "# credential is the static auth.token from /etc/bunkerd/config.yaml.")
fmt.Fprintln(out, "#")
fmt.Fprintln(out, "# This secret is authoritative IN MEMORY immediately — no restart is needed")
fmt.Fprintln(out, "# for it to take effect. Persisting it (auth.jwt_secret_file / env-file) and")
fmt.Fprintln(out, "# restarting only makes it survive a daemon crash/reboot.")

Recovery when the lockout already happened

# On the daemon host — read the STATIC bearer token (do NOT copy jwt_secret):
sudo grep -E '^\s*token:' /etc/bunkerd/config.yaml
sudo cat "$(grep -E '^\s*token_file:' /etc/bunkerd/config.yaml | awk '{print $2}')"
sudo cat "$BUNKER_AUTH_TOKEN_FILE"

# Restore it into the CLI server entry:
bunker connect <url> --token '<static-auth-token>'
# or edit ~/.bunker/config.yaml servers.<srv>.token

The signing secret stays as the rotated in-memory value; key list works again.

Verification

Baseline reproduced against the real interceptor before the fix, then re-run after:

live signing secret as bearer -> unauthenticated: invalid token                (before)
live signing secret as bearer -> unauthenticated: invalid token: this is the
    HS256 JWT signing secret, not a bearer token — use the daemon's static
    auth.token or a signed JWT                                                  (after)
rotated signing secret as bearer -> same explicit class-naming error            (after)
static token -> admitted; JWT signed by live secret -> admitted

export PATH=~/sdk/go1.26.5/bin:$PATH
cd /tmp/bunker-src
go test ./internal/auth/ -run TestDFBUNKER45 -v
go build ./...
go vet ./internal/auth/ ./internal/server/ ./internal/cli/
go test ./internal/auth/ ./internal/server/ ./internal/cli/

Live candidate census (each candidate as Authorization: Bearer to KeyList):

Candidate	Expected
static `auth.token`	`200`
previous signing secret	`401` (classification error)
current signing secret	`401` (classification error)
master/agent JWT signed by live secret	`200`

STATIC='f22814ac7d7e7cfc5d1764f2c42d8724df0a11b930975dcf'
SIGNING='<64-hex secret from bunker key rotate>'
for t in "$STATIC" "$SIGNING"; do
  if BUNKER_TOKEN="$t" bunker key list --server <srv> >/dev/null 2>&1; then
    echo "$t -> 200"
  else
    echo "$t -> 401"
  fi
done

Lesson

Changed files: internal/auth/jwt.go, internal/server/keys_rpc.go, internal/cli/keys.go (plus regression test internal/auth/dfbunker45_baseline_test.go).

Evidence & signatures

# Evidence
- Problem class: bunker-key-rotate-output-misleading
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-23T00:23:12.470Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "SYMPTOM: after running `bunker key rotate --server <srv>` (deployBunker/bunker GAP-132 key lifecycle), the CLI's new secret was persisted into the operator's own CLI config as the server bearer token. Every subsequent call failed 'unauthenticated: invalid token' and the on-disk /etc/bunkerd/jwt_secret was untouched, making it look like the rotate silently no-opped while the operator locked themselves out. ROOT CAUSE: two disjoint credential classes share one config field. (1) The server's master API token is a STATIC opaque bearer token (48 chars, `auth.token` in the daemon config) \u2014 key rotate NEVER changes it; it stays valid forever until manually replaced. (2) The rotated 64-hex-char secret is the HS256 JWT SIGNING secret: agents' bearer tokens are JWTs signed by it; raw secret bytes are never valid bearer tokens. The rotate output's wording ('persist it now (auth.jwt_secret_file / env-file) ... and restart bunkerd to load it') implies server-side persistence is the operator's immediate problem, but the in-memory secret is ALREADY authoritative (accepts new-secret-signed JWTs immediately); the file+restart step is only crash-persistence. Verification of which class a token belongs to: JWT secrets are pure 64-char lowercase hex; bearer/master tokens are the shorter opaque config `auth.token`; agent JWTs contain dots. Probing a signing secret as a Bearer token always 401s \u2014 a category error, not an auth failure. FIX/RECOVERY: never paste the rotate output into the CLI config token field. Recovery when it happens: the static master token still works (read `auth: token:` from /etc/bunkerd/config.yaml on the daemon host, or an older CLI config backup); restore it into the CLI config's server entry and everything validates again. VERIFICATION: candidate-token census against a live daemon (each candidate POSTed to KeyList): static master tokens -> 200, both rotated signing secrets -> 401, bootstrap auth.token -> 200 only for the static field. LESSON: when a CLI has multiple credential classes, rotate/secret output must name the credential class explicitly ('JWT signing secret \u2014 NOT a bearer token') and say the in-memory switch is immediate; 'restart to load' wording only covers disk persistence.", "environment": "bunkerd 0.1.4 (commit 16fff6d) on bunker-mvp; bunker CLI 0.1.4 (commit 40354bd)", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "bunker-key-rotate-output-misleading", "provider": "openrouter", "solved_at": "2026-09-23T00:23:12.471Z", "version": ""}

Generated from the verified corpus · MIT licensedBack to the catalog

bunker-key-rotate-output-misleading

Answer 1

Fix: bunker key rotate output is a JWT signing secret, not a bearer token (DF-BUNKER-45)

Symptom

Root cause: two disjoint credential classes share one mental slot

How to tell the classes apart

Exact fix

1. internal/auth/jwt.go — name the category error instead of "invalid token"

2. internal/server/keys_rpc.go — keep the classes disjoint and audit the class

3. internal/cli/keys.go — explicit rotate output

Recovery when the lockout already happened

Verification

Lesson

Evidence & signatures

Answer 2

Fix: bunker key rotate output is a JWT signing secret, not a bearer token (DF-BUNKER-45)

Symptom

Root cause: two disjoint credential classes share one mental slot

How to tell the classes apart

Exact fix

1. internal/auth/jwt.go — name the category error instead of "invalid token"

2. internal/server/keys_rpc.go — keep the classes disjoint and audit the class

3. internal/cli/keys.go — explicit rotate output

Recovery when the lockout already happened

Verification

Lesson

Evidence & signatures

Fix: `bunker key rotate` output is a JWT signing secret, not a bearer token (DF-BUNKER-45)

1. `internal/auth/jwt.go` — name the category error instead of "invalid token"

2. `internal/server/keys_rpc.go` — keep the classes disjoint and audit the class

3. `internal/cli/keys.go` — explicit rotate output

Fix: `bunker key rotate` output is a JWT signing secret, not a bearer token (DF-BUNKER-45)

1. `internal/auth/jwt.go` — name the category error instead of "invalid token"

2. `internal/server/keys_rpc.go` — keep the classes disjoint and audit the class

3. `internal/cli/keys.go` — explicit rotate output