Answer

Rootless install: /run/user/<uid> disappears between MkdirAll and chown

Problem class: go-spawn-rootless-install-chown-runtime-dir-missing-run-user

Symptom

A rootless install/spawn path does roughly:

os.MkdirAll("/run/user/"+uid, 0o700)
os.Chown("/run/user/"+uid, uid, gid) // <-- ENOENT

and intermittently fails with:

chown /run/user/1001: no such file or directory

The directory was created moments earlier, so the failure looks impossible.

Root cause

systemd-logind owns /run/user/<uid> through user-runtime-dir@<uid>.service. When a session for a recycled uid is being torn down (e.g. a new container/user reused uid 1001 while the old session is still stopping), logind removes /run/user/1001 asynchronously.

The race window is exactly between our MkdirAll and our Chown:

us:      MkdirAll(/run/user/1001)  ───────────────►  Chown(...)  ── ENOENT
logind:                          RemoveAll(/run/user/1001)

The teardown window is seconds-scale (a user@.service stop can take a while). A flat, micro-pause retry budget (e.g. 10 × 10 ms ≈ 100 ms) never reaches it. Also note: with N attempts there are only N-1 inter-attempt pauses — the budget that must cover the teardown is the sum, not the per-attempt pause.

The fix

Re-assert MkdirAll + non-recursive chown under a bounded, exponentially growing pause schedule whose total spans the observed teardown window, and make each wait context-abortable.

runtimedir/runtimedir.go

// Package runtimedir creates and owns the per-uid XDG runtime directory
// (/run/user/<uid>) for rootless install/spawn flows.
package runtimedir

import (
    "context"
    "errors"
    "fmt"
    "io/fs"
    "os"
    "path/filepath"
    "strconv"
    "time"
)

// Tuning: 3 attempts => 2 inter-attempt waits.
// 600ms * 4^(n-1) = 600ms, 2400ms => 3.0s total, spanning a logind stop window.
const (
    runtimeDirAttempts  = 3
    runtimeDirBasePause = 600 * time.Millisecond
    runtimeDirMaxShift  = 6 // cap 4^(n-1) so the shift can never overflow
)

// runtimeDirBase is /run/user in production; overridable in tests.
var runtimeDirBase = "/run/user"

// testHookAfterMkdir, when non-nil, runs after MkdirAll and before Chown.
// It lets tests inject a logind-style teardown deterministically.
var testHookAfterMkdir func()

// EnsureRuntimeDir creates base/<uid> with mode 0700, owned by uid:gid.
//
// systemd-logind's user-runtime-dir@<uid>.service may be tearing down a
// *recycled* uid at the same time and can unlink the directory after our
// MkdirAll but before our Chown. We therefore re-assert (MkdirAll + Chown)
// under a bounded, exponentially growing pause schedule whose total spans the
// observed teardown window, and abort promptly on ctx cancellation.
func EnsureRuntimeDir(ctx context.Context, uid, gid int) (string, error) {
    dir := filepath.Join(runtimeDirBase, strconv.Itoa(uid))

    var lastErr error
    for attempt := 1; attempt <= runtimeDirAttempts; attempt++ {
        lastErr = assertRuntimeDir(dir, uid, gid)
        if lastErr == nil {
            return dir, nil
        }
        // Only ENOENT is caused by the teardown race. EPERM/ENOTDIR/...
        // will not be fixed by waiting for logind.
        if !errors.Is(lastErr, fs.ErrNotExist) {
            return dir, fmt.Errorf("assert runtime dir %s: %w", dir, lastErr)
        }
        if attempt == runtimeDirAttempts {
            break
        }
        if err := sleepBackoff(ctx, attempt); err != nil {
            return dir, fmt.Errorf("assert runtime dir %s: %w (last attempt: %v)", dir, err, lastErr)
        }
    }
    return dir, fmt.Errorf("assert runtime dir %s: %w", dir, lastErr)
}

// assertRuntimeDir performs the non-recursive re-assertion.
func assertRuntimeDir(dir string, uid, gid int) error {
    if err := os.MkdirAll(dir, 0o700); err != nil {
        return err
    }
    if testHookAfterMkdir != nil {
        testHookAfterMkdir()
    }
    // Non-recursive on purpose: only the runtime dir itself is ours.
    // Walking into children would race the session that owns them.
    return os.Chown(dir, uid, gid)
}

// sleepBackoff waits base * 4^(attempt-1), shift-capped, but returns
// immediately if ctx is cancelled.
func sleepBackoff(ctx context.Context, attempt int) error {
    shift := attempt - 1
    if shift > runtimeDirMaxShift {
        shift = runtimeDirMaxShift
    }
    pause := runtimeDirBasePause * time.Duration(uint64(1)<<uint(shift))

    t := time.NewTimer(pause)
    defer t.Stop()
    select {
    case <-ctx.Done():
        return ctx.Err()
    case <-t.C:
        return nil
    }
}

Call it from the install path instead of the bare MkdirAll/Chown pair:

ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()

dir, err := runtimedir.EnsureRuntimeDir(ctx, uid, gid)
if err != nil {
    return fmt.Errorf("prepare runtime dir: %w", err)
}

Why these numbers

attempt	4^(n-1)	pause	cumulative wait
1	1	0.6 s	0.0 s (initial try)
2	4	2.4 s	0.6 s
3	—	—	3.0 s total

• The total is 3.0 s, not 0.6 s — that is what covers the teardown window.
• N = 3 attempts gives N-1 = 2 pauses. Do not confuse per-attempt pause with budget.
• runtimeDirMaxShift caps the exponent so the shift is safe if the attempt count is ever raised.
• sleepBackoff selects on ctx.Done() so a cancelled install does not sit in a 2.4 s sleep.

Verification

Deterministic test (injects the teardown)

runtimedir/runtimedir_test.go:

package runtimedir

import (
    "context"
    "errors"
    "os"
    "path/filepath"
    "strconv"
    "sync/atomic"
    "syscall"
    "testing"
    "time"
)

func withTempBase(t *testing.T) string {
    t.Helper()
    old := runtimeDirBase
    base := t.TempDir()
    runtimeDirBase = base
    t.Cleanup(func() { runtimeDirBase = old })
    return base
}

func TestEnsureRuntimeDirReassertsAfterTeardown(t *testing.T) {
    base := withTempBase(t)
    uid, gid := os.Getuid(), os.Getgid()
    dir := filepath.Join(base, strconv.Itoa(uid))

    var calls int32
    testHookAfterMkdir = func() {
        // Reproduce logind: delete the dir between MkdirAll and Chown,
        // but only on the first attempt.
        if atomic.AddInt32(&calls, 1) == 1 {
            _ = os.RemoveAll(dir)
        }
    }
    t.Cleanup(func() { testHookAfterMkdir = nil })

    got, err := EnsureRuntimeDir(context.Background(), uid, gid)
    if err != nil {
        t.Fatalf("EnsureRuntimeDir: %v", err)
    }
    if got != dir {
        t.Fatalf("dir = %q, want %q", got, dir)
    }
    if n := atomic.LoadInt32(&calls); n < 2 {
        t.Fatalf("expected >=2 attempts after teardown, got %d", n)
    }

    var st syscall.Stat_t
    if err := syscall.Stat(got, &st); err != nil {
        t.Fatalf("stat: %v", err)
    }
    if int(st.Uid) != uid || int(st.Gid) != gid {
        t.Fatalf("owner = %d:%d, want %d:%d", st.Uid, st.Gid, uid, gid)
    }
}

func TestEnsureRuntimeDirAbortsOnContext(t *testing.T) {
    base := withTempBase(t)
    uid, gid := os.Getuid(), os.Getgid()
    dir := filepath.Join(base, strconv.Itoa(uid))

    testHookAfterMkdir = func() { _ = os.RemoveAll(dir) } // always tear down
    t.Cleanup(func() { testHookAfterMkdir = nil })

    ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
    defer cancel()

    start := time.Now()
    _, err := EnsureRuntimeDir(ctx, uid, gid)
    if !errors.Is(err, context.DeadlineExceeded) {
        t.Fatalf("err = %v, want context.DeadlineExceeded", err)
    }
    if d := time.Since(start); d > time.Second {
        t.Fatalf("context abort took %s, want < 1s", d)
    }
}

func TestBackoffTotalSpansTeardownWindow(t *testing.T) {
    // 600ms + 2400ms = 3000ms across the two inter-attempt waits.
    var total time.Duration
    for attempt := 1; attempt < runtimeDirAttempts; attempt++ {
        shift := attempt - 1
        if shift > runtimeDirMaxShift {
            shift = runtimeDirMaxShift
        }
        total += runtimeDirBasePause * time.Duration(uint64(1)<<uint(shift))
    }
    if total != 3*time.Second {
        t.Fatalf("total backoff = %s, want 3s", total)
    }
}

Run it:

go test ./runtimedir/ -run 'TestEnsureRuntimeDir|TestBackoffTotal' -v -count=1
go test -race ./runtimedir/ -count=50

Expected: all pass; the teardown test reports calls >= 2, proving the re-assertion happened.

Reproducing the original failure

Revert to the naive pair and the deterministic test fails on the first attempt:

// naive, racy
_ = os.MkdirAll(dir, 0o700)
if err := os.Chown(dir, uid, gid); err != nil {
    return err // ENOENT under teardown
}

The injected teardown deletes the dir right after MkdirAll, so Chown returns ENOENT and there is no retry.

End-to-end smoke check (as root)

# Watch logind own the dir, then confirm our helper converges.
systemctl status "user-runtime-dir@$(id -u).service" || true
journalctl -u "user@$(id -u).service" --since '-5 min' | tail -n 50

install -d -m 0700 "/run/user/$(id -u)"
stat -c '%U %G %a %n' "/run/user/$(id -u)"
# expect: <user> <group> 700 /run/user/<uid>

Lesson

• A flat micro-pause budget (e.g. 10 × 10ms ≈ 100ms) is unreachable against seconds-scale user@.service stop windows.
• With N attempts there are only N-1 pauses; compute the total wait, not the per-attempt value.
• Keep the chown non-recursive: only the runtime dir itself is being re-asserted; children belong to the live session.

# Evidence
- Problem class: go-spawn-rootless-install-chown-runtime-dir-missing-run-user
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-10-03T11:35:16.365Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "go-spawn-rootless-install-chown-runtime-dir-missing-run-user", "provider": "openrouter", "solved_at": "2026-10-03T11:35:16.367Z", "version": ""}