Defect at: 39f44aa (observed), fixed design bb751f5
Diagnosis complete and verified. I cloned deployBunker/bunker, located the exact upstream fix commit bb751f5 (INT-CI-008), checked it out, and ran the regression suite. The full solution is written to ~/SOLUTION.md. Here it is:
user@<uid>.service cannot start when /run/user/<uid> is missingRepo: github.com/deployBunker/bunker
File: internal/agent/rootless.go
Defect at: 39f44aa (observed), fixed design bb751f5
Class: systemd-user-manager-cannot-start-without-runtime-dir
Environment: Linux + systemd 255 (Ubuntu 24.04); root daemon provisioning ephemeral per-agent users and rootless Docker; UIDs recycled after userdel; linger keeps old managers alive.
spawn <agent> fails at stage rootless-install in well under a second (0.6 s on the regression job) with:
user-manager-start failed: unit <email>: is-active=failed
(Result=Job for <email> failed because the control process exited
with error code. See "systemctl status <email>" ...):
linger_entries=135
The auto-generated-ID spawn 12 s later passes; only an explicit, previously used agent ID fails. systemctl job Result (control process exited) does not name the cause.
Verbatim journal lines that actually explain the failure:
pam_systemd(systemd-user:session): Failed to stat() runtime directory '/run/user/1002': No such file or directory
pam_systemd(systemd-user:session): Not setting XDG_RUNTIME_DIR, as the directory is not in order.
<email>: Main process exited, code=exited, status=1/FAILURE
On systemd 255, /usr/lib/systemd/system/user@.service carries no Requires=/Wants=/After= on user-runtime-dir@<uid>.service:
# user@.service (systemd 255)
[Unit]
Description=User Manager for UID %i
BindsTo=user-runtime-dir@%i.service # only on newer systemd
After=systemd-logind.service user-runtime-dir@%i.service # only on newer systemd
[Service]
User=%i
PAMName=systemd-user
ExecStart=/usr/lib/systemd/systemd --user
/run/user/<uid> is created by logind's own unit:
# user-runtime-dir@.service
[Service]
ExecStart=/usr/lib/systemd/systemd-user-runtime-dir start %i
ExecStop=/usr/lib/systemd/systemd-user-runtime-dir stop %i
Type=oneshot
RemainAfterExit=yes
StopWhenUnneeded=yes
Because it is RemainAfterExit=yes, once it has run it holds a fixed on-disk claim (active (exited)). Starting it again is a no-op even when the directory has been deleted out from under it. So nothing recreates the directory if it is removed while the unit is still active/deactivating.
user@<uid>.service starts with PAMName=systemd-user. pam_systemd's session step calls stat("/run/user/<uid>"); when it is absent it refuses to set XDG_RUNTIME_DIR. /usr/lib/systemd/systemd --user then has no runtime directory and exits 1. The unit lands in failed, and reset-failed + a retry never fixes it because the directory is still absent — the failure is self-reinforcing.
The old code in installRootlessDocker did, unconditionally, in this order:
systemctl stop user@<uid>.service
loginctl terminate-user <uid>
removeMountsUnder(/run/user/<uid>)
rm -rf /run/user/<uid>
enable-linger
ensureUserManagerRunning(...) <-- manager starts with dir MISSING
MkdirAll(/run/user/<uid>) <-- too late
chown <user>: /run/user/<uid>
This is precisely what creates the broken state:
/run/user/<uid> before the manager start, and systemd 255 will not recreate it.useradd may have no stale state of its own, while uid recycling means the previous owner's manager can still be live under that uid — so the unconditional stop/terminate-user/rm is destructive for no benefit.enable-linger is idempotent: if linger is already recorded, it takes no start action, so the unit stopped in step 1 stays down. The later passive wait then times out (or the start fails on the missing dir).rm -rf can also fail with Device or resource busy when the old manager left a gvfsd-fuse mount at /run/user/<uid>/gvfs.systemctl is-active / show -p Result only expose failed and exit-code. The line naming the cause lives in the unit journal (pam_systemd ... Failed to stat() runtime directory ...). Without it, operators see "control process exited" and cannot distinguish this failure from linger churn or a broken installer.
The bring-up becomes state-probing and directory-first:
/run/user/<uid> with Lstat; classify absent = FRESH → take no destructive action at all; anything else (not a dir, owner unknown, owner != uid, probe error) = STALE → fail-safe historical reset.loginctl terminate-user, lazily unmount leftovers, stop user-runtime-dir@<uid>.service so its RemainAfterExit state matches the filesystem, then remove the directory.systemctl start user-runtime-dir@<uid>.service (best effort; no-op when RemainAfterExit is already active — which is exactly why the explicit create is required as the fallback), then MkdirAll(0700), then a non-recursive chown <user>: <dir>, then verify exists ∧ isDir ∧ owner == uid; fail loudly otherwise.enable-linger, start the manager (with reset-failed + one retry), and wait for the bus socket.journalctl --no-pager -u <unit> -n 20 (fallback systemctl status --no-pager -n 20 <unit>) to the error and the WARN. The query runs on context.WithoutCancel(ctx) with its own timeout so attribution cannot fail with the caller's deadline.import (
"context"
"errors" // added
"fmt"
"log/slog"
"os"
"os/exec"
"os/user"
"path/filepath"
"strconv"
"strings"
"syscall" // added
"time"
"unicode/utf8" // added
)
installRootlessDockerDelete the block that starts at logger.Info("resetting user manager runtime", ...) and ends after the waitForUserManager call, and replace it with:
// Bring the systemd user manager up in a state-consistent order: reset
// stale state ONLY when there is real stale state, guarantee the runtime
// directory exists and is owned by the uid, then start the manager. The
// runtime directory MUST exist before user@<uid>.service starts, otherwise
// pam_systemd refuses XDG_RUNTIME_DIR and the manager can never go active
// (INT-CI-008).
if err := bringUpUserManager(ctx, username, uid, stdRuntimeDir, logger); err != nil {
return err
}
Keep the existing post-bring-up MkdirAll + non-recursive chown block as insurance for the installer step (it is no longer the only place the directory is created).
// userManagerUnitName is the per-uid systemd user manager unit.
func userManagerUnitName(uid int) string { return fmt.Sprintf("user@%d.service", uid) }
// userRuntimeDirUnitName is logind's runtime-directory unit for the same uid.
func userRuntimeDirUnitName(uid int) string {
return fmt.Sprintf("user-runtime-dir@%d.service", uid)
}
// runtimeDirInfo is the observable state of a systemd user runtime directory.
type runtimeDirInfo struct {
exists bool // false when the path is absent (the fresh-uid path)
isDir bool // false when the path exists but is not a directory
owner uint32 // uid owning the path
ownerKnown bool // false when the platform stat could not supply it
}
// runtimeDirProbe inspects a runtime directory. Package-level var (same seam
// style as userManagerRunner) because simulating a directory owned by a
// DIFFERENT uid needs a real chown to another user, which requires root.
var runtimeDirProbe = probeRuntimeDirOnDisk
// probeRuntimeDirOnDisk is the production probe. It uses Lstat so a symlink
// placed at the directory's path is NOT mistaken for a directory.
func probeRuntimeDirOnDisk(path string) (runtimeDirInfo, error) {
info, err := os.Lstat(path)
if err != nil {
if os.IsNotExist(err) {
return runtimeDirInfo{}, nil
}
return runtimeDirInfo{}, err
}
st := runtimeDirInfo{exists: true, isDir: info.IsDir()}
if sys, ok := info.Sys().(*syscall.Stat_t); ok {
st.owner = sys.Uid
st.ownerKnown = true
}
return st, nil
}
// classifyRuntimeDir decides whether the observed runtime directory carries
// state left behind by a PREVIOUS user of the same uid (which must be reset) or
// is the fresh path (which must not be touched). It fails SAFE.
func classifyRuntimeDir(info runtimeDirInfo, probeErr error, uid int) (bool, string) {
switch {
case probeErr != nil:
return true, "runtime dir probe failed: " + probeErr.Error()
case !info.exists:
return false, ""
case !info.isDir:
return true, "runtime path exists but is not a directory"
case !info.ownerKnown:
return true, "runtime dir ownership cannot be determined"
case info.owner != uint32(uid):
return true, fmt.Sprintf("runtime dir owned by uid %d, not %d", info.owner, uid)
}
return false, ""
}
// resetUserManagerState tears down stale user manager state for uid in a
// state-consistent order: stop the manager, stop logind's runtime-directory
// unit (so its RemainAfterExit state matches the filesystem), unmount leftovers,
// remove the directory. Every step is best effort.
func resetUserManagerState(ctx context.Context, uid int, runtimeDir string, logger *slog.Logger) {
_, _ = userManagerRunner(ctx, "systemctl", "stop", userManagerUnitName(uid))
_, _ = userManagerRunner(ctx, "loginctl", "terminate-user", strconv.Itoa(uid))
removeMountsUnder(ctx, runtimeDir, logger)
_, _ = userManagerRunner(ctx, "systemctl", "stop", userRuntimeDirUnitName(uid))
if _, err := os.Stat(runtimeDir); err == nil {
if out, err := userManagerRunner(ctx, "rm", "-rf", runtimeDir); err != nil && logger != nil {
logger.Warn("failed to remove stale runtime dir",
"dir", runtimeDir, "error", err, "output", strings.TrimSpace(string(out)))
}
}
}
// ensureUserRuntimeDir guarantees the runtime directory exists and is owned by
// uid BEFORE the user manager is started.
func ensureUserRuntimeDir(ctx context.Context, username string, uid int, stdRuntimeDir string, logger *slog.Logger) error {
if out, err := userManagerRunner(ctx, "systemctl", "start", userRuntimeDirUnitName(uid)); err != nil && logger != nil {
logger.Warn("systemd user-runtime-dir unit did not start; creating the runtime dir directly",
"unit", userRuntimeDirUnitName(uid), "dir", stdRuntimeDir,
"error", err, "output", strings.TrimSpace(string(out)))
}
if err := os.MkdirAll(stdRuntimeDir, 0o700); err != nil {
return fmt.Errorf("create runtime dir %s: %w", stdRuntimeDir, err)
}
// Non-recursive on purpose: a gvfsd-fuse mount under the dir denies even root.
if out, err := userManagerRunner(ctx, "chown", username+":", stdRuntimeDir); err != nil {
return fmt.Errorf("chown runtime dir %s: %w (output: %s)", stdRuntimeDir, err, string(out))
}
info, err := runtimeDirProbe(stdRuntimeDir)
if err != nil {
return fmt.Errorf("verify runtime dir %s after creation: %w", stdRuntimeDir, err)
}
if !info.exists || !info.isDir {
return fmt.Errorf("runtime dir %s is missing after creation", stdRuntimeDir)
}
if !info.ownerKnown {
return fmt.Errorf("runtime dir %s ownership cannot be verified", stdRuntimeDir)
}
if info.owner != uint32(uid) {
return fmt.Errorf("runtime dir %s is owned by uid %d, expected %d", stdRuntimeDir, info.owner, uid)
}
return nil
}
// bringUpUserManager brings the systemd user manager for uid up in a
// state-consistent order and waits for its bus socket.
func bringUpUserManager(ctx context.Context, username string, uid int, stdRuntimeDir string, logger *slog.Logger) error {
info, probeErr := runtimeDirProbe(stdRuntimeDir)
if stale, reason := classifyRuntimeDir(info, probeErr, uid); stale {
if logger != nil {
logger.Info("resetting stale user manager runtime",
"user", username, "uid", uid, "runtime_dir", stdRuntimeDir, "reason", reason)
}
resetUserManagerState(ctx, uid, stdRuntimeDir, logger)
}
if err := ensureUserRuntimeDir(ctx, username, uid, stdRuntimeDir, logger); err != nil {
return err
}
if out, err := userManagerRunner(ctx, "loginctl", "enable-linger", username); err != nil {
return fmt.Errorf("enable linger for %s: %w (output: %s)", username, err, string(out))
}
if err := ensureUserManagerRunning(ctx, uid, logger); err != nil {
return err
}
if err := waitForUserManager(ctx, stdRuntimeDir); err != nil {
return fmt.Errorf("user manager did not start for %s: %w", username, err)
}
return nil
}
const userManagerJournalQueryTimeout = 5 * time.Second
const userManagerJournalMaxLen = 400
// fetchUserManagerJournal returns a bounded, single-line excerpt of the unit's
// journal, detached from the caller's deadline.
func fetchUserManagerJournal(ctx context.Context, unit string, logger *slog.Logger) string {
queryCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), userManagerJournalQueryTimeout)
defer cancel()
out, err := userManagerRunner(queryCtx, "journalctl", "--no-pager", "-u", unit, "-n", "20")
if err != nil && strings.TrimSpace(string(out)) == "" {
if st, stErr := userManagerRunner(queryCtx, "systemctl", "status", "--no-pager", "-n", "20", unit); stErr == nil {
return condenseJournal(string(st))
}
if logger != nil {
logger.Debug("no journal available for user manager attribution", "unit", unit, "error", err)
}
}
return condenseJournal(string(out))
}
// condenseJournal folds a journal excerpt into ONE line and truncates at a rune
// boundary.
func condenseJournal(raw string) string {
var parts []string
for _, line := range strings.Split(raw, "\n") {
if fields := strings.Fields(line); len(fields) > 0 {
parts = append(parts, strings.Join(fields, " "))
}
}
joined := strings.Join(parts, " | ")
if len(joined) > userManagerJournalMaxLen {
cut := userManagerJournalMaxLen
for cut > 0 && !utf8.RuneStart(joined[cut]) {
cut--
}
joined = strings.TrimSpace(joined[:cut]) + "..."
}
return joined
}
And in ensureUserManagerRunning, change the terminal block to:
state := lastState
if state.active == "" {
state = fetchUserManagerState(ctx, unit)
}
lingerCount := countLingerEntries()
journal := fetchUserManagerJournal(ctx, unit, logger)
if logger != nil {
logger.Warn("systemd user manager did not start; host linger churn is a common cause",
"unit", unit,
"state", state.describe(unit),
"linger_entries", lingerCount,
"journal", journal,
)
}
// errors.New (not fmt.Errorf): the journal excerpt is arbitrary host output
// and must never be re-interpreted as a format string.
msg := fmt.Sprintf("%s failed: %s: linger_entries=%d",
userManagerStartStage, state.describe(unit), lingerCount)
if journal != "" {
msg += ": journal=" + journal
}
return errors.New(msg)
Note.
removeMountsUnder,ensureUserManagerRunning,fetchUserManagerState,startUserManagerUnit,countLingerEntries,waitForUserManager, and theuserManagerRunnerseam already exist and are reused unchanged.spawn_failure.gostage taxonomy is unchanged: the sub-stage name staysuser-manager-start.
go test ./internal/agent/ \
-run 'TestBringUpUserManager|TestEnsureUserRuntimeDir|TestClassifyRuntimeDir|TestCondenseJournal|TestEnsureUserManagerRunning' \
-count=1 -v
Verified result (Go 1.26.5, checkout of bb751f5):
--- PASS: TestBringUpUserManager_FreshPathIsNonDestructive
--- PASS: TestBringUpUserManager_StaleRuntimeDirTriggersReset
--- PASS: TestBringUpUserManager_FailedStartCarriesJournalReason
--- PASS: .../journalctl
--- PASS: .../systemctl_status_fallback
--- PASS: .../no_journal_is_not_a_new_failure
--- PASS: TestEnsureUserRuntimeDir_VerifiesOwnership
--- PASS: .../owned_by_uid
--- PASS: .../owner_not_applied
--- PASS: TestClassifyRuntimeDir_FailsSafe
--- PASS: .../fresh_path_missing
--- PASS: .../owned_by_uid
--- PASS: .../owned_by_other_uid
--- PASS: .../not_a_directory
--- PASS: .../owner_unknown
--- PASS: .../probe_failed
--- PASS: TestCondenseJournal
--- PASS: .../folds_lines
--- PASS: .../empty_input
--- PASS: .../truncates
--- PASS: .../truncates_on_rune_boundary
--- PASS: TestEnsureUserManagerRunning_Scenarios
--- PASS: .../already_active
--- PASS: .../reset_then_start
--- PASS: .../both_starts_fail
--- PASS: .../ctx_canceled
--- PASS: .../not_state_active
PASS
ok github.com/deployBunker/bunker/internal/agent 0.024s
| Acceptance point | How it is pinned |
|---|---|
| Fresh path issues no stop / terminate-user / rm | FreshPathIsNonDestructive asserts rm never ran and none of the four destructive argv appear |
| Runtime-dir bring-up + chown precede manager start | argv index of systemctl start user-runtime-dir@<uid> and chown <user>: <dir> < manager-start index |
| Directory exists ∧ isDir ∧ owner == uid at manager-start time | fake host probes the real on-disk dir inside its manager-start handler (dirExistedAtManagerStart) |
| Failed start carries the journal reason | journalctl and systemctl status arms contain pam_systemd/XDG_RUNTIME_DIR; single-line; % stays literal |
| Stale path resets, in order | stop user@ → stop user-runtime-dir@ → rm → start user-runtime-dir@ → start user@ strictly increasing |
| Ownership verification fails loudly | owner_not_applied error names observed and expected uid |
A pre-existing, unrelated failure exists in this environment: TestApplyUserSliceLimits_NotRoot_Coverage fails because /etc/systemd/system is read-only here (read-only file system), not because of this change.
# 1. Reproduce the blocking behavior: rm alone fails while a gvfsd-fuse mount exists.
rm -rf /run/user/1002 # -> Device or resource busy
# 2. Lazy unmount clears it.
mount | awk '/\/run\/user\/1002/ {print $3}' | while read -r m; do umount -l "$m"; done
rm -rf /run/user/1002
# 3. Pre-fix: the manager cannot come up without the directory.
systemctl start <email> # -> failed
journalctl -u <email> -n 20 --no-pager | grep -i 'runtime directory\|XDG_RUNTIME_DIR'
# 4. Fix: ensure the dir first, then start.
systemctl start <email> # best effort (RemainAfterExit no-op)
install -d -m 0700 -o 1002 -g 1002 /run/user/1002
systemctl reset-failed <email> || true
systemctl start <email>
systemctl is-active <email> # -> active
Reported live results: both remedies made the manager active — directory ensured beforehand → start rc 0 active; <email> explicitly stopped during the reset → start rc 0 active.
Four attempts to reach the racy state deterministically on a live host all ended with a healthy manager (fresh uid, ghost uid whose user was deleted while its manager lived on, hand-made directory, explicit unit stop), because the host recreates the directory whenever pam_systemd's ensure-step actually runs. The defect is a state race with an active/deactivating user-runtime-dir@ unit. The fix does not win the race — it removes the dependency on it.
user@.service on systemd 255 does not depend on user-runtime-dir@.service, so /run/user/<uid> must be created and verified before the manager start, the reset must stop systemd's own runtime-dir unit and only run when the directory is provably stale, and a failed start must surface the unit journal — because systemctl's Result does not name the cause.
# Evidence - Problem class: systemd-user-manager-cannot-start-without-runtime-dir - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-16T19:45:43.819Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "PROBLEM. `systemctl start user@<uid>.service` fails permanently (unit goes is-active=failed, Main process exited code=exited status=1) when the systemd user runtime directory /run/user/<uid> does not exist at start time. On systemd 255, user@.service carries NO Requires/Wants/After on user-runtime-dir@<uid>.service; that directory is created by logind's user-runtime-dir@<uid>.service (Type=oneshot, RemainAfterExit=yes, StopWhenUnneeded=yes), and pam_systemd only ensures it when its own ensure-step actually runs. If the directory has been deleted while the unit is still active or deactivating (RemainAfterExit holds a fixed on-disk claim), nothing recreates it, the PAM session for systemd-user cannot set XDG_RUNTIME_DIR, and /usr/lib/systemd/systemd --user exits 1. The failure is self-reinforcing: it is not fixed by reset-failed plus a retry. Verbatim journal: 'pam_systemd(systemd-user:session): Failed to stat() runtime directory '/run/user/1002': No such file or directory' -> 'pam_systemd(systemd-user:session): Not setting XDG_RUNTIME_DIR, as the directory is not in order.' -> '<email>: Main process exited, code=exited, status=1/FAILURE'. The systemd job Result text alone ('control process exited with error code') does NOT name this cause.\n\nWHY THE OBVIOUS FIX ORDER IS WRONG. The natural order (stop manager -> terminate-user -> rm -rf the dir -> enable-linger -> start manager) is precisely what creates the broken state, and it is destructive on the FRESH path too: a uid just allocated by useradd has no stale state of its own, while uid recycling means the previous owner's manager may still be live under that uid.\n\nFIX (Go, internal/agent/rootless.go, commit bb751f5). 1) Probe /run/user/<uid> with Lstat before touching anything and classify: absent = FRESH (take no destructive action at all: no systemctl stop, no loginctl terminate-user, no rm); exists but not a directory, owner unknown, owner != uid, or probe error = STALE (fail safe -> do the historical reset). 2) On the stale path, reset in a state-consistent order: systemctl stop user@<uid>.service, loginctl terminate-user <uid>, lazily umount anything under the directory (a gvfsd-fuse mount makes rm -rf fail with 'Device or resource busy'), systemctl stop user-runtime-dir@<uid>.service so its RemainAfterExit state matches the filesystem, then remove the directory. 3) BEFORE starting the manager, guarantee the directory: `systemctl start user-runtime-dir@<uid>.service` (best effort; a start of an already-active RemainAfterExit unit is a no-op, which is exactly why creating it by hand is required as the fallback), then MkdirAll(0700), non-recursive chown to the uid, then VERIFY exists + isDir + owner == uid and fail loudly otherwise. 4) Only then enable-linger, start the manager (with reset-failed + one retry), and wait for the bus. 5) Attribution: on terminal failure append a bounded single-line excerpt of `journalctl --no-pager -u <unit> -n 20` (fallback `systemctl status --no-pager -n 20 <unit>`) into the error and the WARN; query it on a context detached from the caller deadline (context.WithoutCancel + its own timeout) so attribution itself cannot turn into a failure.\n\nVERIFICATION. Unit tests drive the whole bring-up through an injected command-runner seam: fresh path emits no stop/terminate/rm; the runtime-dir bring-up and chown argv precede the manager start and the directory is verified present and uid-owned at manager-start time; a simulated failed start carries the journal reason in the returned error (journalctl and systemctl-status arms); the stale path still resets, in order. Live: on the host, `rm -rf /run/user/<uid>` alone fails with 'Device or resource busy' until mounts are lazily unmounted; and both remedies were shown to make the manager active under the reset scenario (directory ensured beforehand -> start rc 0 active; runtime-dir unit explicitly stopped during the reset -> start rc 0 active). NOTE ON REPRODUCIBILITY: four attempts to reach the racy state deterministically on a live host (fresh uid, ghost uid whose user was deleted while its manager lived on, hand-made directory, explicit unit stop) all ended with a healthy manager, because the host recreates the directory whenever pam_systemd's ensure-step actually runs; the defect is a state race with an active/deactivating user-runtime-dir@ unit, and the fix removes the dependency on that race rather than the race itself.", "environment": "Linux + systemd 255 (Ubuntu 24.04); a root daemon that provisions ephemeral per-agent Linux users and runs rootless Docker through each user's systemd user manager; uids are recycled after userdel and linger keeps old managers alive; the daemon itself stops the manager, terminate-users the uid and rm -rf /run/user/<uid> as a reset preamble before starting a new manager for the newly created user of that uid", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "systemd-user-manager-cannot-start-without-runtime-dir", "provider": "openrouter", "solved_at": "2026-09-16T19:45:43.820Z", "version": "bunker main bb751f5 (defect seen at 39f44aa)"}Diagnosis complete and verified. I cloned deployBunker/bunker, located the exact upstream fix commit bb751f5 (INT-CI-008), checked it out, and ran the regression suite. The full solution is written to ~/SOLUTION.md. Here it is:
user@<uid>.service cannot start when /run/user/<uid> is missingRepo: github.com/deployBunker/bunker
File: internal/agent/rootless.go
Defect at: 39f44aa (observed), fixed design bb751f5
Class: systemd-user-manager-cannot-start-without-runtime-dir
Environment: Linux + systemd 255 (Ubuntu 24.04); root daemon provisioning ephemeral per-agent users and rootless Docker; UIDs recycled after userdel; linger keeps old managers alive.
spawn <agent> fails at stage rootless-install in well under a second (0.6 s on the regression job) with:
user-manager-start failed: unit <email>: is-active=failed
(Result=Job for <email> failed because the control process exited
with error code. See "systemctl status <email>" ...):
linger_entries=135
The auto-generated-ID spawn 12 s later passes; only an explicit, previously used agent ID fails. systemctl job Result (control process exited) does not name the cause.
Verbatim journal lines that actually explain the failure:
pam_systemd(systemd-user:session): Failed to stat() runtime directory '/run/user/1002': No such file or directory
pam_systemd(systemd-user:session): Not setting XDG_RUNTIME_DIR, as the directory is not in order.
<email>: Main process exited, code=exited, status=1/FAILURE
On systemd 255, /usr/lib/systemd/system/user@.service carries no Requires=/Wants=/After= on user-runtime-dir@<uid>.service:
# user@.service (systemd 255)
[Unit]
Description=User Manager for UID %i
BindsTo=user-runtime-dir@%i.service # only on newer systemd
After=systemd-logind.service user-runtime-dir@%i.service # only on newer systemd
[Service]
User=%i
PAMName=systemd-user
ExecStart=/usr/lib/systemd/systemd --user
/run/user/<uid> is created by logind's own unit:
# user-runtime-dir@.service
[Service]
ExecStart=/usr/lib/systemd/systemd-user-runtime-dir start %i
ExecStop=/usr/lib/systemd/systemd-user-runtime-dir stop %i
Type=oneshot
RemainAfterExit=yes
StopWhenUnneeded=yes
Because it is RemainAfterExit=yes, once it has run it holds a fixed on-disk claim (active (exited)). Starting it again is a no-op even when the directory has been deleted out from under it. So nothing recreates the directory if it is removed while the unit is still active/deactivating.
user@<uid>.service starts with PAMName=systemd-user. pam_systemd's session step calls stat("/run/user/<uid>"); when it is absent it refuses to set XDG_RUNTIME_DIR. /usr/lib/systemd/systemd --user then has no runtime directory and exits 1. The unit lands in failed, and reset-failed + a retry never fixes it because the directory is still absent — the failure is self-reinforcing.
The old code in installRootlessDocker did, unconditionally, in this order:
systemctl stop user@<uid>.service
loginctl terminate-user <uid>
removeMountsUnder(/run/user/<uid>)
rm -rf /run/user/<uid>
enable-linger
ensureUserManagerRunning(...) <-- manager starts with dir MISSING
MkdirAll(/run/user/<uid>) <-- too late
chown <user>: /run/user/<uid>
This is precisely what creates the broken state:
/run/user/<uid> before the manager start, and systemd 255 will not recreate it.useradd may have no stale state of its own, while uid recycling means the previous owner's manager can still be live under that uid — so the unconditional stop/terminate-user/rm is destructive for no benefit.enable-linger is idempotent: if linger is already recorded, it takes no start action, so the unit stopped in step 1 stays down. The later passive wait then times out (or the start fails on the missing dir).rm -rf can also fail with Device or resource busy when the old manager left a gvfsd-fuse mount at /run/user/<uid>/gvfs.systemctl is-active / show -p Result only expose failed and exit-code. The line naming the cause lives in the unit journal (pam_systemd ... Failed to stat() runtime directory ...). Without it, operators see "control process exited" and cannot distinguish this failure from linger churn or a broken installer.
The bring-up becomes state-probing and directory-first:
/run/user/<uid> with Lstat; classify absent = FRESH → take no destructive action at all; anything else (not a dir, owner unknown, owner != uid, probe error) = STALE → fail-safe historical reset.loginctl terminate-user, lazily unmount leftovers, stop user-runtime-dir@<uid>.service so its RemainAfterExit state matches the filesystem, then remove the directory.systemctl start user-runtime-dir@<uid>.service (best effort; no-op when RemainAfterExit is already active — which is exactly why the explicit create is required as the fallback), then MkdirAll(0700), then a non-recursive chown <user>: <dir>, then verify exists ∧ isDir ∧ owner == uid; fail loudly otherwise.enable-linger, start the manager (with reset-failed + one retry), and wait for the bus socket.journalctl --no-pager -u <unit> -n 20 (fallback systemctl status --no-pager -n 20 <unit>) to the error and the WARN. The query runs on context.WithoutCancel(ctx) with its own timeout so attribution cannot fail with the caller's deadline.import (
"context"
"errors" // added
"fmt"
"log/slog"
"os"
"os/exec"
"os/user"
"path/filepath"
"strconv"
"strings"
"syscall" // added
"time"
"unicode/utf8" // added
)
installRootlessDockerDelete the block that starts at logger.Info("resetting user manager runtime", ...) and ends after the waitForUserManager call, and replace it with:
// Bring the systemd user manager up in a state-consistent order: reset
// stale state ONLY when there is real stale state, guarantee the runtime
// directory exists and is owned by the uid, then start the manager. The
// runtime directory MUST exist before user@<uid>.service starts, otherwise
// pam_systemd refuses XDG_RUNTIME_DIR and the manager can never go active
// (INT-CI-008).
if err := bringUpUserManager(ctx, username, uid, stdRuntimeDir, logger); err != nil {
return err
}
Keep the existing post-bring-up MkdirAll + non-recursive chown block as insurance for the installer step (it is no longer the only place the directory is created).
// userManagerUnitName is the per-uid systemd user manager unit.
func userManagerUnitName(uid int) string { return fmt.Sprintf("user@%d.service", uid) }
// userRuntimeDirUnitName is logind's runtime-directory unit for the same uid.
func userRuntimeDirUnitName(uid int) string {
return fmt.Sprintf("user-runtime-dir@%d.service", uid)
}
// runtimeDirInfo is the observable state of a systemd user runtime directory.
type runtimeDirInfo struct {
exists bool // false when the path is absent (the fresh-uid path)
isDir bool // false when the path exists but is not a directory
owner uint32 // uid owning the path
ownerKnown bool // false when the platform stat could not supply it
}
// runtimeDirProbe inspects a runtime directory. Package-level var (same seam
// style as userManagerRunner) because simulating a directory owned by a
// DIFFERENT uid needs a real chown to another user, which requires root.
var runtimeDirProbe = probeRuntimeDirOnDisk
// probeRuntimeDirOnDisk is the production probe. It uses Lstat so a symlink
// placed at the directory's path is NOT mistaken for a directory.
func probeRuntimeDirOnDisk(path string) (runtimeDirInfo, error) {
info, err := os.Lstat(path)
if err != nil {
if os.IsNotExist(err) {
return runtimeDirInfo{}, nil
}
return runtimeDirInfo{}, err
}
st := runtimeDirInfo{exists: true, isDir: info.IsDir()}
if sys, ok := info.Sys().(*syscall.Stat_t); ok {
st.owner = sys.Uid
st.ownerKnown = true
}
return st, nil
}
// classifyRuntimeDir decides whether the observed runtime directory carries
// state left behind by a PREVIOUS user of the same uid (which must be reset) or
// is the fresh path (which must not be touched). It fails SAFE.
func classifyRuntimeDir(info runtimeDirInfo, probeErr error, uid int) (bool, string) {
switch {
case probeErr != nil:
return true, "runtime dir probe failed: " + probeErr.Error()
case !info.exists:
return false, ""
case !info.isDir:
return true, "runtime path exists but is not a directory"
case !info.ownerKnown:
return true, "runtime dir ownership cannot be determined"
case info.owner != uint32(uid):
return true, fmt.Sprintf("runtime dir owned by uid %d, not %d", info.owner, uid)
}
return false, ""
}
// resetUserManagerState tears down stale user manager state for uid in a
// state-consistent order: stop the manager, stop logind's runtime-directory
// unit (so its RemainAfterExit state matches the filesystem), unmount leftovers,
// remove the directory. Every step is best effort.
func resetUserManagerState(ctx context.Context, uid int, runtimeDir string, logger *slog.Logger) {
_, _ = userManagerRunner(ctx, "systemctl", "stop", userManagerUnitName(uid))
_, _ = userManagerRunner(ctx, "loginctl", "terminate-user", strconv.Itoa(uid))
removeMountsUnder(ctx, runtimeDir, logger)
_, _ = userManagerRunner(ctx, "systemctl", "stop", userRuntimeDirUnitName(uid))
if _, err := os.Stat(runtimeDir); err == nil {
if out, err := userManagerRunner(ctx, "rm", "-rf", runtimeDir); err != nil && logger != nil {
logger.Warn("failed to remove stale runtime dir",
"dir", runtimeDir, "error", err, "output", strings.TrimSpace(string(out)))
}
}
}
// ensureUserRuntimeDir guarantees the runtime directory exists and is owned by
// uid BEFORE the user manager is started.
func ensureUserRuntimeDir(ctx context.Context, username string, uid int, stdRuntimeDir string, logger *slog.Logger) error {
if out, err := userManagerRunner(ctx, "systemctl", "start", userRuntimeDirUnitName(uid)); err != nil && logger != nil {
logger.Warn("systemd user-runtime-dir unit did not start; creating the runtime dir directly",
"unit", userRuntimeDirUnitName(uid), "dir", stdRuntimeDir,
"error", err, "output", strings.TrimSpace(string(out)))
}
if err := os.MkdirAll(stdRuntimeDir, 0o700); err != nil {
return fmt.Errorf("create runtime dir %s: %w", stdRuntimeDir, err)
}
// Non-recursive on purpose: a gvfsd-fuse mount under the dir denies even root.
if out, err := userManagerRunner(ctx, "chown", username+":", stdRuntimeDir); err != nil {
return fmt.Errorf("chown runtime dir %s: %w (output: %s)", stdRuntimeDir, err, string(out))
}
info, err := runtimeDirProbe(stdRuntimeDir)
if err != nil {
return fmt.Errorf("verify runtime dir %s after creation: %w", stdRuntimeDir, err)
}
if !info.exists || !info.isDir {
return fmt.Errorf("runtime dir %s is missing after creation", stdRuntimeDir)
}
if !info.ownerKnown {
return fmt.Errorf("runtime dir %s ownership cannot be verified", stdRuntimeDir)
}
if info.owner != uint32(uid) {
return fmt.Errorf("runtime dir %s is owned by uid %d, expected %d", stdRuntimeDir, info.owner, uid)
}
return nil
}
// bringUpUserManager brings the systemd user manager for uid up in a
// state-consistent order and waits for its bus socket.
func bringUpUserManager(ctx context.Context, username string, uid int, stdRuntimeDir string, logger *slog.Logger) error {
info, probeErr := runtimeDirProbe(stdRuntimeDir)
if stale, reason := classifyRuntimeDir(info, probeErr, uid); stale {
if logger != nil {
logger.Info("resetting stale user manager runtime",
"user", username, "uid", uid, "runtime_dir", stdRuntimeDir, "reason", reason)
}
resetUserManagerState(ctx, uid, stdRuntimeDir, logger)
}
if err := ensureUserRuntimeDir(ctx, username, uid, stdRuntimeDir, logger); err != nil {
return err
}
if out, err := userManagerRunner(ctx, "loginctl", "enable-linger", username); err != nil {
return fmt.Errorf("enable linger for %s: %w (output: %s)", username, err, string(out))
}
if err := ensureUserManagerRunning(ctx, uid, logger); err != nil {
return err
}
if err := waitForUserManager(ctx, stdRuntimeDir); err != nil {
return fmt.Errorf("user manager did not start for %s: %w", username, err)
}
return nil
}
const userManagerJournalQueryTimeout = 5 * time.Second
const userManagerJournalMaxLen = 400
// fetchUserManagerJournal returns a bounded, single-line excerpt of the unit's
// journal, detached from the caller's deadline.
func fetchUserManagerJournal(ctx context.Context, unit string, logger *slog.Logger) string {
queryCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), userManagerJournalQueryTimeout)
defer cancel()
out, err := userManagerRunner(queryCtx, "journalctl", "--no-pager", "-u", unit, "-n", "20")
if err != nil && strings.TrimSpace(string(out)) == "" {
if st, stErr := userManagerRunner(queryCtx, "systemctl", "status", "--no-pager", "-n", "20", unit); stErr == nil {
return condenseJournal(string(st))
}
if logger != nil {
logger.Debug("no journal available for user manager attribution", "unit", unit, "error", err)
}
}
return condenseJournal(string(out))
}
// condenseJournal folds a journal excerpt into ONE line and truncates at a rune
// boundary.
func condenseJournal(raw string) string {
var parts []string
for _, line := range strings.Split(raw, "\n") {
if fields := strings.Fields(line); len(fields) > 0 {
parts = append(parts, strings.Join(fields, " "))
}
}
joined := strings.Join(parts, " | ")
if len(joined) > userManagerJournalMaxLen {
cut := userManagerJournalMaxLen
for cut > 0 && !utf8.RuneStart(joined[cut]) {
cut--
}
joined = strings.TrimSpace(joined[:cut]) + "..."
}
return joined
}
And in ensureUserManagerRunning, change the terminal block to:
state := lastState
if state.active == "" {
state = fetchUserManagerState(ctx, unit)
}
lingerCount := countLingerEntries()
journal := fetchUserManagerJournal(ctx, unit, logger)
if logger != nil {
logger.Warn("systemd user manager did not start; host linger churn is a common cause",
"unit", unit,
"state", state.describe(unit),
"linger_entries", lingerCount,
"journal", journal,
)
}
// errors.New (not fmt.Errorf): the journal excerpt is arbitrary host output
// and must never be re-interpreted as a format string.
msg := fmt.Sprintf("%s failed: %s: linger_entries=%d",
userManagerStartStage, state.describe(unit), lingerCount)
if journal != "" {
msg += ": journal=" + journal
}
return errors.New(msg)
Note.
removeMountsUnder,ensureUserManagerRunning,fetchUserManagerState,startUserManagerUnit,countLingerEntries,waitForUserManager, and theuserManagerRunnerseam already exist and are reused unchanged.spawn_failure.gostage taxonomy is unchanged: the sub-stage name staysuser-manager-start.
go test ./internal/agent/ \
-run 'TestBringUpUserManager|TestEnsureUserRuntimeDir|TestClassifyRuntimeDir|TestCondenseJournal|TestEnsureUserManagerRunning' \
-count=1 -v
Verified result (Go 1.26.5, checkout of bb751f5):
--- PASS: TestBringUpUserManager_FreshPathIsNonDestructive
--- PASS: TestBringUpUserManager_StaleRuntimeDirTriggersReset
--- PASS: TestBringUpUserManager_FailedStartCarriesJournalReason
--- PASS: .../journalctl
--- PASS: .../systemctl_status_fallback
--- PASS: .../no_journal_is_not_a_new_failure
--- PASS: TestEnsureUserRuntimeDir_VerifiesOwnership
--- PASS: .../owned_by_uid
--- PASS: .../owner_not_applied
--- PASS: TestClassifyRuntimeDir_FailsSafe
--- PASS: .../fresh_path_missing
--- PASS: .../owned_by_uid
--- PASS: .../owned_by_other_uid
--- PASS: .../not_a_directory
--- PASS: .../owner_unknown
--- PASS: .../probe_failed
--- PASS: TestCondenseJournal
--- PASS: .../folds_lines
--- PASS: .../empty_input
--- PASS: .../truncates
--- PASS: .../truncates_on_rune_boundary
--- PASS: TestEnsureUserManagerRunning_Scenarios
--- PASS: .../already_active
--- PASS: .../reset_then_start
--- PASS: .../both_starts_fail
--- PASS: .../ctx_canceled
--- PASS: .../not_state_active
PASS
ok github.com/deployBunker/bunker/internal/agent 0.024s
| Acceptance point | How it is pinned |
|---|---|
| Fresh path issues no stop / terminate-user / rm | FreshPathIsNonDestructive asserts rm never ran and none of the four destructive argv appear |
| Runtime-dir bring-up + chown precede manager start | argv index of systemctl start user-runtime-dir@<uid> and chown <user>: <dir> < manager-start index |
| Directory exists ∧ isDir ∧ owner == uid at manager-start time | fake host probes the real on-disk dir inside its manager-start handler (dirExistedAtManagerStart) |
| Failed start carries the journal reason | journalctl and systemctl status arms contain pam_systemd/XDG_RUNTIME_DIR; single-line; % stays literal |
| Stale path resets, in order | stop user@ → stop user-runtime-dir@ → rm → start user-runtime-dir@ → start user@ strictly increasing |
| Ownership verification fails loudly | owner_not_applied error names observed and expected uid |
A pre-existing, unrelated failure exists in this environment: TestApplyUserSliceLimits_NotRoot_Coverage fails because /etc/systemd/system is read-only here (read-only file system), not because of this change.
# 1. Reproduce the blocking behavior: rm alone fails while a gvfsd-fuse mount exists.
rm -rf /run/user/1002 # -> Device or resource busy
# 2. Lazy unmount clears it.
mount | awk '/\/run\/user\/1002/ {print $3}' | while read -r m; do umount -l "$m"; done
rm -rf /run/user/1002
# 3. Pre-fix: the manager cannot come up without the directory.
systemctl start <email> # -> failed
journalctl -u <email> -n 20 --no-pager | grep -i 'runtime directory\|XDG_RUNTIME_DIR'
# 4. Fix: ensure the dir first, then start.
systemctl start <email> # best effort (RemainAfterExit no-op)
install -d -m 0700 -o 1002 -g 1002 /run/user/1002
systemctl reset-failed <email> || true
systemctl start <email>
systemctl is-active <email> # -> active
Reported live results: both remedies made the manager active — directory ensured beforehand → start rc 0 active; <email> explicitly stopped during the reset → start rc 0 active.
Four attempts to reach the racy state deterministically on a live host all ended with a healthy manager (fresh uid, ghost uid whose user was deleted while its manager lived on, hand-made directory, explicit unit stop), because the host recreates the directory whenever pam_systemd's ensure-step actually runs. The defect is a state race with an active/deactivating user-runtime-dir@ unit. The fix does not win the race — it removes the dependency on it.
user@.service on systemd 255 does not depend on user-runtime-dir@.service, so /run/user/<uid> must be created and verified before the manager start, the reset must stop systemd's own runtime-dir unit and only run when the directory is provably stale, and a failed start must surface the unit journal — because systemctl's Result does not name the cause.
# Evidence - Problem class: systemd-user-manager-cannot-start-without-runtime-dir - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-16T19:45:43.819Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "PROBLEM. `systemctl start user@<uid>.service` fails permanently (unit goes is-active=failed, Main process exited code=exited status=1) when the systemd user runtime directory /run/user/<uid> does not exist at start time. On systemd 255, user@.service carries NO Requires/Wants/After on user-runtime-dir@<uid>.service; that directory is created by logind's user-runtime-dir@<uid>.service (Type=oneshot, RemainAfterExit=yes, StopWhenUnneeded=yes), and pam_systemd only ensures it when its own ensure-step actually runs. If the directory has been deleted while the unit is still active or deactivating (RemainAfterExit holds a fixed on-disk claim), nothing recreates it, the PAM session for systemd-user cannot set XDG_RUNTIME_DIR, and /usr/lib/systemd/systemd --user exits 1. The failure is self-reinforcing: it is not fixed by reset-failed plus a retry. Verbatim journal: 'pam_systemd(systemd-user:session): Failed to stat() runtime directory '/run/user/1002': No such file or directory' -> 'pam_systemd(systemd-user:session): Not setting XDG_RUNTIME_DIR, as the directory is not in order.' -> '<email>: Main process exited, code=exited, status=1/FAILURE'. The systemd job Result text alone ('control process exited with error code') does NOT name this cause.\n\nWHY THE OBVIOUS FIX ORDER IS WRONG. The natural order (stop manager -> terminate-user -> rm -rf the dir -> enable-linger -> start manager) is precisely what creates the broken state, and it is destructive on the FRESH path too: a uid just allocated by useradd has no stale state of its own, while uid recycling means the previous owner's manager may still be live under that uid.\n\nFIX (Go, internal/agent/rootless.go, commit bb751f5). 1) Probe /run/user/<uid> with Lstat before touching anything and classify: absent = FRESH (take no destructive action at all: no systemctl stop, no loginctl terminate-user, no rm); exists but not a directory, owner unknown, owner != uid, or probe error = STALE (fail safe -> do the historical reset). 2) On the stale path, reset in a state-consistent order: systemctl stop user@<uid>.service, loginctl terminate-user <uid>, lazily umount anything under the directory (a gvfsd-fuse mount makes rm -rf fail with 'Device or resource busy'), systemctl stop user-runtime-dir@<uid>.service so its RemainAfterExit state matches the filesystem, then remove the directory. 3) BEFORE starting the manager, guarantee the directory: `systemctl start user-runtime-dir@<uid>.service` (best effort; a start of an already-active RemainAfterExit unit is a no-op, which is exactly why creating it by hand is required as the fallback), then MkdirAll(0700), non-recursive chown to the uid, then VERIFY exists + isDir + owner == uid and fail loudly otherwise. 4) Only then enable-linger, start the manager (with reset-failed + one retry), and wait for the bus. 5) Attribution: on terminal failure append a bounded single-line excerpt of `journalctl --no-pager -u <unit> -n 20` (fallback `systemctl status --no-pager -n 20 <unit>`) into the error and the WARN; query it on a context detached from the caller deadline (context.WithoutCancel + its own timeout) so attribution itself cannot turn into a failure.\n\nVERIFICATION. Unit tests drive the whole bring-up through an injected command-runner seam: fresh path emits no stop/terminate/rm; the runtime-dir bring-up and chown argv precede the manager start and the directory is verified present and uid-owned at manager-start time; a simulated failed start carries the journal reason in the returned error (journalctl and systemctl-status arms); the stale path still resets, in order. Live: on the host, `rm -rf /run/user/<uid>` alone fails with 'Device or resource busy' until mounts are lazily unmounted; and both remedies were shown to make the manager active under the reset scenario (directory ensured beforehand -> start rc 0 active; runtime-dir unit explicitly stopped during the reset -> start rc 0 active). NOTE ON REPRODUCIBILITY: four attempts to reach the racy state deterministically on a live host (fresh uid, ghost uid whose user was deleted while its manager lived on, hand-made directory, explicit unit stop) all ended with a healthy manager, because the host recreates the directory whenever pam_systemd's ensure-step actually runs; the defect is a state race with an active/deactivating user-runtime-dir@ unit, and the fix removes the dependency on that race rather than the race itself.", "environment": "Linux + systemd 255 (Ubuntu 24.04); a root daemon that provisions ephemeral per-agent Linux users and runs rootless Docker through each user's systemd user manager; uids are recycled after userdel and linger keeps old managers alive; the daemon itself stops the manager, terminate-users the uid and rm -rf /run/user/<uid> as a reset preamble before starting a new manager for the newly created user of that uid", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "systemd-user-manager-cannot-start-without-runtime-dir", "provider": "openrouter", "solved_at": "2026-09-16T19:45:43.820Z", "version": "bunker main bb751f5 (defect seen at 39f44aa)"}