docker-compose-fresh-clone-alembic-localhost-crash-loop

Answer

I reconstructed the project shape, reproduced the failure mode, verified the fix with docker compose config and a startup-path simulation, and wrote the full solution to /workspace/SOLUTION.md.

Verification evidence (from this workspace)

Before fix — buggy compose, no .env: app service renders no DATABASE_URL*:

app:
  environment:
    REDIS_URL: redis://redis:6379/0
    TEST_DATABASE_URL: postgresql+asyncpg://app:app@postgres:5432/app_test

→ code falls back to localhost:5435 → Connection refused crash loop.

After fix:

DATABASE_URL:       postgresql+asyncpg://app:app@postgres:5432/app
DATABASE_URL_SYNC:  postgresql+psycopg://app:app@postgres:5432/app
OPENSEARCH_URL:     http://opensearch:9200
SECRET_KEY:         dev-insecure-change-me

Entrypoint fallback simulation:

# no DATABASE_URL_SYNC (fresh clone, buggy):  -> postgresql://...@localhost:5435/app
# DATABASE_URL_SYNC injected (fixed):         -> postgresql+psycopg://...@postgres:5432/app

Solution (full document)

# Fresh-clone `docker compose up` crash-loop: Alembic/bootstrap fall back to `localhost:5435`

**Problem class:** `docker-compose-fresh-clone-alembic-localhost-crash-loop`

## Symptom

On a clean clone with no `.env`:

```
docker compose up -d --build
docker compose ps        # app never becomes healthy, restarts forever
docker compose logs app
# sqlalchemy.exc.OperationalError: connection to server at "localhost",
# port 5435 failed: Connection refused
```

The API/migration container enters a crash loop. On hosts where a `.env` happens to exist (the everyday dev box), the same code works, so the bug is invisible until CI/another machine clones fresh.

## Root cause

The container's configuration is supplied through two independent channels, and only one of them has a safe fallback:

```
compose service
├── env_file: [{path: .env, required: false}]   # loads the gitignored .env
└── environment: {...}                          # service-level vars
```

The `environment:` block set *other* things (redis, a `TEST_DATABASE_URL`, legacy `OPENSEARCH_HOST`/`OPENSEARCH_PORT`) but **not the migration URL**. On a fresh clone `.env` is absent (`required: false` means compose silently continues), so nothing overrides the defaults baked into the code:

1. **Alembic** reads `alembic.ini`:
   ```ini
   sqlalchemy.url = postgresql://app:app@localhost:5435/app
   ```
   `localhost:5435` is the **host-mapped** port (`ports: "5435:5432"`). Inside the `app` container `localhost` is the container itself, so `alembic upgrade head` fails with `Connection refused`. The entrypoint uses `set -e`, the container exits, Docker restarts it → crash loop.

2. **Settings** (pydantic `BaseSettings`) has localhost defaults for the same reasons:
   ```python
   DATABASE_URL: str = "postgresql+asyncpg://app:app@localhost:5435/app"
   DATABASE_URL_SYNC: str = "postgresql+psycopg://app:app@localhost:5435/app"
   OPENSEARCH_URL: str = "http://localhost:9205"
   SECRET_KEY: str          # required, no default
   ```
   Even after migrations are fixed, the worker dies with a `secret_key` `ValidationError` because `SECRET_KEY` was never provided by compose.

3. **Stale legacy vars:** compose set `OPENSEARCH_HOST` / `OPENSEARCH_PORT`, which the app no longer reads. The field the app *actually* reads, `OPENSEARCH_URL`, still defaulted to `localhost:9205`.

The dev box masks all three because a `.env` (from `.env.example`, gitignored) is present and supplies working values.

## The fix

`env_file: required:false` gives no fallback. **Every variable the entrypoint or `Settings` reads must also appear in the compose `environment:` block with a `${VAR:-default}` interpolation default that points at the in-network service hostname** (`postgres`, `redis`, `opensearch`) — never `localhost`.

### 1. `docker-compose.yml` (the actual fix)

```yaml
services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: ${POSTGRES_USER:-app}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-app}
      POSTGRES_DB: ${POSTGRES_DB:-app}
    ports:
      - "5435:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-app}"]
      interval: 5s
      timeout: 5s
      retries: 10

  redis:
    image: redis:7

  opensearch:
    image: opensearchproject/opensearch:2
    environment:
      discovery.type: single-node

  app:
    build: .
    env_file:
      - path: .env
        required: false
    environment:
      # --- in-network URLs, with interpolation defaults so a fresh clone
      #     (no .env) never falls back to the code's localhost defaults ---
      DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER:-app}:${POSTGRES_PASSWORD:-app}@postgres:5432/${POSTGRES_DB:-app}
      DATABASE_URL_SYNC: postgresql+psycopg://${POSTGRES_USER:-app}:${POSTGRES_PASSWORD:-app}@postgres:5432/${POSTGRES_DB:-app}
      OPENSEARCH_URL: ${OPENSEARCH_URL:-http://opensearch:9200}
      REDIS_URL: redis://redis:6379/0
      # Was legacy/unused: OPENSEARCH_HOST / OPENSEARCH_PORT (removed)
      # SECRET_KEY has no code default and must be supplied somewhere:
      SECRET_KEY: ${SECRET_KEY:-dev-insecure-change-me}
    depends_on:
      postgres:
        condition: service_healthy
```

Notes:

- Compose interpolation (`${...}`) resolves from the shell and the **project-level `.env`**, not from the service `env_file`. A stray `DATABASE_URL=...localhost...` in the project `.env` cannot leak in, because the URL is *constructed* from `POSTGRES_*` rather than interpolated from `DATABASE_URL`.
- Compose `environment:` takes precedence over `env_file:`, so these values always win inside the container.
- The `SECRET_KEY` default is for dev only — a real deployment must set it (or keep it required and provide it via the project `.env`/secret store).

### 2. Harden the startup path so a missing URL fails loudly

Do not let `alembic.ini` carry a real host-mapped URL. Remove it and read the env var in `alembic/env.py`:

`alembic.ini`
```ini
[alembic]
script_location = alembic
# sqlalchemy.url intentionally omitted — comes from DATABASE_URL_SYNC
```

`alembic/env.py` (top of file)
```python
import os
from sqlalchemy import engine_from_config, pool

config = context.config  # (existing import/context setup)

db_url = os.environ.get("DATABASE_URL_SYNC")
if not db_url:
    raise RuntimeError(
        "DATABASE_URL_SYNC is not set. Provide it (compose sets it, or export it "
        "for host runs). Refusing to fall back to a localhost default."
    )
config.set_main_option("sqlalchemy.url", db_url)
```

Same pattern for any bootstrap script: read `DATABASE_URL_SYNC` and raise if absent instead of using a `DEFAULT_DATABASE_URL = "...localhost..."`.

### 3. Document the required vars

`.env.example`
```dotenv
POSTGRES_USER=app
POSTGRES_PASSWORD=app
POSTGRES_DB=app
SECRET_KEY=change-me
# DATABASE_URL / DATABASE_URL_SYNC / OPENSEARCH_URL are derived by compose and
# normally do NOT need to be set. Override only if you run services outside compose.
```

### 4. Audit for every remaining localhost default

Run this against the app and startup scripts; each hit is a potential fresh-clone crash:

```bash
# App defaults / Settings
grep -rnE 'localhost|127\.0\.0\.1' app/ --include='*.py' | grep -iE 'url|host|dsn|port'

# Entrypoints, scripts, alembic
grep -rnE 'localhost|127\.0\.0\.1|DEFAULT_.*URL' \
  entrypoint.sh scripts/ alembic/ alembic.ini docker/ 2>/dev/null
```

Fix each one by adding a matching `environment:` entry with an in-network default. In this project the findings were:

| Variable | Code default (bug) | Compose fix |
|---|---|---|
| `DATABASE_URL` | `postgresql+asyncpg://...@localhost:5435/app` | `...@postgres:5432/...` |
| `DATABASE_URL_SYNC` | `postgresql+psycopg://...@localhost:5435/app` | `...@postgres:5432/...` |
| `SECRET_KEY` | *(required, no default)* → ValidationError | `${SECRET_KEY:-dev-insecure-change-me}` |
| `OPENSEARCH_URL` | `http://localhost:9205` | `http://opensearch:9200` |
| `REDIS_URL` | `redis://localhost:6379/0` | `redis://redis:6379/0` |

## Verification

### Client-side (no daemon needed) — confirms interpolation is correct

```bash
cd <repo>
mv .env .env.bak 2>/dev/null || true      # simulate fresh clone

docker compose config | grep -E 'DATABASE_URL|OPENSEARCH_URL|SECRET_KEY'
```

Expected (verified output from the reproduction):

```
DATABASE_URL: postgresql+asyncpg://app:app@postgres:5432/app
DATABASE_URL_SYNC: postgresql+psycopg://app:app@postgres:5432/app
OPENSEARCH_URL: http://opensearch:9200
SECRET_KEY: dev-insecure-change-me
```

The buggy compose renders **no** `DATABASE_URL*` at all (so the code's `localhost:5435` default wins). Confirmed side by side with the reproduction in this workspace.

### Runtime — the end-to-end acceptance check

```bash
mv .env .env.bak 2>/dev/null || true
docker compose down -v
docker compose up -d --build

# 1. Migrations hit the service, not the host port
docker compose logs app | grep -E 'localhost:5435|OperationalError'   # expect NO output

# 2. Schema is current
docker compose exec app alembic current        # expect: <rev> (head)

# 3. Worker/web starts instead of secret_key ValidationError
docker compose logs app | grep -iE 'ValidationError|secret_key'  # expect NO output
docker compose logs app | grep -iE 'startup complete|worker.*ready|listening'

# 4. Stack is healthy
docker compose ps                              # all services "running"/"healthy"

mv .env.bak .env 2>/dev/null || true           # restore
```

If you don't have a docker daemon, the interpolation and fallback logic can be reproduced without it:

```bash
# 1. Compose interpolation (works offline, client-side)
docker compose config | grep DATABASE_URL

# 2. Entrypoint fallback (stub alembic reads DATABASE_URL_SYNC, else alembic.ini)
DATABASE_URL_SYNC='postgresql+psycopg://app:app@postgres:5432/app' ./entrypoint.sh
#   -> "alembic would connect to: postgresql+psycopg://app:app@postgres:5432/app"
unset DATABASE_URL_SYNC; ./entrypoint.sh
#   -> "alembic would connect to: postgresql://...@localhost:5435/app"  (the bug)
```

## One-line rule

> When a compose service uses `env_file: {required: false}`, every env var the entrypoint or `Settings` reads must *also* be present in the `environment:` block with a `${VAR:-default}` that points at the in-network service hostname. `localhost` inside a container is itself — never the Postgres/Redis/OpenSearch service.

Reproduction artifacts (buggy vs. fixed compose, config.py, alembic.ini, entrypoint.sh) are under /workspace/repro/.