Production

A production keelwave deployment is two moving parts: the keelwave container (API + embedded dashboard, one process, one port) and a TimescaleDB database. Everything below assumes that shape.

Baseline settings to change from the defaults:

ENV=production
DB_ADDR=postgres://keelwave:<password>@db:5432/keelwave?sslmode=require
PUBLIC_URL=https://keelwave.example.com
DASHBOARD_URL=https://keelwave.example.com
CORS_ALLOWED_ORIGINS=https://keelwave.example.com

ENV=production is what sets the Secure flag on session cookies, so it is required once you are behind TLS. See Configuration for the full variable list.

Upgrades

The image never applies migrations on start. Apply them yourself, then roll the new image.

# 1. apply migrations against the running database
docker compose --profile app run --rm migrate

# or, from a checkout of core/ with the golang-migrate CLI:
make migrate-up DB_ADDR="postgres://keelwave:<password>@host:5432/keelwave?sslmode=require"

# 2. roll the app
docker compose --profile app up -d app

Migration state lives in the database and can be inspected or repaired with the same Makefile targets:

make migrate-version            # current schema version
make migrate-down               # roll back one migration
make migrate-force version=<n>  # clear a dirty state after a failed migration

Recommended order for a release:

1. Take a backup (below).
2. Apply migrations. They are additive in the common case, but read the diff for the release before assuming so.
3. Deploy the new image.

Rolling back an application version does not roll back the schema. If you must revert, apply the matching down migrations explicitly with make migrate-down, one at a time.

Shutdown is graceful on SIGINT/SIGTERM: in-flight requests drain, batch buffers flush to the database, the alerting scheduler and worker stop, and the pool closes — all within SHUTDOWN_TIMEOUT (default 10s). Give your orchestrator a termination grace period at least as long, or buffered ingest rows that have not been flushed will be lost.

Backups

Everything is in Postgres — the keelwave container itself is stateless, so there is nothing to back up on the app side.

TimescaleDB is a Postgres extension, so the standard tools apply, with one caveat: pg_dump needs the extension present on the restore target, and hypertable chunks and continuous aggregates need the TimescaleDB-documented dump/restore procedure rather than a naive dump. Follow the TimescaleDB backup and restore documentation for the version you run — the schema here uses hypertables (ai_traces, api_events, infra_metrics, agent_runs, agent_steps) and a continuous aggregate (agent_runs_5m) with a refresh policy, all of which the procedure covers.

A logical dump of the whole database from the Compose stack looks like:

docker compose exec -T db \
  pg_dump -U keelwave -d keelwave -Fc > keelwave-$(date +%F).dump

Verify a restore into a scratch database before you rely on it. For anything where losing hours of data matters, prefer continuous physical backups (pg_basebackup plus WAL archiving, or a managed Postgres provider's point-in-time recovery) over periodic logical dumps.

The Compose db service stores its data in the keelwave_db named volume. Deleting that volume deletes the database — docker compose down -v is destructive.

Reverse proxy and TLS

The keelwave binary speaks plain HTTP on ADDR (default :8080) and does not terminate TLS. Put a reverse proxy (Caddy, nginx, Traefik, a cloud load balancer) in front of it and terminate TLS there.

Because the dashboard is embedded in the same binary and served same-origin with the API, proxy the whole origin to one upstream — do not try to split /v1/* from / across different backends.

keelwave.example.com {
    reverse_proxy keelwave:8080
}

nginx equivalent:

server {
    listen 443 ssl;
    server_name keelwave.example.com;

    # ssl_certificate / ssl_certificate_key ...

    location / {
        proxy_pass http://keelwave:8080;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Things to get right at the proxy:

• Forwarded client IP. The per-IP ingest rate limiter reads the client address through chi's RealIP middleware, which trusts X-Forwarded-For / X-Real-IP. Your proxy must set those headers, and must overwrite rather than append attacker-supplied values — otherwise the per-IP limit is trivially bypassed. Only expose the app port through the proxy, never directly.
• Public URLs. Set PUBLIC_URL and DASHBOARD_URL to the external https:// origin. OAuth callbacks and email verification links are built from them, so leaving the localhost defaults produces links that do not work.
• CORS. CORS_ALLOWED_ORIGINS must list the external origin. * is rejected at startup because responses carry credentials.
• Timeouts. The server uses 30s read and write timeouts, a 60s per-request handler timeout, and a 1-minute idle timeout. Keep the proxy's timeouts in the same range so it does not cut requests short of the handler timeout.

The runtime image is gcr.io/distroless/static:nonroot — no shell, no package manager, runs as a non-root user. Keep it that way; there is nothing to exec into for debugging, so rely on logs and /v1/health.

Health checks and observability

GET /v1/health returns status, version, and environment:

curl -fsS https://keelwave.example.com/v1/health

It is unauthenticated and cheap — suitable as a liveness and readiness probe. Note that it does not probe the database; the process exits at startup if the initial database ping fails, so a running process implies the pool was established at boot.

Logs are structured JSON on stdout (zap production encoder). Collect them with whatever you already run.

Resource guidance

The stack is a single static Go binary plus Postgres. Practically all of the sizing work is the database.

Application container. One process, no local state, no CGO. Memory is dominated by the in-memory ingest buffers: four queues (ai_traces, api_events, infra_metrics, agent_steps) of BATCH_QUEUE_DEPTH rows each, 10,000 by default. A small container is enough to start; scale up only after you raise the queue depths or see sustained 503 responses. Because the queues are in-process memory, scaling horizontally is safe for throughput but means each replica holds its own unflushed rows.

Database. TimescaleDB does the real work. Size it against ingest volume: agent_steps and ai_traces grow fastest, one row per agent step and one per LLM call. Chunk sizing works best when the most recent chunk fits comfortably in RAM, so give the database enough memory to hold the active chunks plus your query working set, and use fast local disk. DB_MAX_CONNS (default 30) caps connections per keelwave replica — keep the total across replicas below the database's max_connections.

Retention. The migrations create the hypertables (declared with tsdb.segmentby for columnstore efficiency) and one continuous aggregate, but no add_retention_policy call is part of the schema. Long-running deployments should add a TimescaleDB retention policy sized to how far back you actually query, otherwise the raw hypertables grow without bound.

Rate limits. Defaults are 100 ingest requests per minute per IP and 1000 per minute per API key. Agents behind a shared NAT or a single Kubernetes egress IP hit the per-IP limit first; raise RATE_LIMIT_INGEST_IP_PER_MINUTE accordingly.