Operational Runbooks¶

Step-by-step procedures for the single-VM Docker Compose beta deployment. Each runbook assumes you are SSH'd into the host as the deploy user with the repo checked out at /opt/redhound and a filled-in .env (from .env.prod.template).

The production stack is launched with:

make prod-up    # docker compose -f docker-compose.yml -f docker-compose.prod.yml ...

Deploy a new release¶

Releases are SHA-pinned. CI publishes images to ghcr.io/redhound-labs/* on every merge to dev/main.

Pick the target image tag (the long sha-<...> tag from the CI run).
Update IMAGE_TAG in /opt/redhound/.env.
From your laptop:
```
make deploy DEPLOY_HOST=deploy@<host-ip>
```
This SSHes in, git pulls, pulls the new images, recreates containers, and runs alembic upgrade head.

To deploy manually on the host:

cd /opt/redhound
git pull --ff-only
make prod-up                       # pulls + recreates
docker compose -f docker-compose.yml -f docker-compose.prod.yml exec -T app \
  alembic upgrade head

Smoke-test after every deploy (see checklist at the bottom).

First admin (one-time, launch day)¶

Public signup is closed (REDHOUND_AUTH_SIGNUP_OPEN=false) and /api/v1/auth/invite is superuser-gated — so the very first admin has to be created out-of-band on the host. Beta testers are then added via the invite flow (needs working email) or by re-running this command per tester:

docker compose -f docker-compose.yml -f docker-compose.prod.yml exec -T app \
  python scripts/create_user.py --email you@example.com \
  --password '<strong-password>' --superuser

Rollback¶

Rollback is a tag swap — no image rebuild needed because every release is already published and SHA-tagged.

Find the previous good SHA (CI history or docker compose images).
Set IMAGE_TAG=<previous-sha> in .env.
Re-run the deploy:
```
make prod-up
```
If the bad release ran a migration, downgrade it. Check current revision first:
```
docker compose -f docker-compose.yml -f docker-compose.prod.yml exec -T app \
  alembic current
docker compose -f docker-compose.yml -f docker-compose.prod.yml exec -T app \
  alembic downgrade -1
```
Only downgrade if the rolled-back code is incompatible with the new schema. Migrations should be written additively (see CLAUDE.md) so most rollbacks need no downgrade.

Restore from backup¶

The pg-backup sidecar writes nightly pg_dump custom-format dumps to the pg_backups volume (/backups inside the container), retained for BACKUP_RETENTION_DAYS (default 7). When BACKUP_RCLONE_REMOTE is set, each dump is also uploaded (encrypted, via an rclone crypt remote) offsite.

List available dumps:

docker compose -f docker-compose.yml -f docker-compose.prod.yml exec -T \
  pg-backup ls -lh /backups

If the local volume is gone (VM/disk loss), pull a dump back from offsite — the same crypt remote and rclone.conf decrypt it transparently:

docker compose -f docker-compose.yml -f docker-compose.prod.yml exec -T \
  pg-backup sh -c 'rclone lsl "$BACKUP_RCLONE_REMOTE"'
docker compose -f docker-compose.yml -f docker-compose.prod.yml exec -T \
  pg-backup sh -c 'rclone copy "$BACKUP_RCLONE_REMOTE/redhound_<timestamp>.dump" /backups'

⚠️ Restoring offsite dumps requires the crypt password from rclone.conf. If that file/password is lost, the encrypted dumps are unrecoverable — keep a copy in your password manager.

Stop the app + workers so nothing writes during restore (leave Postgres up):

docker compose -f docker-compose.yml -f docker-compose.prod.yml stop \
  app frontend analyst-worker langgraph-worker data-worker job-worker \
  result-consumer trigger-consumer

Restore into the database (this DROPs and recreates objects via --clean --if-exists):

DUMP=/backups/redhound_<timestamp>.dump
docker compose -f docker-compose.yml -f docker-compose.prod.yml exec -T \
  pg-backup pg_restore \
    --host=postgres --port=5432 --username=redhound --dbname=redhound \
    --clean --if-exists --no-owner "$DUMP"

(PGPASSWORD is already set in the pg-backup container's env.)

Verify the schema is at the expected revision:

docker compose -f docker-compose.yml -f docker-compose.prod.yml start app
docker compose -f docker-compose.yml -f docker-compose.prod.yml exec -T app \
  alembic current

Bring the rest of the stack back up:
```
make prod-up
```

Test restores periodically

An untested backup is not a backup. Restore the latest dump into a throwaway Postgres at least once before launch and confirm row counts.

Rotate secrets¶

When a credential leaks or on a routine rotation:

Generate the new value (see .env.prod.template for per-secret commands, e.g. openssl rand -base64 48 for REDHOUND_AUTH_SECRET).
Update the value in /opt/redhound/.env.
Recreate the affected services so they pick up the new env:

Secret rotated	Restart
`POSTGRES_PASSWORD`	all services (also run `ALTER ROLE redhound PASSWORD '...'` in Postgres first)
`OPENAI_API_KEY`, `REDHOUND_FMP_API_KEY`	`app` + all workers
`REDHOUND_AUTH_SECRET`	`app` (invalidates all sessions — users re-login)
`REDHOUND_ENCRYPTION_KEY`	`app` (⚠️ existing encrypted broker creds become unreadable — only rotate with a re-encryption plan)
`GF_SECURITY_ADMIN_PASSWORD`	`grafana`
`REDHOUND_AUTH_SMTP_PASSWORD`	`app`
`SLACK_ALERTMANAGER_WEBHOOK_URL`	`alertmanager`

docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d \
  --force-recreate <service>

4. For POSTGRES_PASSWORD, change it inside Postgres first, then recreate:

docker compose -f docker-compose.yml -f docker-compose.prod.yml exec -T postgres \
  psql -U redhound -c "ALTER ROLE redhound PASSWORD '<new>';"

Post-deploy smoke test¶

Run after every deploy. Stop and roll back if any step fails.

docker compose ... ps — all services healthy (not just running)
https://<host>/ loads the frontend
Login with a test account succeeds (cookie set, redirects to app)
Trigger one analysis end-to-end; it reaches a final decision
WebSocket live updates stream during the analysis
Grafana (ssh -L 3030:127.0.0.1:3030) panels render with data
docker compose ... logs --tail=50 result-consumer shows no errors
A test Prometheus alert reaches the Slack #redhound-alerts channel

Accessing admin dashboards¶

Admin services bind to 127.0.0.1 on the host only. Reach them via SSH tunnel:

ssh -L 9090:127.0.0.1:9090 \
    -L 3030:127.0.0.1:3030 \
    -L 8233:127.0.0.1:8233 \
    -L 9093:127.0.0.1:9093 \
    deploy@<host-ip>

Then open localhost:9090 (Prometheus), localhost:3030 (Grafana), localhost:8233 (Temporal UI), localhost:9093 (Alertmanager).