Skip to content

Sentry Setup & Slack Alerting Runbook

Redhound uses Sentry (SaaS) for error and crash tracking across the FastAPI backend, the Temporal/Kafka workers, and the Next.js frontend. Integration is a no-op until a DSN is configured, so local dev and tests stay quiet.

This runbook covers the one-time operator setup: creating the Sentry projects, mapping DSNs to env vars, wiring source maps + release tagging in CI, and routing alerts to Slack.

1. Create the Sentry projects

Create two projects in your Sentry org (separate projects keep backend and browser issues triaged independently):

Project Platform Covers
redhound-backend Python FastAPI API process + all workers (analyst, data, job, langgraph, result, trigger-consumer)
redhound-frontend Next.js Browser, RSC/server, and edge runtime

Each project has its own DSN (Settings → Client Keys (DSN)).

2. Map DSNs to environment variables

Set these in the deployment environment (the single-host docker-compose .env on the Hetzner box — see .env.example for the full list). Leave any blank to disable that surface.

Backend (API + workers — backend/observability/sentry.py::init_sentry):

SENTRY_DSN=https://<key>@<org>.ingest.sentry.io/<backend-project-id>
SENTRY_ENVIRONMENT=production         # or staging
SENTRY_RELEASE=<git-sha>              # set in CI, see §3
SENTRY_TRACES_SAMPLE_RATE=0.05        # performance trace sampling; malformed values fall back to 0.05

Frontend (frontend/instrumentation.ts, frontend/instrumentation-client.ts):

NEXT_PUBLIC_SENTRY_DSN=https://<key>@<org>.ingest.sentry.io/<frontend-project-id>
NEXT_PUBLIC_SENTRY_ENVIRONMENT=production
# Server/edge runtime reuses SENTRY_DSN / SENTRY_ENVIRONMENT (above).

The backend reads raw env (it initializes before the Pydantic config loads). init_sentry() is called early in create_app (API) and inside setup_logging() (every worker process), so a single DSN covers all six worker processes plus the API.

3. Source maps + release tagging (CI only)

The Next.js build uploads source maps to Sentry via withSentryConfig in frontend/next.config.ts. Upload is skipped silently when the auth token is absent, so it never breaks a local or env-less build. Provide these as CI secrets (not in the runtime .env):

SENTRY_ORG=<org-slug>
SENTRY_PROJECT=redhound-frontend
SENTRY_AUTH_TOKEN=<token>      # Sentry → Settings → Auth Tokens; scope: project:releases

For correlated releases, set SENTRY_RELEASE to the deployed git SHA in both the backend runtime env and the frontend build. Use the same SHA value on both projects so an issue can be traced to a single deploy.

4. Route alerts to Slack (native integration — no code)

Alerting for Sentry issues is configured in the Sentry UI, using Sentry's native Slack integration (this is separate from the Prometheus → Alertmanager → Slack pipeline that handles infra/cost/DLT/data-freshness alerts).

  1. Install the integration: Sentry → Settings → Integrations → Slack → Add to your workspace and authorize. Pick the channel (e.g. #redhound-alerts).
  2. Create an Issue Alert per project (Alerts → Create Alert → Issues):
  3. When: "A new issue is created" and "The issue changes state from resolved to unresolved" (regression).
  4. If: (optional) limit to environment = production to suppress staging noise.
  5. Then: "Send a Slack notification" → channel → include the issue link.
  6. Tune volume: add a rate-limit action ("if more than N events in M minutes") so a crash loop posts once, not thousands of times.

5. Verify

  • Backend: with SENTRY_DSN set, hit a route that raises (or temporarily add a throwaway raise RuntimeError("sentry smoke") endpoint) and confirm the issue lands in redhound-backend and posts to Slack. Remove the throwaway route.
  • Frontend: with NEXT_PUBLIC_SENTRY_DSN set, trigger a client error and confirm the issue appears in redhound-frontend with readable stack frames (source maps resolved) and the correct release.
  • No-op check: unset the DSNs in dev and confirm no events are sent and startup is clean.

Notes

  • Sentry coexists with the existing OpenTelemetry tracing (Tempo) — neither replaces the other.
  • The model-server (backend/model_server/) also picks up Sentry via setup_logging() if it runs with a DSN, but it is otherwise out of scope for this integration.