Skip to content

Model Server

The model-server (backend/model_server/) is a standalone gRPC inference service. It is the only place where torch/transformers are loaded — the Temporal workers stay slim and call it over gRPC. In production it runs as a dedicated pod; the analyst-worker reaches it for DeBERTa sentiment inference.

  • gRPC: port 50051 (inference)
  • HTTP: port 8080 (/health, /ready)
  • All models load at startup; /ready gates traffic so a pod only receives requests once its weights are resident.
  • The Docker image bakes the model weights in at build time (make build-model-server).

Running locally

make run-model-server        # start locally (gRPC :50051, HTTP :8080)
make model-server-health     # check /health + /ready
make build-model-server      # build the Docker image (weights baked in)
make proto-gen               # regenerate gRPC stubs after editing model_service.proto

Workers connect using REDHOUND_MODEL_SERVER_ADDR (e.g. model-server:50051).

Architecture

File Responsibility
main.py Process entrypoint — starts the gRPC server + HTTP health app, loads configured models
servicer.py gRPC servicer — routes each request to a model by model_name
registry.py Loads and holds the active model instances
health.py /health and /ready HTTP endpoints
models/base.py BaseInferenceModel protocol that every model implements
models/__init__.py MODEL_MAP — string name → model class
models/deberta_sentiment.py DeBERTa financial-news sentiment model (default)

Callers pass model_name as a string, so no proto changes are needed to add a model — the servicer looks it up in MODEL_MAP.

Adding a new model

  1. Create backend/model_server/models/<your_model>.py implementing BaseInferenceModel.
  2. Register it in MODEL_MAP in backend/model_server/models/__init__.py.
  3. Add its name to REDHOUND_MODEL_SERVER_MODELS.
  4. No proto changes required — callers pass model_name as a string.

Environment variables

REDHOUND_MODEL_SERVER_MODELS=deberta-sentiment         # comma-separated; add new models here
REDHOUND_MODEL_SERVER_ADDR=model-server:50051          # used by workers to connect
REDHOUND_MODEL_SERVER_GRPC_PORT=50051
REDHOUND_MODEL_SERVER_HTTP_PORT=8080
REDHOUND_MODEL_DEBERTA_SENTIMENT_HF_NAME=mrm8488/deberta-v3-ft-financial-news-sentiment-analysis

See Event-Driven Workers for how the analyst-worker calls the model-server during the sentiment activity.