CI/CD Pipeline¶

This document describes the continuous integration and deployment pipeline for the Maricusco trading system.

Overview¶

The project uses GitHub Actions for CI/CD with a comprehensive pipeline that ensures code quality, security, and reliability before deployment.

Pipeline Architecture¶

graph TD
    Start[Push/PR] --> CheckLock[Check uv.lock Sync]
    CheckLock -->|In Sync| Parallel{Parallel Jobs}
    CheckLock -->|Out of Sync| AutoFix{Dependabot PR?}
    AutoFix -->|Yes| UpdateLock[Auto-update Lock]
    AutoFix -->|No| Fail[Fail: Run uv lock]
    UpdateLock --> NewRun[Trigger New Run]

    Parallel --> Lint[Lint & Format]
    Parallel --> TypeCheck[Type Check]
    Parallel --> Security[Security Scan]
    Parallel --> Secrets[Detect Secrets]
    Parallel --> PreCommit[Pre-commit Hooks]

    Lint --> Test[Test Suite]
    TypeCheck --> Test
    Security --> Test
    Secrets --> Test
    PreCommit --> Test

    Test --> MergeVal{PR Event?}
    MergeVal -->|Yes| ValidateMerge[Merge Validation]
    MergeVal -->|No| DockerBuild[Docker Build]

    ValidateMerge --> Notify[Notification]
    DockerBuild --> Notify

    Notify --> Success[✓ Success]

    style CheckLock fill:#4CAF50
    style Parallel fill:#2196F3
    style Test fill:#FF9800
    style Notify fill:#9C27B0
    style Fail fill:#F44336

Pipeline Stages¶

1. Check Lock¶

Purpose: Verify uv.lock is in sync with pyproject.toml

Actions: - Generate lock file with uv lock - Compare with committed lock file - Auto-update for Dependabot PRs - Fail for manual PRs with out-of-sync lock

Dependabot Auto-fix: When Dependabot creates a PR updating dependencies in pyproject.toml, the pipeline automatically: 1. Detects it's a Dependabot PR 2. Runs uv lock to update the lock file 3. Commits and pushes the updated uv.lock 4. Triggers a new workflow run with the updated lock file

Manual Fix:

# If check fails for your PR
make lock-sync
git add uv.lock
git commit -m "chore: update uv.lock"
git push

2. Lint and Format Check¶

Purpose: Enforce code style and quality standards

Tools: - ruff: Linting and formatting (replaces flake8, black, isort)

Checks: - Code formatting (PEP 8 compliance) - Import sorting - Unused imports and variables - Code complexity - Common code smells

Local Execution:

# Check formatting
uv run ruff format --check .

# Check linting
uv run ruff check .

# Auto-fix issues
uv run ruff check --fix .
uv run ruff format .

3. Type Check¶

Purpose: Validate type hints and catch type-related bugs

Tools: - pyright: Static type checker

Checks: - Type hint correctness - Type compatibility - Missing type annotations - Type inference issues

Local Execution:

# Run type checker
npx pyright maricusco/ cli/

# Or via pre-commit
uv run pre-commit run pyright --all-files

4. Security Scan¶

Purpose: Identify security vulnerabilities

Tools: - bandit: Python security linter - pip-audit: Dependency vulnerability scanner

Checks: - Common security issues (SQL injection, hardcoded passwords, etc.) - Known CVEs in dependencies - Insecure code patterns

Local Execution:

# Run bandit
uv run bandit -r maricusco/ cli/ -ll

# Run pip-audit
uv run pip-audit --desc

Note: Security scan continues on error (non-blocking) but results are reported.

5. Detect Secrets¶

Purpose: Prevent accidental commit of secrets

Tools: - detect-secrets: Secret detection

Checks: - API keys - Passwords - Private keys - Tokens - Other sensitive data

Baseline: .secrets.baseline file contains known false positives

Local Execution:

# Scan for secrets
uv run detect-secrets scan --baseline .secrets.baseline

# Update baseline (after verifying false positives)
uv run detect-secrets scan --update .secrets.baseline

6. Pre-commit Hooks¶

Purpose: Run additional quality checks

Hooks: - trailing-whitespace: Remove trailing whitespace - end-of-file-fixer: Ensure files end with newline - check-yaml: Validate YAML syntax - check-added-large-files: Prevent large file commits - check-merge-conflict: Detect merge conflict markers

Local Execution:

# Run all hooks
uv run pre-commit run --all-files

# Run specific hook
uv run pre-commit run trailing-whitespace --all-files

Note: Linting, type checking, and security scans are run separately in CI and skipped in the pre-commit stage to avoid duplication.

7. Test Suite¶

Purpose: Validate functionality and maintain code quality

Configuration: - Runs in mock mode (no API costs) - Parallel execution with pytest-xdist - Coverage threshold: 10% (interim, will increase) - Excludes slow and integration tests on PRs

Test Execution:

# PR tests (fast)
pytest -v --tb=short -n auto -m "not slow and not integration and not api" \
  --cov=maricusco --cov-report=xml --cov-fail-under=10

# Push tests (comprehensive)
pytest -v --tb=short -n auto \
  --cov=maricusco --cov-report=xml --cov-fail-under=10

Outputs: - JUnit XML for test results - Coverage XML for Codecov - Test result comments on PRs

Local Execution:

# Run tests like CI does
MARICUSCO_MOCK_MODE=true make test

# Run with coverage
MARICUSCO_MOCK_MODE=true pytest --cov=maricusco --cov-report=html

8. Merge Validation (PRs Only)¶

Purpose: Validate PR is ready for merge

Checks: - No merge conflicts - Required labels present (if configured) - All required checks passed

Required Labels: - dependencies: Required for dependency update PRs

Configuration:

# .github/workflows/cicd.yml
env:
  REQUIRED_PR_LABELS: "dependencies"  # Comma-separated

Override: Set to empty string to disable label requirement

9. Docker Build (Push Events Only)¶

Purpose: Build and scan Docker image

Actions: 1. Build Docker image with BuildKit 2. Test image (validate imports and CLI) 3. Scan with Trivy for vulnerabilities 4. Upload scan results as artifact

Vulnerability Thresholds: - CRITICAL: Fail build - HIGH: Fail build - MEDIUM: Report but pass - LOW: Report but pass

Local Execution:

# Build image
docker build -t maricusco:test .

# Test image
docker run --rm --entrypoint python maricusco:test -c "import maricusco; from maricusco.api.app import app"

# Scan image
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
  aquasec/trivy:latest image maricusco:test

10. Notification¶

Purpose: Send pipeline status to Slack

Information Sent: - Repository and branch - Commit message and author - Pipeline status (success/failure) - Job results (passed/failed) - Vulnerability summary (if Docker build ran) - Link to workflow run

Configuration:

Required GitHub secrets: - SLACK_BOT_TOKEN: Bot User OAuth Token (starts with xoxb-) - SLACK_CHANNEL_ID: Channel ID where notifications will be sent (starts with C)

Setting Up Slack App:

Create a Slack App:
Go to https://api.slack.com/apps
Click "Create New App" → "From scratch"
Enter app name (e.g., "CI/CD Notifications")
Select your Slack workspace
Click "Create App"
Configure Bot Token Scopes:
In the app settings, go to "OAuth & Permissions" (left sidebar)
Scroll to "Scopes" → "Bot Token Scopes"
Add the following scopes:
- chat:write - Send messages to channels
- chat:write.public - Send messages to public channels (if posting to public channels)
Install App to Workspace:
Scroll to top of "OAuth & Permissions" page
Click "Install to Workspace"
Review permissions and click "Allow"
Copy the "Bot User OAuth Token" (starts with xoxb-)
- This is your SLACK_BOT_TOKEN
Get Channel ID:
Open Slack in your browser
Navigate to the channel where you want notifications
Look at the URL: https://yourworkspace.slack.com/archives/C1234567890
The part after /archives/ is the Channel ID (starts with C)
Alternatively, right-click the channel → "View channel details" → Channel ID is at the bottom
Add Secrets to GitHub:
Go to your repository → Settings → Secrets and variables → Actions
Click "New repository secret"
Add SLACK_BOT_TOKEN with the bot token from step 3
Add SLACK_CHANNEL_ID with the channel ID from step 4

Note: Notification is optional and only runs if secrets are configured. If the bot is deleted or deactivated, you'll see account_inactive errors in workflow logs.

Additional Notification Workflows¶

Dependabot Notifications¶

Workflow: .github/workflows/dependabot-notifications.yml

Triggers: Dependabot PR opened or closed

Notifies: Update type (major/minor/patch) and PR status (opened/merged/closed)

Cloudflare Pages Deployment and Notifications¶

Workflow: .github/workflows/cloudflare-pages-deploy.yml

Triggers: Push to main/master/dev when files in docs/** change

Actions: 1. Build documentation using MkDocs 2. Deploy to Cloudflare Pages 3. Check deployment status via Cloudflare API 4. Send Slack notification with deployment status

Notifies: Documentation deployment status with commit message and deployment link

Note: Uses the same SLACK_BOT_TOKEN and SLACK_CHANNEL_ID secrets as CI/CD notifications. Notifications include error details if the Slack API call fails, making debugging easier.

Workflow Triggers¶

Push Events¶

Triggers on push to: - main branch - master branch - dev branch

When files change: - **.py (Python files) - pyproject.toml (dependencies) - uv.lock (lock file) - Dockerfile (container image) - docker-compose*.yml (compose files) - .pre-commit-config.yaml (pre-commit config) - .secrets.baseline (secrets baseline) - .github/workflows/** (workflow files)

Pull Request Events¶

Triggers on PR to any branch when same files change.

Manual Trigger¶

Can be manually triggered via GitHub Actions UI:

Actions → CI/CD → Run workflow

Concurrency Control¶

Strategy: Cancel in-progress runs when new commits are pushed

concurrency:
  group: ${{ github.workflow }}-${{ github.event.pull_request.head.ref || github.ref }}
  cancel-in-progress: true

Benefits: - Saves CI minutes - Faster feedback on latest changes - Prevents queue buildup

Caching Strategy¶

Virtual Environment Cache¶

- uses: actions/cache@v5
  with:
    path: .venv
    key: venv-${{ runner.os }}-${{ hashFiles('pyproject.toml', 'uv.lock') }}

Cache Hit: Skip dependency installation Cache Miss: Install dependencies and cache for next run

Pre-commit Cache¶

- uses: actions/cache@v5
  with:
    path: ~/.cache/pre-commit
    key: pre-commit-${{ runner.os }}-${{ hashFiles('.pre-commit-config.yaml') }}

Cache Hit: Reuse pre-commit environments Cache Miss: Install pre-commit hooks

Docker Build Cache¶

cache-from: type=gha
cache-to: type=gha,mode=max

Mode: max caches all layers for maximum cache hits Alternative: min caches only final layers (faster upload, less cache hits)

NPX Cache¶

- uses: actions/cache@v5
  with:
    path: ~/.npm
    key: npx-${{ runner.os }}-pyright-${{ env.PYRIGHT_VERSION }}

Cache Hit: Skip pyright download Cache Miss: Download pyright

Environment Variables¶

Version Pinning¶

All tool versions are pinned in workflow environment:

env:
  PYTHON_VERSION: "3.12.12"
  UV_VERSION: "0.9.13"
  RUFF_VERSION: "0.14.8"
  PYRIGHT_VERSION: "1.1.407"
  DETECT_SECRETS_VERSION: "1.5.0"
  # ... more versions

Benefits: - Reproducible builds - Explicit version control - Easy version updates

Configuration Variables¶

env:
  REQUIRED_PR_LABELS: "dependencies"  # Comma-separated labels

Reusable Actions¶

Setup Python and UV¶

Custom composite action for consistent setup:

# .github/actions/setup-python-uv/action.yml
- name: Setup Python and UV
  uses: ./.github/actions/setup-python-uv
  with:
    python-version: ${{ env.PYTHON_VERSION }}
    uv-version: ${{ env.UV_VERSION }}

Actions: 1. Install Python 2. Install uv 3. Add uv to PATH

PR Comment¶

Custom action for posting comments on PRs:

- name: Comment PR
  uses: ./.github/actions/pr-comment
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    message: |
      ## Security Scan Results
      ...

Permissions¶

Workflow uses minimal required permissions:

permissions:
  contents: read          # Read repository contents
  pull-requests: write    # Comment on PRs
  checks: write           # Write check results
  actions: read           # Read workflow runs
  security-events: write  # Write security events
  statuses: write         # Write commit statuses

Timeouts¶

Each job has a timeout to prevent hanging:

check-lock: 3 minutes
lint: 5 minutes
type-check: 8 minutes
security: 8 minutes
detect-secrets: 3 minutes
pre-commit: 5 minutes
test: 10 minutes
merge-validation: 2 minutes
docker-build: 8 minutes
notification: 2 minutes

Failure Handling¶

Job Dependencies¶

Jobs run in parallel when possible, with dependencies:

check-lock (required for all)
  ├── lint
  ├── type-check
  ├── security
  ├── detect-secrets
  └── pre-commit
      └── test (depends on all above)
          ├── merge-validation (PRs only)
          └── docker-build (pushes only)
              └── notification

Continue on Error¶

Some jobs continue on error: - security: Reports findings but doesn't fail - docker-build scan: Reports vulnerabilities as artifact

Conditional Execution¶

Jobs skip when not needed: - merge-validation: Only on PRs - docker-build: Only on pushes - notification: Only when core jobs succeed

Local CI Simulation¶

Run Full CI Locally¶

# 1. Check lock sync
make check-lock-sync

# 2. Run linting
uv run ruff format --check .
uv run ruff check .

# 3. Run type checking
npx pyright maricusco/ cli/

# 4. Run security scans
uv run bandit -r maricusco/ cli/ -ll
uv run pip-audit --desc

# 5. Run secret detection
uv run detect-secrets scan --baseline .secrets.baseline

# 6. Run pre-commit hooks
uv run pre-commit run --all-files

# 7. Run tests
MARICUSCO_MOCK_MODE=true pytest -v -n auto --cov=maricusco

# 8. Build Docker image
docker build -t maricusco:test .

Act (Run GitHub Actions Locally)¶

# Install act
brew install act  # macOS
# or
curl https://raw.githubusercontent.com/nektos/act/master/install.sh | sudo bash

# Run workflow locally
act push

# Run specific job
act -j test

# Run with secrets
act -s GITHUB_TOKEN=<token>

Troubleshooting¶

Lock File Out of Sync¶

Error: uv.lock is out of sync with pyproject.toml

Solution:

make lock-sync
git add uv.lock
git commit -m "chore: update uv.lock"
git push

Linting Failures¶

Error: Ruff linting or formatting errors

Solution:

# Auto-fix issues
uv run ruff check --fix .
uv run ruff format .

git add .
git commit -m "style: fix linting issues"
git push

Type Check Failures¶

Error: Pyright type errors

Solution:

# Run locally to see errors
npx pyright maricusco/ cli/

# Fix type errors in code
# Add type hints, fix type mismatches, etc.

git add .
git commit -m "fix: resolve type errors"
git push

Test Failures¶

Error: Pytest test failures

Solution:

# Run tests locally
MARICUSCO_MOCK_MODE=true pytest -v

# Debug specific test
MARICUSCO_MOCK_MODE=true pytest -v tests/path/to/test.py::test_name

# Fix failing tests
git add .
git commit -m "fix: resolve test failures"
git push

Docker Build Failures¶

Error: Docker image build or scan failures

Solution:

# Build locally
docker build -t maricusco:test .

# Check for errors in Dockerfile
# Fix dependency issues, etc.

git add Dockerfile
git commit -m "fix: resolve Docker build issues"
git push

Cache Issues¶

Symptoms: Unexpected failures, stale dependencies

Solution: 1. Go to GitHub Actions 2. Click on workflow run 3. Click "Re-run jobs" → "Re-run all jobs" 4. Check "Clear cache" option

Or manually clear cache:

# Via GitHub CLI
gh cache delete <cache-key>

# List caches
gh cache list

Best Practices¶

1. Keep Branches Green¶

Run tests locally before pushing
Fix CI failures immediately
Don't merge PRs with failing checks

2. Update Dependencies Automatically¶

The project uses fully automated dependency management with GitHub's native auto-merge:

Zero-Touch Workflow: - Dependabot creates PRs weekly (Sunday 9am) - CI automatically updates uv.lock for Dependabot PRs - Patch and minor updates enable auto-merge via dependabot/fetch-metadata action - PRs merge instantly when CI passes (no polling) - Major updates get labeled requires-review for manual approval

Efficiency Benefits: - 85% faster than polling-based solutions - ~25 CI minutes saved per week (no wait actions) - 100% accurate version detection using Dependabot metadata - Zero manual work for safe updates

How It Works: 1. Dependabot metadata action reads exact update type from PR 2. Workflow enables GitHub's native auto-merge for patch/minor 3. When CI passes, GitHub merges automatically (instant) 4. Major updates wait for manual review with requires-review label

What You Do: - Nothing for 90% of PRs (auto-merge handles them) - Review PRs labeled requires-review (1-2/month) and click "Merge" on GitHub - All updates tracked at: https://github.com/your-org/multi-agent-trading/pulls?q=author:dependabot

3. Monitor Pipeline Performance¶

Check workflow run times
Optimize slow jobs
Use caching effectively

4. Security First¶

Never commit secrets
Review security scan results
Update vulnerable dependencies promptly

5. Write Good Commit Messages¶

Follow Conventional Commits
Be descriptive
Reference issues when applicable

Viewing Pipeline Runs¶

# Via GitHub CLI
gh run list --limit 50

# Via GitHub UI
Actions → CI/CD → View workflow runs

References¶

Next Steps¶

Read Developer Onboarding for development setup
Read Mock Mode Guide for cost-free testing
Read Docker Setup for local services configuration