Spaces:

engineportf
/

math-backend

Sleeping

App Files Files Community

math-backend / docs /DEPLOY.md

engineportf

Upload folder using huggingface_hub

558db1e verified 16 days ago

preview code

Raw

History Blame Contribute Delete

10.6 kB

Deployment Module

Abstract

The deploy/ directory contains the infrastructure-as-code definitions required to deploy the Portfolio Engine as a production-grade, containerised microservice on Kubernetes. The deployment architecture follows a stateless application pattern backed by managed PostgreSQL and Redis services, orchestrated via Helm charts and Docker Compose for local development parity.

1. Architectural Overview

The production deployment comprises three services:

┌─────────────────────────────────────────────────────────────────┐
│                     Kubernetes Cluster                          │
│                                                                 │
│  ┌──────────────────┐  ┌──────────────┐  ┌──────────────────┐  │
│  │  portfolio-engine │  │  PostgreSQL  │  │      Redis       │  │
│  │  (Python 3.11)   │──│  (v15)       │  │  (v7)            │  │
│  │  Uvicorn / FastAPI│  │  Persistent  │  │  Ephemeral Cache │  │
│  │  Port 8080       │  │  Port 5432   │  │  Port 6379       │  │
│  └──────────────────┘  └──────────────┘  └──────────────────┘  │
└─────────────────────────────────────────────────────────────────┘

Portfolio Engine: Stateless Python application serving the FastAPI-based REST API via Uvicorn. Handles all optimisation, backtesting, and reporting logic.
PostgreSQL 15: Persistent storage for daily price data, portfolio state, and transaction history. The engine writes to the daily_prices table and reads during the data-loading phase.
Redis 7: Optional caching layer for rate-limited API responses and inter-process synchronisation in multi-replica deployments.

2. Container Image — `Dockerfile`

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8080
CMD ["python", "core_engine.py", "--headless"]

Design Decisions:

Base image: python:3.11-slim minimises attack surface and image size while retaining the manylinux wheels required by NumPy, SciPy, and CVXPY.
Dependency installation before source copy: Exploits Docker layer caching. Dependencies change infrequently; source code changes do not invalidate the pip install layer.
Headless mode: The default entrypoint runs the engine in non-interactive mode, suitable for scheduled batch jobs. For API serving, override with uvicorn api:app --host 0.0.0.0 --port 8080.

3. Local Development — `docker-compose.yml`

The Compose file provisions all three services for local development with production-equivalent topology.

Service Configuration

Service	Image	Ports	Health Check
`db`	`postgres:15`	5432	`pg_isready -U engine_user -d portfolio_db`
`redis`	`redis:7`	6379	`redis-cli ping`
`engine`	Build from `.`	8080	`curl -f http://localhost:8080/health`

Dependency Ordering

The engine container specifies depends_on with condition: service_healthy, ensuring that PostgreSQL and Redis are fully operational before the application starts. This prevents race conditions during cold-start data loading.

Environment Variables

Variable	Value
`DATABASE_URL`	`postgresql://engine_user:engine_password@db:5432/portfolio_db`
`REDIS_URL`	`redis://redis:6379/0`

Security Note. The Compose file uses plaintext credentials for local development only. Production deployments must inject secrets via Kubernetes Secrets, HashiCorp Vault, or an equivalent secrets manager.

Usage

# Build and start all services
docker-compose up --build

# Run in detached mode
docker-compose up -d

# View logs
docker-compose logs -f engine

# Tear down
docker-compose down -v

4. Kubernetes Deployment — Helm Chart

4.1 Chart Metadata — `Chart.yaml`

Field	Value
`apiVersion`	`v2` (Helm 3)
`name`	`portfolio-engine`
`description`	Quantitative Portfolio Allocation Engine
`type`	`application`
`version`	`0.1.0` (chart version)
`appVersion`	`1.0` (application version)

4.2 Configurable Values — `values.yaml`

Parameter	Default	Description
`replicaCount`	`2`	Number of engine pod replicas
`image.repository`	`portfolio-engine`	Container registry path
`image.pullPolicy`	`IfNotPresent`	Image pull policy
`image.tag`	`latest`	Image tag
`service.type`	`ClusterIP`	Kubernetes service type
`service.port`	`8080`	Service port
`env[0] (API_KEY)`	`changeme`	External data API key
`env[1] (DATABASE_URL)`	(empty)	PostgreSQL connection string
`resources.limits.cpu`	`2000m`	CPU limit per pod
`resources.limits.memory`	`4Gi`	Memory limit per pod
`resources.requests.cpu`	`500m`	CPU request per pod
`resources.requests.memory`	`1Gi`	Memory request per pod

Resource Sizing Rationale. The 4Gi memory limit accommodates the in-memory covariance matrices and Monte Carlo simulation arrays for portfolios of up to ~200 assets. The 2-core CPU limit supports CVXPY's OSQP solver, which benefits from vectorised linear algebra but does not parallelise across cores.

4.3 Templates

Template	Purpose
`_helpers.tpl`	Reusable Go template functions for generating consistent names and labels
`deployment.yaml`	Kubernetes Deployment resource defining pod spec, container image, ports, environment variables, and resource constraints
`service.yaml`	Kubernetes Service resource exposing the engine pods via ClusterIP

4.4 Deployment Commands

# Install the chart
helm install portfolio-engine ./deploy/helm/portfolio-engine \
  --set env[0].value="YOUR_API_KEY" \
  --set env[1].value="postgresql://user:pass@host:5432/db"

# Upgrade an existing release
helm upgrade portfolio-engine ./deploy/helm/portfolio-engine \
  --set image.tag="v2.0"

# Uninstall
helm uninstall portfolio-engine

# Dry-run to preview generated manifests
helm template portfolio-engine ./deploy/helm/portfolio-engine

5. CI/CD Pipeline — `.github/workflows/ci.yml`

The GitHub Actions workflow runs on every push and pull request to main:

Step	Tool	Purpose
Checkout	`actions/checkout@v3`	Clone repository
Python Setup	`actions/setup-python@v4`	Install Python 3.11
Install Dependencies	`pip install .[dev]`	Install production and development dependencies
Lint	`black --check .`	Enforce PEP 8 code formatting
Type Check	`mypy .`	Static type analysis
Test	`pytest tests/`	Run the full test suite

Pipeline Philosophy. The CI pipeline enforces a "green trunk" policy: no merge is permitted unless all linting, typing, and testing gates pass. This is consistent with the engine's design principle of no overfitting—code quality gates prevent regressions that could silently corrupt optimisation results.

6. Production Considerations

Scaling

The engine is designed for horizontal scaling behind a load balancer. Each pod is stateless; all persistent state resides in PostgreSQL. Redis can be used for distributed rate-limiting across replicas to honour external API quotas (max 2 requests/second globally).

Observability

The engine emits structured JSON logs (see output/engine.log) compatible with ELK, Datadog, or Grafana Loki. Health check endpoints at /health support Kubernetes liveness and readiness probes.

Security

API keys must never be committed to version control. Use Kubernetes Secrets or a secrets manager.
The DATABASE_URL should use TLS-encrypted connections (sslmode=require) in production.
Container images should be scanned for CVEs before deployment (e.g., Trivy, Snyk Container).

References

Kubernetes Documentation. (2024). Deployments. https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
Helm Documentation. (2024). Chart Template Guide. https://helm.sh/docs/chart_template_guide/
Docker Documentation. (2024). Best practices for writing Dockerfiles. https://docs.docker.com/develop/develop-images/dockerfile_best-practices/