Spaces:
Sleeping
Sleeping
File size: 10,581 Bytes
558db1e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 | # Deployment Module
## Abstract
The `deploy/` directory contains the infrastructure-as-code definitions required to deploy the Portfolio Engine as a production-grade, containerised microservice on Kubernetes. The deployment architecture follows a stateless application pattern backed by managed PostgreSQL and Redis services, orchestrated via Helm charts and Docker Compose for local development parity.
---
## 1. Architectural Overview
The production deployment comprises three services:
```
┌─────────────────────────────────────────────────────────────────┐
│ Kubernetes Cluster │
│ │
│ ┌──────────────────┐ ┌──────────────┐ ┌──────────────────┐ │
│ │ portfolio-engine │ │ PostgreSQL │ │ Redis │ │
│ │ (Python 3.11) │──│ (v15) │ │ (v7) │ │
│ │ Uvicorn / FastAPI│ │ Persistent │ │ Ephemeral Cache │ │
│ │ Port 8080 │ │ Port 5432 │ │ Port 6379 │ │
│ └──────────────────┘ └──────────────┘ └──────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
```
- **Portfolio Engine:** Stateless Python application serving the FastAPI-based REST API via Uvicorn. Handles all optimisation, backtesting, and reporting logic.
- **PostgreSQL 15:** Persistent storage for daily price data, portfolio state, and transaction history. The engine writes to the `daily_prices` table and reads during the data-loading phase.
- **Redis 7:** Optional caching layer for rate-limited API responses and inter-process synchronisation in multi-replica deployments.
---
## 2. Container Image — `Dockerfile`
```dockerfile
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8080
CMD ["python", "core_engine.py", "--headless"]
```
**Design Decisions:**
- **Base image:** `python:3.11-slim` minimises attack surface and image size while retaining the `manylinux` wheels required by NumPy, SciPy, and CVXPY.
- **Dependency installation before source copy:** Exploits Docker layer caching. Dependencies change infrequently; source code changes do not invalidate the pip install layer.
- **Headless mode:** The default entrypoint runs the engine in non-interactive mode, suitable for scheduled batch jobs. For API serving, override with `uvicorn api:app --host 0.0.0.0 --port 8080`.
---
## 3. Local Development — `docker-compose.yml`
The Compose file provisions all three services for local development with production-equivalent topology.
### Service Configuration
| Service | Image | Ports | Health Check |
|-----------|----------------|----------|-------------------------------------------------------|
| `db` | `postgres:15` | 5432 | `pg_isready -U engine_user -d portfolio_db` |
| `redis` | `redis:7` | 6379 | `redis-cli ping` |
| `engine` | Build from `.` | 8080 | `curl -f http://localhost:8080/health` |
### Dependency Ordering
The engine container specifies `depends_on` with `condition: service_healthy`, ensuring that PostgreSQL and Redis are fully operational before the application starts. This prevents race conditions during cold-start data loading.
### Environment Variables
| Variable | Value |
|----------------|------------------------------------------------------------------|
| `DATABASE_URL` | `postgresql://engine_user:engine_password@db:5432/portfolio_db` |
| `REDIS_URL` | `redis://redis:6379/0` |
**Security Note.** The Compose file uses plaintext credentials for local development only. Production deployments must inject secrets via Kubernetes Secrets, HashiCorp Vault, or an equivalent secrets manager.
### Usage
```bash
# Build and start all services
docker-compose up --build
# Run in detached mode
docker-compose up -d
# View logs
docker-compose logs -f engine
# Tear down
docker-compose down -v
```
---
## 4. Kubernetes Deployment — Helm Chart
### 4.1 Chart Metadata — `Chart.yaml`
| Field | Value |
|----------------|----------------------------------------|
| `apiVersion` | `v2` (Helm 3) |
| `name` | `portfolio-engine` |
| `description` | Quantitative Portfolio Allocation Engine |
| `type` | `application` |
| `version` | `0.1.0` (chart version) |
| `appVersion` | `1.0` (application version) |
### 4.2 Configurable Values — `values.yaml`
| Parameter | Default | Description |
|-------------------------|----------------------|------------------------------------------------|
| `replicaCount` | `2` | Number of engine pod replicas |
| `image.repository` | `portfolio-engine` | Container registry path |
| `image.pullPolicy` | `IfNotPresent` | Image pull policy |
| `image.tag` | `latest` | Image tag |
| `service.type` | `ClusterIP` | Kubernetes service type |
| `service.port` | `8080` | Service port |
| `env[0] (API_KEY)` | `changeme` | External data API key |
| `env[1] (DATABASE_URL)` | (empty) | PostgreSQL connection string |
| `resources.limits.cpu` | `2000m` | CPU limit per pod |
| `resources.limits.memory` | `4Gi` | Memory limit per pod |
| `resources.requests.cpu` | `500m` | CPU request per pod |
| `resources.requests.memory` | `1Gi` | Memory request per pod |
**Resource Sizing Rationale.** The 4Gi memory limit accommodates the in-memory covariance matrices and Monte Carlo simulation arrays for portfolios of up to ~200 assets. The 2-core CPU limit supports CVXPY's OSQP solver, which benefits from vectorised linear algebra but does not parallelise across cores.
### 4.3 Templates
| Template | Purpose |
|----------------------|----------------------------------------------------------------------|
| `_helpers.tpl` | Reusable Go template functions for generating consistent names and labels |
| `deployment.yaml` | Kubernetes Deployment resource defining pod spec, container image, ports, environment variables, and resource constraints |
| `service.yaml` | Kubernetes Service resource exposing the engine pods via ClusterIP |
### 4.4 Deployment Commands
```bash
# Install the chart
helm install portfolio-engine ./deploy/helm/portfolio-engine \
--set env[0].value="YOUR_API_KEY" \
--set env[1].value="postgresql://user:pass@host:5432/db"
# Upgrade an existing release
helm upgrade portfolio-engine ./deploy/helm/portfolio-engine \
--set image.tag="v2.0"
# Uninstall
helm uninstall portfolio-engine
# Dry-run to preview generated manifests
helm template portfolio-engine ./deploy/helm/portfolio-engine
```
---
## 5. CI/CD Pipeline — `.github/workflows/ci.yml`
The GitHub Actions workflow runs on every push and pull request to `main`:
| Step | Tool | Purpose |
|-----------------------|--------------|------------------------------------------------------|
| Checkout | `actions/checkout@v3` | Clone repository |
| Python Setup | `actions/setup-python@v4` | Install Python 3.11 |
| Install Dependencies | `pip install .[dev]` | Install production and development dependencies |
| Lint | `black --check .` | Enforce PEP 8 code formatting |
| Type Check | `mypy .` | Static type analysis |
| Test | `pytest tests/` | Run the full test suite |
**Pipeline Philosophy.** The CI pipeline enforces a "green trunk" policy: no merge is permitted unless all linting, typing, and testing gates pass. This is consistent with the engine's design principle of no overfitting—code quality gates prevent regressions that could silently corrupt optimisation results.
---
## 6. Production Considerations
### Scaling
The engine is designed for horizontal scaling behind a load balancer. Each pod is stateless; all persistent state resides in PostgreSQL. Redis can be used for distributed rate-limiting across replicas to honour external API quotas (max 2 requests/second globally).
### Observability
The engine emits structured JSON logs (see `output/engine.log`) compatible with ELK, Datadog, or Grafana Loki. Health check endpoints at `/health` support Kubernetes liveness and readiness probes.
### Security
- API keys must never be committed to version control. Use Kubernetes Secrets or a secrets manager.
- The `DATABASE_URL` should use TLS-encrypted connections (`sslmode=require`) in production.
- Container images should be scanned for CVEs before deployment (e.g., Trivy, Snyk Container).
---
## References
- Kubernetes Documentation. (2024). *Deployments*. https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
- Helm Documentation. (2024). *Chart Template Guide*. https://helm.sh/docs/chart_template_guide/
- Docker Documentation. (2024). *Best practices for writing Dockerfiles*. https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
|