Merge pull request #34 from se4ai2526-uniba/Milestone-6

Add GitHub Actions CI pipeline for unit tests and Docker image building.

.github/workflows/ci.yml

@@ -7,9 +7,8 @@ on:
     branches: [ "main" ]
 
 jobs:
-  build-and-test:
+  unit-tests:
     runs-on: ubuntu-latest
-
     steps:
     - name: Checkout code
       uses: actions/checkout@v3
@@ -26,7 +25,7 @@ jobs:
       uses: actions/setup-python@v4
       with:
         python-version: "3.10"
-        cache: 'pip' # Enable caching for pip
+        cache: 'pip'
 
     - name: Install dependencies
       run: |
@@ -35,17 +34,17 @@ jobs:
         pip install torch --index-url https://download.pytorch.org/whl/cpu
         # Install other dependencies
         pip install -r requirements.txt --no-cache-dir
+        pip install -e .
 
     - name: Lint with Ruff
       run: |
-        # Using make lint as defined in Makefile
         make lint
 
     - name: Run Unit Tests
       run: |
-        # Run tests and generate HTML report
         pytest tests/unit/ -v -m unit --html=report.html --self-contained-html
 
+    # Preserved contribution from Antonio Fratta
     - name: Upload Test Report
       if: failure()
       uses: actions/upload-artifact@v4
@@ -53,6 +52,32 @@ jobs:
         name: test-report
         path: report.html
 
+  build-image:
+    needs: unit-tests
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v3
+
+    - name: Free Disk Space
+      run: |
+        sudo rm -rf /usr/share/dotnet
+        sudo rm -rf /usr/local/lib/android
+        sudo rm -rf /opt/ghc
+        sudo rm -rf /opt/hostedtoolcache/CodeQL
+        sudo docker image prune --all --force
+
+    - name: Set up Python 3.10
+      uses: actions/setup-python@v4
+      with:
+        python-version: "3.10"
+        cache: 'pip'
+
+    - name: Install DVC
+      run: |
+        python -m pip install --upgrade pip
+        pip install dvc dvc-s3
+
     - name: Configure DVC
       run: |
         dvc remote modify origin --local auth basic

Dockerfile

@@ -14,6 +14,7 @@ RUN apt-get update && apt-get install -y \
     nginx \
     procps \
     curl \
+    prometheus \
     && rm -rf /var/lib/apt/lists/*
 
 # Create a non-root user

hopcroft_skill_classification_tool_competition/main.py

@@ -22,7 +22,7 @@ import os
 import time
 from typing import List
 
-from fastapi import FastAPI, HTTPException, status, Request, Response
+from fastapi import FastAPI, HTTPException, Request, Response, status
 from fastapi.responses import JSONResponse, RedirectResponse
 import mlflow
 from prometheus_client import (
@@ -135,20 +135,14 @@ async def monitor_requests(request: Request, call_next):
     try:
         response = await call_next(request)
         status_code = response.status_code
-        REQUESTS_TOTAL.labels(
-            method=method, endpoint=endpoint, http_status=status_code
-        ).inc()
+        REQUESTS_TOTAL.labels(method=method, endpoint=endpoint, http_status=status_code).inc()
         return response
     except Exception as e:
-        REQUESTS_TOTAL.labels(
-            method=method, endpoint=endpoint, http_status=500
-        ).inc()
+        REQUESTS_TOTAL.labels(method=method, endpoint=endpoint, http_status=500).inc()
         raise e
     finally:
         duration = time.time() - start_time
-        REQUEST_DURATION_SECONDS.labels(method=method, endpoint=endpoint).observe(
-            duration
-        )
+        REQUEST_DURATION_SECONDS.labels(method=method, endpoint=endpoint).observe(duration)
         IN_PROGRESS_REQUESTS.labels(method=method, endpoint=endpoint).dec()
 
 

monitoring/README.md

@@ -49,6 +49,11 @@ We used Better Stack Uptime to monitor the availability of the production deploy
 - https://dacrow13-hopcroft-skill-classification.hf.space/openapi.json
 - https://dacrow13-hopcroft-skill-classification.hf.space/docs
 
+## Prometheus on Hugging Face Space
+
+Prometheus is also running directly on the Hugging Face Space and is accessible at:
+- https://dacrow13-hopcroft-skill-classification.hf.space/prometheus/
+
 **Checks and alerts**
 - Monitors are configured to run from multiple locations.
 - Email notifications are enabled for failures.

monitoring/locust/README.md

@@ -28,16 +28,32 @@ Apri il browser e vai a: **http://localhost:8089**
 
 Nella Web UI, configura i seguenti parametri:
 
-| Parametro | Descrizione | Valore Consigliato |
+| Parametro | Descrizione | Valore di Default |
 |-----------|-------------|-------------------|
-| **Host** | URL dell'API da testare | `http://localhost:8080` (Docker) o `http://localhost:8000` (locale) |
-| **Number of users** | Numero totale di utenti simulati | 10-100 |
-| **Spawn rate** | Utenti da creare al secondo | 1-10 |
+| **Host** | URL dell'API da testare | `https://dacrow13-hopcroft-skill-classification.hf.space` |
+| **Number of users** | Numero totale di utenti simulati | 5-10 (HF Spaces) / 10-100 (locale) |
+| **Spawn rate** | Utenti da creare al secondo | 1-2 |
 
 ### 4. Avvia il Test
 
 Clicca su **"Start swarming"** per avviare il test di carico.
 
+## Ambienti di Test
+
+Lo script è preconfigurato per testare l'API deployata su **HuggingFace Spaces**:
+
+| Ambiente | URL | Note |
+|----------|-----|------|
+| **HuggingFace Spaces** (default) | `https://dacrow13-hopcroft-skill-classification.hf.space` | Produzione |
+| Docker locale | `http://localhost:8080` | Sviluppo con Docker |
+| Uvicorn locale | `http://localhost:8000` | Sviluppo senza Docker |
+
+Per testare un ambiente diverso, usa il flag `--host`:
+
+```bash
+locust -f locustfile.py --host http://localhost:8080
+```
+
 ## Task Implementati
 
 Lo script simula il comportamento di utenti reali con i seguenti task:
@@ -73,21 +89,25 @@ Durante il test, Locust fornisce le seguenti metriche in tempo reale:
 ### Esecuzione Headless (senza UI)
 
 ```bash
-locust -f locustfile.py --headless -u 50 -r 5 -t 5m --host http://localhost:8000
+# Test su HuggingFace Spaces (default)
+locust -f locustfile.py --headless -u 5 -r 1 -t 2m
+
+# Test su Docker locale
+locust -f locustfile.py --headless -u 50 -r 5 -t 5m --host http://localhost:8080
 ```
 
 | Opzione | Descrizione |
 |---------|-------------|
 | `--headless` | Esegui senza Web UI |
-| `-u 50` | 50 utenti simulati |
-| `-r 5` | 5 utenti creati al secondo |
-| `-t 5m` | Durata del test: 5 minuti |
-| `--host` | URL dell'API |
+| `-u 5` | 5 utenti simulati |
+| `-r 1` | 1 utente creato al secondo |
+| `-t 2m` | Durata del test: 2 minuti |
+| `--host` | URL dell'API (override) |
 
 ### Esportazione Risultati
 
 ```bash
-locust -f locustfile.py --headless -u 50 -r 5 -t 5m --host http://localhost:8000 --csv=results
+locust -f locustfile.py --headless -u 5 -r 1 -t 2m --csv=results
 ```
 
 Questo creerà file CSV con i risultati del test.
@@ -99,3 +119,9 @@ monitoring/locust/
 ├── locustfile.py   # Script principale di load testing
 └── README.md       # Questa documentazione
 ```
+
+## Note Importanti
+
+- **HuggingFace Spaces**: Usa un numero ridotto di utenti (5-10) per non sovraccaricare il servizio gratuito
+- **Latenza**: I test su HF Spaces avranno latenze maggiori rispetto ai test locali
+- **Cold Start**: Il primo request potrebbe essere lento se lo Space è in sleep mode

monitoring/locust/locustfile.py

@@ -19,8 +19,8 @@ class SkillClassificationUser(HttpUser):
     """
     
     # Default host for the API (can be overridden via --host flag or Web UI)
-    # Use http://localhost:8080 for Docker or http://localhost:8000 for local dev
-    host = "http://localhost:8080"
+    # HuggingFace Spaces URL (or use http://localhost:8080 for Docker, http://localhost:8000 for local dev)
+    host = "https://dacrow13-hopcroft-skill-classification.hf.space"
     
     # Wait between 1 and 5 seconds between tasks to simulate real user behavior
     wait_time = between(1, 5)

nginx.conf

@@ -29,6 +29,10 @@ http {
         server 127.0.0.1:8000;
     }
 
+    upstream prometheus {
+        server 127.0.0.1:9090;
+    }
+
     server {
         listen 7860;
         server_name localhost;
@@ -69,6 +73,15 @@ http {
             proxy_set_header Host $host;
         }
 
+        # Prometheus UI
+        location /prometheus/ {
+            proxy_pass http://prometheus/prometheus/;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+        }
+
         # Streamlit (Catch-all)
         location / {
             proxy_pass http://streamlit;

scripts/start_space.sh

@@ -52,6 +52,35 @@ for i in {1..30}; do
     sleep 2
 done
 
+echo "$(date) - Configuring and starting Prometheus..."
+# Create a config for the space
+cat <<EOF > /tmp/prometheus.yml
+global:
+  scrape_interval: 15s
+  evaluation_interval: 15s
+
+scrape_configs:
+  - job_name: 'hopcroft-api'
+    metrics_path: '/metrics'
+    static_configs:
+      - targets: ['127.0.0.1:8000']
+    scrape_interval: 10s
+
+  - job_name: 'prometheus'
+    static_configs:
+      - targets: ['127.0.0.1:9090']
+EOF
+
+# Start Prometheus
+# --web.external-url needs to match the path in Nginx
+prometheus \
+    --config.file=/tmp/prometheus.yml \
+    --storage.tsdb.path=/tmp/prometheus_data \
+    --web.listen-address=0.0.0.0:9090 \
+    --web.external-url=/prometheus/ \
+    --web.route-prefix=/prometheus/ \
+    >> /tmp/prometheus.log 2>&1 &
+
 echo "$(date) - Starting Nginx reverse proxy..."
 if ! command -v nginx &> /dev/null; then
     echo "$(date) - ERROR: nginx not found in PATH"