ai-engineering-project / docs /PIPELINE_MONITORING.md
GitHub Action
Clean deployment without binary files
f884e6e

CI/CD Pipeline Monitoring Guide

πŸ” Where to Monitor Pipeline Execution

Primary Dashboard

GitHub Actions: https://github.com/sethmcknight/msse-ai-engineering/actions

Real-Time Monitoring Locations

1. GitHub Actions Tab

  • Navigate to your repository
  • Click the "Actions" tab
  • See all workflow runs with real-time status

2. Pull Request Checks

  • Open any Pull Request
  • Scroll to bottom to see "All checks have passed" or "Some checks were not successful"
  • Click "Details" to see full logs

3. Commit Status

  • Each commit shows βœ… or ❌ status
  • Click the status icon to see detailed results

πŸ›‘οΈ Pipeline Safeguards - Tests MUST Pass Before Deployment

Pipeline Flow:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                    SAFETY GATES                            β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

1. πŸ” Code Quality Checks (ALL MUST PASS)
   β”œβ”€β”€ Black formatting validation
   β”œβ”€β”€ isort import ordering
   β”œβ”€β”€ flake8 linting
   └── Pre-commit hooks (PR only)

2. πŸ§ͺ Comprehensive Testing (ALL MUST PASS)
   β”œβ”€β”€ Python 3.10 & 3.11 matrix testing
   β”œβ”€β”€ 27+ unit tests with coverage
   β”œβ”€β”€ HF embedding service tests (12 tests)
   β”œβ”€β”€ Citation validation tests
   β”œβ”€β”€ LLM component tests
   └── End-to-end pipeline validation

3. πŸš€ Deployment (ONLY IF TESTS PASS)
   β”œβ”€β”€ needs: test-hybrid-architecture ← CRITICAL DEPENDENCY
   β”œβ”€β”€ Deploy to Render
   β”œβ”€β”€ Deploy to HuggingFace Spaces
   └── Post-deployment validation

4. βœ… Final Validation
   └── Health checks across all platforms

πŸ”’ Critical Safety Configuration

Deployment Dependency Chain

deploy-to-render:
  needs: test-hybrid-architecture  # ← BLOCKS deployment if tests fail

deploy-to-huggingface:
  needs: test-hybrid-architecture  # ← BLOCKS deployment if tests fail

post-deployment-validation:
  needs: [deploy-to-render, deploy-to-huggingface]  # ← Waits for both deployments

Test Requirements

  • All 27+ tests must pass
  • Code formatting must be compliant
  • Citation validation must succeed
  • Service integration must work

πŸ“Š How to Monitor a Pipeline Run

Step 1: Trigger Pipeline

# Push to main or hf-main-local triggers pipeline
git push origin main

Step 2: Monitor in Real-Time

  1. Go to Actions: https://github.com/sethmcknight/msse-ai-engineering/actions
  2. Click latest workflow run (shows "Enhanced CI/CD - HuggingFace + Hybrid Architecture")
  3. Watch job progress:
    • 🟑 Yellow: Job running
    • βœ… Green: Job passed
    • ❌ Red: Job failed (BLOCKS deployment)

Step 3: Review Test Results

  • Click "test-hybrid-architecture" job
  • Expand test steps to see detailed results
  • Check coverage reports
  • Verify all 27+ tests passed

Step 4: Deployment Monitoring

  • Only runs if tests pass
  • Monitor Render deployment
  • Monitor HuggingFace deployment
  • Watch health check validation

🚨 What Happens if Tests Fail

Immediate Blocking

❌ test-hybrid-architecture job fails
    ↓
πŸ›‘ deploy-to-render: SKIPPED (dependency failed)
πŸ›‘ deploy-to-huggingface: SKIPPED (dependency failed)
πŸ›‘ post-deployment-validation: SKIPPED (dependencies failed)

Deployment Prevention

  • No code reaches production if any test fails
  • Email notification sent about failure
  • GitHub status shows ❌ on commit
  • Must fix tests before deployment can proceed

🎯 Emergency Override

If you need to skip deployment for testing:

git commit -m "Fix critical bug [skip-deploy]"

This will run tests but skip deployment.

πŸ“ˆ Pipeline Success Indicators

All Green Status

  • βœ… Pre-commit checks (PR only)
  • βœ… Code quality validation
  • βœ… 27+ comprehensive tests
  • βœ… Service integration tests
  • βœ… Citation validation
  • βœ… Render deployment + health check
  • βœ… HuggingFace deployment + health check
  • βœ… Post-deployment validation

Deployment URLs (only available after successful deployment)

πŸ”§ Test Locally Before Push

# Run the same tests locally
./scripts/hf_test_runner.sh

# Or run specific test categories
pytest tests/ -v
pytest -k "citation" -v
python scripts/test_e2e_pipeline.py

Bottom Line: The pipeline is configured with mandatory test gates that PREVENT deployment if any tests fail. Monitor at: https://github.com/sethmcknight/msse-ai-engineering/actions