Spaces:
Sleeping
Sleeping
CI/CD Pipeline Monitoring Guide
π Where to Monitor Pipeline Execution
Primary Dashboard
GitHub Actions: https://github.com/sethmcknight/msse-ai-engineering/actions
Real-Time Monitoring Locations
1. GitHub Actions Tab
- Navigate to your repository
- Click the "Actions" tab
- See all workflow runs with real-time status
2. Pull Request Checks
- Open any Pull Request
- Scroll to bottom to see "All checks have passed" or "Some checks were not successful"
- Click "Details" to see full logs
3. Commit Status
- Each commit shows β or β status
- Click the status icon to see detailed results
π‘οΈ Pipeline Safeguards - Tests MUST Pass Before Deployment
Pipeline Flow:
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β SAFETY GATES β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
1. π Code Quality Checks (ALL MUST PASS)
βββ Black formatting validation
βββ isort import ordering
βββ flake8 linting
βββ Pre-commit hooks (PR only)
2. π§ͺ Comprehensive Testing (ALL MUST PASS)
βββ Python 3.10 & 3.11 matrix testing
βββ 27+ unit tests with coverage
βββ HF embedding service tests (12 tests)
βββ Citation validation tests
βββ LLM component tests
βββ End-to-end pipeline validation
3. π Deployment (ONLY IF TESTS PASS)
βββ needs: test-hybrid-architecture β CRITICAL DEPENDENCY
βββ Deploy to Render
βββ Deploy to HuggingFace Spaces
βββ Post-deployment validation
4. β
Final Validation
βββ Health checks across all platforms
π Critical Safety Configuration
Deployment Dependency Chain
deploy-to-render:
needs: test-hybrid-architecture # β BLOCKS deployment if tests fail
deploy-to-huggingface:
needs: test-hybrid-architecture # β BLOCKS deployment if tests fail
post-deployment-validation:
needs: [deploy-to-render, deploy-to-huggingface] # β Waits for both deployments
Test Requirements
- All 27+ tests must pass
- Code formatting must be compliant
- Citation validation must succeed
- Service integration must work
π How to Monitor a Pipeline Run
Step 1: Trigger Pipeline
# Push to main or hf-main-local triggers pipeline
git push origin main
Step 2: Monitor in Real-Time
- Go to Actions: https://github.com/sethmcknight/msse-ai-engineering/actions
- Click latest workflow run (shows "Enhanced CI/CD - HuggingFace + Hybrid Architecture")
- Watch job progress:
- π‘ Yellow: Job running
- β Green: Job passed
- β Red: Job failed (BLOCKS deployment)
Step 3: Review Test Results
- Click "test-hybrid-architecture" job
- Expand test steps to see detailed results
- Check coverage reports
- Verify all 27+ tests passed
Step 4: Deployment Monitoring
- Only runs if tests pass
- Monitor Render deployment
- Monitor HuggingFace deployment
- Watch health check validation
π¨ What Happens if Tests Fail
Immediate Blocking
β test-hybrid-architecture job fails
β
π deploy-to-render: SKIPPED (dependency failed)
π deploy-to-huggingface: SKIPPED (dependency failed)
π post-deployment-validation: SKIPPED (dependencies failed)
Deployment Prevention
- No code reaches production if any test fails
- Email notification sent about failure
- GitHub status shows β on commit
- Must fix tests before deployment can proceed
π― Emergency Override
If you need to skip deployment for testing:
git commit -m "Fix critical bug [skip-deploy]"
This will run tests but skip deployment.
π Pipeline Success Indicators
All Green Status
- β Pre-commit checks (PR only)
- β Code quality validation
- β 27+ comprehensive tests
- β Service integration tests
- β Citation validation
- β Render deployment + health check
- β HuggingFace deployment + health check
- β Post-deployment validation
Deployment URLs (only available after successful deployment)
- Render: Available in workflow logs
- HF Team: https://huggingface.co/spaces/msse-team-3/ai-engineering-project
- HF Personal: https://huggingface.co/spaces/sethmcknight/msse-ai-engineering
π§ Test Locally Before Push
# Run the same tests locally
./scripts/hf_test_runner.sh
# Or run specific test categories
pytest tests/ -v
pytest -k "citation" -v
python scripts/test_e2e_pipeline.py
Bottom Line: The pipeline is configured with mandatory test gates that PREVENT deployment if any tests fail. Monitor at: https://github.com/sethmcknight/msse-ai-engineering/actions