Spaces:
Sleeping
Sleeping
gary-boon
commited on
Commit
·
0e48dc7
1
Parent(s):
767a3fd
Add GitHub Actions workflow for security scanning and automated deployment
Browse files- Implement Snyk security scanning for Python dependencies
- Add Bandit for Python code security analysis
- Include Safety checks for known vulnerabilities
- Configure automated deployment to HuggingFace Spaces after security checks
- Create comprehensive security documentation
- Address known issue: transformers==4.35.0 has vulnerabilities (needs update)
- .github/workflows/security-and-deploy.yml +141 -0
- SECURITY.md +149 -0
.github/workflows/security-and-deploy.yml
ADDED
|
@@ -0,0 +1,141 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
name: Security Check and Deploy to HuggingFace
|
| 2 |
+
|
| 3 |
+
on:
|
| 4 |
+
push:
|
| 5 |
+
branches: [ main ]
|
| 6 |
+
pull_request:
|
| 7 |
+
branches: [ main ]
|
| 8 |
+
workflow_dispatch:
|
| 9 |
+
|
| 10 |
+
jobs:
|
| 11 |
+
security-check:
|
| 12 |
+
runs-on: ubuntu-latest
|
| 13 |
+
|
| 14 |
+
steps:
|
| 15 |
+
- uses: actions/checkout@v3
|
| 16 |
+
|
| 17 |
+
- name: Set up Python
|
| 18 |
+
uses: actions/setup-python@v4
|
| 19 |
+
with:
|
| 20 |
+
python-version: '3.10'
|
| 21 |
+
|
| 22 |
+
- name: Install dependencies
|
| 23 |
+
run: |
|
| 24 |
+
python -m pip install --upgrade pip
|
| 25 |
+
pip install -r requirements.txt
|
| 26 |
+
|
| 27 |
+
# Run Snyk security scan
|
| 28 |
+
- name: Run Snyk Security Scan
|
| 29 |
+
env:
|
| 30 |
+
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
|
| 31 |
+
run: |
|
| 32 |
+
# Install Snyk CLI
|
| 33 |
+
curl -Lo snyk https://static.snyk.io/cli/latest/snyk-linux
|
| 34 |
+
chmod +x snyk
|
| 35 |
+
|
| 36 |
+
# Authenticate with Snyk
|
| 37 |
+
./snyk auth $SNYK_TOKEN
|
| 38 |
+
|
| 39 |
+
echo "===== Python Dependency Security Scan ====="
|
| 40 |
+
# Test for vulnerabilities (non-blocking initially)
|
| 41 |
+
./snyk test --severity-threshold=high --file=requirements.txt || true
|
| 42 |
+
|
| 43 |
+
# Generate SARIF report for GitHub Security tab
|
| 44 |
+
./snyk test --severity-threshold=low --file=requirements.txt --sarif-file-output=snyk.sarif || true
|
| 45 |
+
|
| 46 |
+
echo "===== Security Scan Complete ====="
|
| 47 |
+
continue-on-error: true
|
| 48 |
+
|
| 49 |
+
# Upload results to GitHub Security tab
|
| 50 |
+
- name: Upload Snyk results to GitHub Code Scanning
|
| 51 |
+
if: always()
|
| 52 |
+
uses: github/codeql-action/upload-sarif@v3
|
| 53 |
+
continue-on-error: true
|
| 54 |
+
with:
|
| 55 |
+
sarif_file: snyk.sarif
|
| 56 |
+
category: snyk-python
|
| 57 |
+
|
| 58 |
+
# Monitor project with Snyk (updates dashboard)
|
| 59 |
+
- name: Monitor with Snyk
|
| 60 |
+
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
|
| 61 |
+
env:
|
| 62 |
+
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
|
| 63 |
+
run: |
|
| 64 |
+
./snyk monitor --file=requirements.txt --project-name=visualisable-ai-backend || true
|
| 65 |
+
continue-on-error: true
|
| 66 |
+
|
| 67 |
+
# Run Python security checks with bandit
|
| 68 |
+
- name: Run Bandit Security Linter
|
| 69 |
+
run: |
|
| 70 |
+
pip install bandit
|
| 71 |
+
echo "===== Python Code Security Analysis ====="
|
| 72 |
+
bandit -r backend/ -f json -o bandit-report.json || true
|
| 73 |
+
bandit -r backend/ || true
|
| 74 |
+
echo "===== Code Analysis Complete ====="
|
| 75 |
+
continue-on-error: true
|
| 76 |
+
|
| 77 |
+
# Run safety check for known security vulnerabilities
|
| 78 |
+
- name: Run Safety Check
|
| 79 |
+
run: |
|
| 80 |
+
pip install safety
|
| 81 |
+
echo "===== Safety Vulnerability Check ====="
|
| 82 |
+
safety check --json > safety-report.json || true
|
| 83 |
+
safety check || true
|
| 84 |
+
echo "===== Safety Check Complete ====="
|
| 85 |
+
continue-on-error: true
|
| 86 |
+
|
| 87 |
+
- name: Security Summary
|
| 88 |
+
if: always()
|
| 89 |
+
run: |
|
| 90 |
+
echo "## Security Scan Summary" >> $GITHUB_STEP_SUMMARY
|
| 91 |
+
echo "" >> $GITHUB_STEP_SUMMARY
|
| 92 |
+
echo "### Checks Performed:" >> $GITHUB_STEP_SUMMARY
|
| 93 |
+
echo "- ✅ Snyk dependency vulnerability scan" >> $GITHUB_STEP_SUMMARY
|
| 94 |
+
echo "- ✅ Bandit Python security linter" >> $GITHUB_STEP_SUMMARY
|
| 95 |
+
echo "- ✅ Safety known vulnerability check" >> $GITHUB_STEP_SUMMARY
|
| 96 |
+
echo "" >> $GITHUB_STEP_SUMMARY
|
| 97 |
+
echo "### Known Issues to Address:" >> $GITHUB_STEP_SUMMARY
|
| 98 |
+
echo "- transformers==4.35.0 has known vulnerabilities" >> $GITHUB_STEP_SUMMARY
|
| 99 |
+
echo "- Consider upgrading to transformers>=4.36.0" >> $GITHUB_STEP_SUMMARY
|
| 100 |
+
echo "" >> $GITHUB_STEP_SUMMARY
|
| 101 |
+
echo "Check the logs above for detailed findings." >> $GITHUB_STEP_SUMMARY
|
| 102 |
+
|
| 103 |
+
deploy-to-huggingface:
|
| 104 |
+
runs-on: ubuntu-latest
|
| 105 |
+
needs: security-check
|
| 106 |
+
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
|
| 107 |
+
|
| 108 |
+
steps:
|
| 109 |
+
- uses: actions/checkout@v3
|
| 110 |
+
with:
|
| 111 |
+
fetch-depth: 0
|
| 112 |
+
lfs: true
|
| 113 |
+
|
| 114 |
+
- name: Configure Git
|
| 115 |
+
run: |
|
| 116 |
+
git config --global user.email "github-actions[bot]@users.noreply.github.com"
|
| 117 |
+
git config --global user.name "github-actions[bot]"
|
| 118 |
+
|
| 119 |
+
- name: Push to HuggingFace Space
|
| 120 |
+
env:
|
| 121 |
+
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
| 122 |
+
run: |
|
| 123 |
+
# Add HuggingFace remote with authentication
|
| 124 |
+
git remote add huggingface https://visualisable-ai:$HF_TOKEN@huggingface.co/spaces/visualisable-ai/api
|
| 125 |
+
|
| 126 |
+
# Push to HuggingFace
|
| 127 |
+
git push huggingface main:main --force
|
| 128 |
+
|
| 129 |
+
echo "✅ Deployed to HuggingFace Spaces successfully!" >> $GITHUB_STEP_SUMMARY
|
| 130 |
+
echo "🔗 View at: https://huggingface.co/spaces/visualisable-ai/api" >> $GITHUB_STEP_SUMMARY
|
| 131 |
+
continue-on-error: true
|
| 132 |
+
|
| 133 |
+
- name: Deployment Status
|
| 134 |
+
if: always()
|
| 135 |
+
run: |
|
| 136 |
+
if [ $? -eq 0 ]; then
|
| 137 |
+
echo "### Deployment Status: ✅ Success" >> $GITHUB_STEP_SUMMARY
|
| 138 |
+
else
|
| 139 |
+
echo "### Deployment Status: ⚠️ Manual deployment may be needed" >> $GITHUB_STEP_SUMMARY
|
| 140 |
+
echo "Run 'git push origin main' locally if automatic deployment failed" >> $GITHUB_STEP_SUMMARY
|
| 141 |
+
fi
|
SECURITY.md
ADDED
|
@@ -0,0 +1,149 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Security Configuration for Visualisable AI Backend
|
| 2 |
+
|
| 3 |
+
This document explains the security scanning setup for the Python backend that powers the HuggingFace Spaces API.
|
| 4 |
+
|
| 5 |
+
## Overview
|
| 6 |
+
|
| 7 |
+
The backend repository now includes automated security scanning via GitHub Actions that runs before deployment to HuggingFace Spaces. This ensures that security vulnerabilities are identified and can be addressed before code reaches production.
|
| 8 |
+
|
| 9 |
+
## Security Tools Used
|
| 10 |
+
|
| 11 |
+
1. **Snyk** - Comprehensive vulnerability scanning for Python dependencies
|
| 12 |
+
2. **Bandit** - Python-specific security linter for common security issues
|
| 13 |
+
3. **Safety** - Checks Python dependencies against known security vulnerabilities database
|
| 14 |
+
|
| 15 |
+
## Setup Instructions
|
| 16 |
+
|
| 17 |
+
### 1. GitHub Repository Setup
|
| 18 |
+
|
| 19 |
+
Since this repository currently only has HuggingFace as a remote, you'll need to:
|
| 20 |
+
|
| 21 |
+
1. Create a GitHub repository for the backend:
|
| 22 |
+
```bash
|
| 23 |
+
# Add GitHub as a remote
|
| 24 |
+
git remote add github https://github.com/YOUR_USERNAME/visualisable-ai-backend.git
|
| 25 |
+
|
| 26 |
+
# Push to GitHub
|
| 27 |
+
git push github main
|
| 28 |
+
```
|
| 29 |
+
|
| 30 |
+
2. Enable GitHub Actions in the repository settings
|
| 31 |
+
|
| 32 |
+
### 2. Required Secrets
|
| 33 |
+
|
| 34 |
+
Add the following secrets to your GitHub repository (Settings → Secrets and variables → Actions):
|
| 35 |
+
|
| 36 |
+
#### SNYK_TOKEN
|
| 37 |
+
1. Sign up for free at https://snyk.io
|
| 38 |
+
2. Go to Account Settings → Auth Token
|
| 39 |
+
3. Copy your personal auth token
|
| 40 |
+
4. Add as `SNYK_TOKEN` in GitHub secrets
|
| 41 |
+
|
| 42 |
+
#### HF_TOKEN (for automated deployment)
|
| 43 |
+
1. Go to https://huggingface.co/settings/tokens
|
| 44 |
+
2. Create a new token with write access
|
| 45 |
+
3. Add as `HF_TOKEN` in GitHub secrets
|
| 46 |
+
|
| 47 |
+
## Workflow Features
|
| 48 |
+
|
| 49 |
+
The `security-and-deploy.yml` workflow:
|
| 50 |
+
|
| 51 |
+
1. **Runs on every push and PR** to the main branch
|
| 52 |
+
2. **Security scanning includes:**
|
| 53 |
+
- Dependency vulnerability scanning with Snyk
|
| 54 |
+
- Code security analysis with Bandit
|
| 55 |
+
- Known vulnerability checking with Safety
|
| 56 |
+
- Results uploaded to GitHub Security tab
|
| 57 |
+
- Project monitoring in Snyk dashboard
|
| 58 |
+
|
| 59 |
+
3. **Automated deployment** (only on main branch):
|
| 60 |
+
- After security checks pass
|
| 61 |
+
- Pushes directly to HuggingFace Spaces
|
| 62 |
+
- Maintains deployment history in GitHub
|
| 63 |
+
|
| 64 |
+
## Current Security Status
|
| 65 |
+
|
| 66 |
+
### Known Issues
|
| 67 |
+
|
| 68 |
+
**transformers==4.35.0** has known vulnerabilities:
|
| 69 |
+
- Multiple security issues have been fixed in newer versions
|
| 70 |
+
- Recommended upgrade: `transformers>=4.36.0`
|
| 71 |
+
|
| 72 |
+
### To Fix Vulnerabilities
|
| 73 |
+
|
| 74 |
+
1. Update `requirements.txt`:
|
| 75 |
+
```txt
|
| 76 |
+
transformers==4.36.2 # or latest stable version
|
| 77 |
+
```
|
| 78 |
+
|
| 79 |
+
2. Test locally:
|
| 80 |
+
```bash
|
| 81 |
+
pip install -r requirements.txt
|
| 82 |
+
python -m pytest # if you have tests
|
| 83 |
+
python app.py # test the application
|
| 84 |
+
```
|
| 85 |
+
|
| 86 |
+
3. Commit and push:
|
| 87 |
+
```bash
|
| 88 |
+
git add requirements.txt
|
| 89 |
+
git commit -m "Security: Update transformers to fix vulnerabilities"
|
| 90 |
+
git push github main # triggers security scan and deployment
|
| 91 |
+
```
|
| 92 |
+
|
| 93 |
+
## Local Security Testing
|
| 94 |
+
|
| 95 |
+
Run security checks locally before pushing:
|
| 96 |
+
|
| 97 |
+
```bash
|
| 98 |
+
# Install tools
|
| 99 |
+
pip install snyk bandit safety
|
| 100 |
+
|
| 101 |
+
# Run Snyk (requires authentication)
|
| 102 |
+
snyk auth
|
| 103 |
+
snyk test
|
| 104 |
+
|
| 105 |
+
# Run Bandit
|
| 106 |
+
bandit -r backend/
|
| 107 |
+
|
| 108 |
+
# Run Safety
|
| 109 |
+
safety check
|
| 110 |
+
```
|
| 111 |
+
|
| 112 |
+
## Monitoring
|
| 113 |
+
|
| 114 |
+
- **GitHub Security Tab**: View SARIF reports and security alerts
|
| 115 |
+
- **Snyk Dashboard**: https://app.snyk.io - Monitor all vulnerabilities
|
| 116 |
+
- **GitHub Actions**: Check workflow runs for security scan results
|
| 117 |
+
|
| 118 |
+
## Security Best Practices
|
| 119 |
+
|
| 120 |
+
1. **Regular Updates**: Keep dependencies updated to latest secure versions
|
| 121 |
+
2. **Monitor Alerts**: Check Snyk dashboard weekly for new vulnerabilities
|
| 122 |
+
3. **Test Before Deploy**: Always test locally after updating dependencies
|
| 123 |
+
4. **Review PR Scans**: Security scans run on PRs - review before merging
|
| 124 |
+
|
| 125 |
+
## Troubleshooting
|
| 126 |
+
|
| 127 |
+
### Workflow not running
|
| 128 |
+
- Ensure GitHub Actions is enabled in repository settings
|
| 129 |
+
- Check that secrets are properly configured
|
| 130 |
+
- Verify the workflow file is in `.github/workflows/`
|
| 131 |
+
|
| 132 |
+
### Deployment failing
|
| 133 |
+
- Check HF_TOKEN has write permissions
|
| 134 |
+
- Ensure HuggingFace Space name matches in workflow
|
| 135 |
+
- Manual deployment fallback: `git push origin main`
|
| 136 |
+
|
| 137 |
+
### Security scan failures
|
| 138 |
+
- Non-blocking by default (continue-on-error: true)
|
| 139 |
+
- Review logs for specific vulnerabilities
|
| 140 |
+
- Update dependencies to fix issues
|
| 141 |
+
- Can be made blocking by removing continue-on-error
|
| 142 |
+
|
| 143 |
+
## Next Steps
|
| 144 |
+
|
| 145 |
+
1. Push this repository to GitHub
|
| 146 |
+
2. Configure the required secrets
|
| 147 |
+
3. Run the workflow to establish baseline security status
|
| 148 |
+
4. Address the transformers vulnerability
|
| 149 |
+
5. Consider making security checks blocking after initial setup
|