PRODUCTION_TESTING_RUNBOOK.md

🚀 PRODUCTION READINESS TESTING GUIDE

Status: Before deployment to HuggingFace Spaces
Time Required: 3 hours
Tools Needed: Browser, Python 3.7+, requests library

---

📋 WHAT'S CHANGED

Added: Comprehensive Production Logging

• Every API endpoint now logs:
  • Operation start/end
  • Success/failure status
  • Input parameters (paths, filenames)
  • Execution time
  • Exceptions with stack traces

Example log output:

[UPLOAD_FILE] START | user=default_user | file=document.pdf | folder=reports | size=1048576
[UPLOAD_FILE] SUCCESS | user=default_user | file=document.pdf | folder=reports
[RENAME] START | user=default_user | path=reports | new_name=archive_reports
[RENAME] SUCCESS | user=default_user | old_path=reports | new_name=archive_reports

Added: Automated Stress Test Script

stress_test.py - Runs 5 comprehensive stress tests:

1. Bulk Upload (50 files)
2. Folder Rename with 30 files
3. Rapid Operations (upload→delete→rename)
4. Cache Behavior (TTL validation)
5. Error Handling (invalid operations)

---

🧪 TESTING EXECUTION PLAN

PHASE 1: Setup (15 minutes)

Step 1: Verify Logging

# Check that logging is configured correctly
cd c:\Users\mohat\OneDrive\Desktop\Doc
python -c "from server.utils.logger import setup_logger; logger = setup_logger('test'); logger.info('Logging works')"

Expected output: Should see log message in console or logs/ directory

Step 2: Install Test Dependencies

pip install requests

---

PHASE 2: Manual Testing (45 minutes)

Test A: Basic Rename (NEW FEATURE)

Goal: Verify rename works end-to-end

1. Open: http://localhost:5000 (or HF Spaces URL)
2. Upload test file: rename_test.txt
3. Right-click file → "Rename" → renamed_file.txt
4. Verify: File appears with new name
5. Download to verify content intact
6. Check logs: Should see [RENAME] SUCCESS

Pass Criteria:

• ✓ File renamed successfully
• ✓ UI updates immediately
• ✓ Log shows [RENAME] SUCCESS

Test B: Folder Operations

Goal: Verify folders work with new rename

1. Create folder: TestFolder
2. Upload 5 files into it
3. Rename folder: TestFolder → RenamedFolder
4. Verify: All 5 files still accessible in new folder
5. Check logs: Should see [RENAME] SUCCESS with all files preserved

Pass Criteria:

• ✓ Folder renamed with contents
• ✓ Files accessible after rename
• ✓ No data loss

Test C: Delete Operations

1. Upload file: delete_test.txt
2. Delete it
3. Verify: File removed from list
4. Check logs: Should see [DELETE_FILE] SUCCESS

Pass Criteria:

• ✓ File removed
• ✓ Delete confirmation modal shown
• ✓ Log shows [DELETE_FILE] SUCCESS

Test D: Cache Behavior

Goal: Verify 60-second cache works

1. Upload file: cache_test.txt
2. Open DevTools (F12) → Network tab
3. Reload page within 30 seconds
4. Observe: List call should be instant (using cache)
5. Wait 65 seconds
6. Reload page
7. Observe: Slightly slower (fresh from server)

Pass Criteria:

• ✓ First reload is instant (<100ms)
• ✓ After 60s, fresh data fetched
• ✓ No errors in console

---

PHASE 3: Automated Stress Testing (60 minutes)

Step 1: Start Server

# Terminal 1: Start Flask backend
cd c:\Users\mohat\OneDrive\Desktop\Doc
python -m server.app
# Should see: Running on http://127.0.0.1:5000

Step 2: Run Stress Tests

# Terminal 2: Run stress test suite (LOCAL mode)
cd c:\Users\mohat\OneDrive\Desktop\Doc
python stress_test.py http://localhost:5000

Expected Output: ```

DOCVAULT STRESS TEST SUITE

Base URL: http://localhost:5000 User: stress_test_user Timeout: 30s

Checking server connectivity... ✓ Server is healthy

🔥 TEST 1: BULK UPLOAD (50 files)

✓ bulk_upload_all_succeed ✓ bulk_upload_speed Uploaded: 50/50 Total time: 25.34s Avg per file: 0.51s

🔥 TEST 2: FOLDER RENAME STRESS (30 files in folder)

✓ folder_rename ✓ folder_rename_speed ✓ folder_rename_data_preserved Renaming folder... Time: 2.15s

🔥 TEST 3: RAPID OPERATIONS (Race Conditions)

✓ rapid_ops_20_cycles ✓ rapid_seq_operations

🧪 TEST 4: CACHE VALIDATION (60s TTL)

✓ cache_first_list ✓ cache_second_list

🧪 TEST 5: ERROR HANDLING

✓ error_handling_delete_nonexistent ✓ error_handling_rename_nonexistent ✓ error_handling_list_nonexistent

============================================================ STRESS TEST RESULTS

✓ Passed: 16 ✗ Failed: 0 Total: 16

PERFORMANCE METRICS: bulk_upload: avg=0.51s, max=1.23s, count=50 folder_rename: avg=2.15s, max=2.15s, count=1 list_fresh: avg=0.05s, max=0.05s, count=1 list_cached: avg=0.02s, max=0.02s, count=1

============================================================

#### Step 3: Check Logs
```bash
# Review logs for any warnings or errors
tail -f logs/docvault.log | grep -E "\[UPLOAD\]|\[DELETE\]|\[RENAME\]|\[ERROR\]"

Look For:

• ✓ All operations have START and SUCCESS
• ✓ No EXCEPTION entries
• ⚠️ Any FAIL entries (investigate)

Step 4: Interpret Results

Perfect Scenario:

• All 16 tests pass
• No race conditions detected
• Performance metrics reasonable
• No errors in logs

Warning Scenarios:

• Bulk upload > 30 seconds: May be slow
• Folder rename > 5 seconds: Performance issue
• Failed rapid operations: Race condition risk
• Exceptions in logs: Critical issue

Failure Scenario:

• Tests fail: Do NOT deploy
• Race conditions detected: Do NOT deploy
• Exceptions: Do NOT deploy

---

📊 TEST RESULTS TEMPLATE

Record results here:

TEST EXECUTION REPORT
═════════════════════════════════════════════════════════

Date: ________________
Environment: [ ] LOCAL  [ ] HF_SPACES
Tester: ________________

MANUAL TESTS:
─────────────────────────────────────────────────────────
Test A - Rename Feature:         [ ] PASS  [ ] FAIL
Test B - Folder Operations:      [ ] PASS  [ ] FAIL
Test C - Delete Operations:      [ ] PASS  [ ] FAIL
Test D - Cache Behavior:         [ ] PASS  [ ] FAIL

AUTOMATED STRESS TESTS:
─────────────────────────────────────────────────────────
Total Tests: ____
Passed: ____
Failed: ____
Race Conditions: [ ] YES  [ ] NO

PERFORMANCE METRICS (from stress test):
─────────────────────────────────────────────────────────
Avg Upload Time: ______ seconds
Max Rename Time: ______ seconds
Cache Hit Time: ______ seconds
List Operation Time: ______ seconds

ISSUES FOUND:
─────────────────────────────────────────────────────────
[ ] None
[ ] Minor (document & monitor)
[ ] Major (fix before deploy)
[ ] Critical (DO NOT DEPLOY)

Issues:
1. ________________________________________
2. ________________________________________
3. ________________________________________

LOGS STATUS:
─────────────────────────────────────────────────────────
[ ] All operations logged correctly
[ ] No exceptions in logs
[ ] Errors are meaningful
[ ] Performance data captured

FINAL VERDICT:
─────────────────────────────────────────────────────────
[ ] READY FOR PRODUCTION
[ ] NEEDS MORE TESTING
[ ] DO NOT DEPLOY

Comments:
__________________________________________________________________
__________________________________________________________________

Signed: _________________________ Date: ________________

---

🔥 CRITICAL ISSUES TO WATCH FOR

Red Flag #1: Upload Failures

[UPLOAD_FILE] FAIL | reason=permission_denied

→ Check HF token permissions

Red Flag #2: Rename Failures

[RENAME] FAIL | error=atomic_commit_failed

→ Check HF API availability

Red Flag #3: Race Conditions

rapid_ops test: 3/20 operations failed

→ Investigate cache invalidation timing

Red Flag #4: Slow Performance

bulk_upload total time: 180s (should be <60s)

→ Check network/HF API performance

Red Flag #5: Memory Leaks

Processing time increases over 150+ operations

→ Check for unclosed file handles

---

🎯 SUCCESS CRITERIA FOR PRODUCTION DEPLOYMENT

All Must Be True:

• ✅ Manual tests: 4/4 passing
• ✅ Stress tests: 16/16 passing
• ✅ No race conditions detected
• ✅ No exceptions in logs
• ✅ Performance acceptable (bulk upload <60s)
• ✅ Cache working (list cached <100ms)
• ✅ All logs readable and structured
• ✅ Zero data loss observed

If Any Fails: → DO NOT DEPLOY
→ Investigate & fix
→ Re-test

---

🚀 DEPLOYMENT STEPS (After Passing Tests)

1. Backup Current HF Repo

   # Create a snapshot of current state
   # Document in deployment log
   

2. Deploy Code

   • Push fixes to HF Space git repo
   • Or: Upload files directly to Space

3. Restart Application

   • HF Spaces: App auto-restarts on code change
   • Local: Restart Flask server

4. Smoke Test

   • Upload test file
   • Rename it
   • Delete it
   • Check logs are generated

5. Monitor for 24 Hours

   • Check logs regularly
   • Monitor error rates
   • Gather user feedback

---

📝 LOGGING REFERENCE

Log Format

[OPERATION] STATUS | user=X | details=Y | duration=Z

Example Logs

[UPLOAD_FILE] START | user=default_user | file=report.pdf | size=2097152
[UPLOAD_FILE] SUCCESS | user=default_user | file=report.pdf
[DELETE_FILE] START | user=default_user | path=report.pdf
[DELETE_FILE] SUCCESS | user=default_user | path=report.pdf
[RENAME] START | user=default_user | path=old_name | new_name=new_name
[RENAME] SUCCESS | user=default_user | old_path=old_name | new_name=new_name
[LIST] START | user=default_user | path=root
[LIST] SUCCESS | user=default_user | path=root | files=42 | folders=5

What Each Field Means

• [OPERATION] - What's happening (UPLOAD, DELETE, RENAME, etc.)
• STATUS - Result (START, SUCCESS, FAIL, EXCEPTION)
• user=X - Which user (always check: should be isolated)
• details - Operation context (files, paths, sizes)
• duration - How long it took (helps spot performance issues)

---

📞 TROUBLESHOOTING

Stress Test Won't Run

# Error: No module named 'requests'
pip install requests

# Error: Connection refused
# Make sure Flask app is running: python -m server.app

Tests Fail With Timeouts

• Check: Server is responding normally
• Check: Network connectivity
• Increase: TIMEOUT variable in stress_test.py

Logs Not Appearing

• Check: Logs directory exists
• Check: config.py LOG_LEVEL = "INFO"
• Check: Flask app has logging enabled

---

NEXT STEP: Execute this entire procedure and record results in the template above.

THEN: If all passes → YOU'RE PRODUCTION READY ✅

---

Ready to test? Let's ensure this app is truly production-grade.