Spaces:

Nicolas53
/

auto-reporting

Sleeping

# User had to:
1. Create a JSON config file manually (vim config.json)
2. Run: python generate_wp_report.py config.json
3. Wait 5 minutes
4. Manually zip the template: zip -r report.zip wp_doc_template/
5. Upload to Overleaf

After (Goal)

1. Open http://localhost:5000 in browser
2. Fill form (30 seconds)
3. Click "Generate Report"
4. Download ZIP
5. Upload to Overleaf → Done!

2. What We Built

Complete Application Stack

✅ Backend (Python/Flask)

Flask web server with 4 routes
Input validation (server-side)
Report generation wrapper
Automatic file cleanup
Error handling (404, 500)

✅ Frontend (HTML/CSS/JavaScript)

Responsive form with Bootstrap 5
AJAX submission (no page reload)
Client-side validation
Progress indicators
ITU-branded design

✅ Documentation

Technical README
User guide (USAGE.md)
Test script
This status document

Statistics

14 files created (~1540 lines of code)
Backend: ~690 lines Python
Frontend: ~450 lines HTML/CSS/JS
Documentation: ~250 lines

3. Architecture

3.1 High-Level Flow

User Browser
    ↓
    ↓ Fill form → Click "Generate"
    ↓
Flask Server (app.py)
    ↓
    ├─→ Validate input (validators.py)
    ├─→ Create temp directory (file_handler.py)
    ├─→ Generate report (report_generator.py)
    │       ↓
    │       └─→ Call existing scripts-new/generate_wp_report.py
    │               ↓
    │               └─→ Scrape ITU website (2-5 min)
    ├─→ Create ZIP (file_handler.py)
    └─→ Return download link
    ↓
User downloads ZIP
    ↓
Upload to Overleaf → Compile → PDF ✅

3.2 Backend Architecture

# app.py (Flask Application)
@app.route('/')                           # Home page
@app.route('/api/generate-wp-report')     # Generate WP report
@app.route('/api/generate-question-report') # Generate Question report
@app.route('/api/download/<id>')          # Download ZIP

# utils/ (Backend Modules)
├── file_handler.py
│   ├── create_temp_directory()      # Create unique temp folder
│   ├── create_zip()                 # Compress template
│   ├── cleanup_old_files()          # Delete files > 1h
│   └── copy_template()              # Copy WP/Question template
│
├── report_generator.py
│   ├── generate_wp_report()         # Wrapper for WP generation
│   └── generate_question_report()   # Wrapper for Question generation
│
└── validators.py
    ├── validate_wp_config()         # Validate WP data
    ├── validate_question_config()   # Validate Question data
    ├── validate_dates()             # Check start < end
    └── sanitize_string()            # Basic XSS protection

3.3 Frontend Architecture

// static/js/app.js
document.addEventListener('DOMContentLoaded', function() {
    // 1. Toggle WP/Question fields
    // 2. Form submission via AJAX (fetch API)
    // 3. Show progress/success/error messages
    // 4. Client-side date validation
});

3.4 Data Flow

// Frontend sends:
{
    "group": 17,
    "workingParty": 1,
    "place": "Geneva",
    "start": "2026/06/03",
    "end": "2026/06/11",
    "sessions": 1,
    "meetingDays": ["2026/06/10"]
}

// Backend returns:
{
    "success": true,
    "download_id": "20260526_153045",
    "filename": "wp_report_20260526_153045.zip"
}

4. File Structure

sg17collab/
├── .gitignore                        # ✅ Created (root level)
│
└── auto-doc-latex/
    ├── scripts-new/                  # Existing (unchanged)
    │   ├── generate_wp_report.py
    │   ├── generate_question_report.py
    │   └── common/
    │
    ├── wp_doc_template/              # Existing (unchanged)
    ├── question_doc_template/        # Existing (unchanged)
    │
    └── web-app/                      # ✅ NEW - All created
        ├── app.py                    # ✅ Flask application (238 lines)
        ├── config.py                 # ✅ Configuration (102 lines)
        ├── requirements.txt          # ✅ Dependencies
        ├── README.md                 # ✅ Technical docs
        ├── USAGE.md                  # ✅ User guide
        ├── PROJECT_STATUS.md         # ✅ This file
        ├── test_setup.py             # ✅ Backend test script
        ├── test_config.json          # ✅ Test data
        │
        ├── utils/                    # ✅ Backend modules
        │   ├── __init__.py
        │   ├── file_handler.py       # ✅ (85 lines)
        │   ├── report_generator.py   # ✅ (95 lines)
        │   └── validators.py         # ✅ (186 lines)
        │
        ├── templates/                # ✅ HTML templates
        │   └── index.html            # ✅ Main form (165 lines)
        │
        ├── static/                   # ✅ Frontend assets
        │   ├── css/
        │   │   └── style.css         # ✅ (115 lines)
        │   └── js/
        │       └── app.js            # ✅ (185 lines)
        │
        └── temp/                     # ✅ Generated files
            ├── .gitkeep
            └── README.md

5. What's Working

✅ Fully Tested and Working

Component	Status	Test Method
Flask server startup	✅	`python app.py` → runs on port 5000
Route `/` (home page)	✅	`curl http://localhost:5000/` → 200 OK
Static files (CSS/JS)	✅	`curl http://localhost:5000/static/css/style.css` → 200
Input validation (dates)	✅	POST with start > end → 400 error returned
Input validation (fields)	✅	POST missing fields → 400 with clear errors
Error handler 404	✅	GET /invalid-route → 404 JSON
Error handler 500	✅	Errors return JSON with success: false
Package imports	✅	`from utils import *` works
`create_temp_directory()`	✅	Creates unique folders with timestamp
`validate_wp_config()`	✅	Detects invalid dates, missing fields
`validate_question_config()`	✅	Works correctly
`sanitize_string()`	✅	Removes XSS characters
Frontend HTML	✅	Renders correctly in browser
Frontend CSS	✅	Bootstrap + custom styles applied
Frontend JS loading	✅	app.js loads without errors

⚠️ Partially Tested

Component	Status	Notes
Report generation (full flow)	⏳	Backend API tested via curl, frontend integration in progress
`generate_wp_report()`	⏳	Function exists, not fully tested (requires ITU connection)
`generate_question_report()`	⏳	Function exists, not fully tested
ZIP creation	⏳	Function exists, not fully tested
Download endpoint	⏳	Route exists, not fully tested

6. Current Issue (Being Debugged)

Problem Description

User filled the form and clicked "Generate Report", but nothing happened (page refreshed instead of AJAX submission).

Root Cause Identified

User was accessing the page via VS Code Live Server (port 5500) instead of Flask (port 5000):

Port 5500: Jinja2 templates ({{ url_for() }}) are NOT compiled → CSS/JS fail to load → JavaScript doesn't run
Port 5000: Flask compiles templates → Everything works

Recent Fix Applied

File: static/js/app.js (line 72)

BEFORE:

const config = {
    documentType: 'report',  // ❌ Backend doesn't expect this
    group: parseInt(formData.get('group')),
    // ...
};

AFTER:

const config = {
    group: parseInt(formData.get('group')),  // ✅ Removed documentType
    // ...
};

Reason: Backend validators (validate_wp_config()) don't expect a documentType field. Including it caused validation errors.

Current Status

User confirmed: now on correct port (5000)
Only one error remaining: favicon.ico 404 (harmless)
Next step: User needs to test form submission again with hard refresh (Ctrl+F5)

7. What Remains to Do

7.1 Critical (Blocks Release)

Test full report generation flow
- Fill form → Submit → Wait 2-5 min → Download ZIP
- Verify ZIP contents match expected structure
- Test both WP and Question reports
Verify generated reports
- Compare output from web interface vs. command-line script
- Ensure all .tex files are identical
- Check that manual sections placeholders are present

7.2 Important (Should Do Before Merge)

Error handling improvements
- Better error messages if ITU website is down
- Timeout handling (currently 5 min, might need adjustment)
- Network error UX (show helpful message)
User experience
- Add favicon.ico to remove 404 error
- Test on mobile/tablet (responsive design should work but not tested)
- Add "Copy download link" button
Documentation
- Add screenshots to USAGE.md
- Create video/GIF demo
- Update main README.md with web interface section

7.3 Nice to Have (Future Enhancements)

Features
- CSV work programme upload
- Progress bar (detailed steps)
- Preview of scraped data before generation
- History of generated reports (session-based)
- Direct Overleaf API integration
Technical
- Add unit tests (pytest)
- Add integration tests
- CI/CD pipeline
- Docker container
- Production WSGI server (Gunicorn)

8. How to Test

8.1 Prerequisites

# From project root
cd auto-doc-latex/web-app

# Activate venv
../../../venv/Scripts/activate  # Windows
source ../../../venv/bin/activate  # Linux/Mac

# Install dependencies
pip install Flask Werkzeug beautifulsoup4 lxml python-dateutil

# Note: pycurl may fail on Windows, it's only needed for backend scripts

8.2 Start the Application

python app.py

Expected output:

* Serving Flask app 'app'
* Debug mode: on
* Running on http://127.0.0.1:5000

8.3 Access the Interface

Open browser: http://localhost:5000 (or http://127.0.0.1:5000)

⚠️ IMPORTANT: Do NOT open the HTML file directly via VS Code Live Server (port 5500)!

8.4 Test Working Party Report

Fill form with:

Report Type: Working Party Report
Study Group: 17
Working Party: 1
Place: Geneva
Start Date: 2026-06-03
End Date: 2026-06-11
Sessions: 1
Meeting Days: 2026/06/10

Click "Generate Report"

Expected behavior:

Button changes to "Generating..." with spinner
Blue alert box: "Generating report... This may take 2-5 minutes"
Wait (script scrapes ITU website)
Green success box with "Download LaTeX Files (ZIP)" button
Click download → ZIP file downloads

If error occurs:

Red alert box appears with error message
Check console (F12) for JavaScript errors
Check Flask terminal for backend errors

8.5 Test Question Report

Same as above but:

Report Type: Question Report
Question: 10 (instead of Working Party)
No Sessions or Meeting Days fields

8.6 Verify Generated Files

Unzip the downloaded file:

wp_doc_template/  (or question_doc_template/)
├── main.tex
├── chapters/
│   ├── *.tex
│   └── results/              # ← Auto-generated files should be here
│       ├── 00-variables.tex
│       ├── 01-introduction-content.tex
│       ├── 02-executive-summary-content.tex
│       └── ...
├── styles/
└── images/

Check:

results/ folder contains generated .tex files
00-variables.tex has correct values (place, dates, etc.)
Manual sections have \textit{For manual entry.} placeholders
No error messages in .tex files

8.7 Backend Tests (Without Browser)

# Run test script
python test_setup.py

# All tests should pass (except Flask dependency if not installed)

Test API directly with curl:

# Test validation (should return error)
curl -X POST http://127.0.0.1:5000/api/generate-wp-report \
  -H "Content-Type: application/json" \
  -d '{"group":17,"workingParty":1,"place":"","start":"2026/06/03","end":"2026/06/11"}'

# Expected: {"success": false, "error": "Place is required"}

9. Technical Details

9.1 Routes and Endpoints

Route	Method	Purpose	Request Body	Response
`/`	GET	Render home page	N/A	HTML
`/api/generate-wp-report`	POST	Generate WP report	JSON config	JSON {success, download_id}
`/api/generate-question-report`	POST	Generate Question report	JSON config	JSON {success, download_id}
`/api/download/<id>`	GET	Download ZIP	N/A	File (ZIP)
`/static/<path>`	GET	Serve static files	N/A	File (CSS/JS/images)

9.2 Configuration Values

File: config.py

# Arbitrary values (can be adjusted)
MAX_CONTENT_LENGTH = 16 MB        # Max upload size
TEMP_FILE_MAX_AGE = 1 hour       # Auto-cleanup age
GENERATION_TIMEOUT = 300 seconds  # 5 minutes

Paths (auto-detected):

BASE_DIR           # web-app/
SCRIPTS_DIR        # scripts-new/
WP_TEMPLATE_DIR    # wp_doc_template/
TEMP_DIR           # web-app/temp/

9.3 Dependencies

Runtime (Python 3.8+):

Flask 3.1.3
Werkzeug 3.1.8
beautifulsoup4 4.14.2
lxml 6.0.2
python-dateutil 2.9.0

Frontend:

Bootstrap 5.3.0 (CDN)
Vanilla JavaScript (no frameworks)

9.4 Key Design Decisions

No database: All data is ephemeral (deleted after 1h)
Session-only: No user accounts, no persistence
Single-threaded: One report generation at a time (Flask dev server)
Local-only: Binds to 127.0.0.1 (not exposed to internet)
No authentication: Assumes trusted local environment
Bootstrap CDN: No offline support, requires internet
Jinja2 templates: Flask renders HTML server-side
AJAX submission: No page reload, better UX

9.5 Security Considerations

Current protections:

Input validation (server + client)
Basic string sanitization (removes < > ; & | etc.)
No file uploads (except future CSV feature)
No database (no SQL injection risk)
No eval() or exec() calls

Known limitations:

No CSRF protection (not needed for local-only app)
No rate limiting
No authentication
Debug mode enabled (shows stack traces)

For production deployment:

Disable debug mode
Add authentication
Use WSGI server (Gunicorn)
Add HTTPS
Add CSRF tokens
Implement rate limiting

10. Important Notes for Developers

10.1 Common Pitfalls

❌ DON'T:

Open templates/index.html directly in browser
Use VS Code Live Server (port 5500)
Forget to activate venv before installing dependencies
Run generate_wp_report.py directly (use wrapper in utils/)
Commit files in temp/ folder (ignored by .gitignore)

✅ DO:

Always access via Flask (port 5000)
Hard refresh (Ctrl+F5) after JS changes
Check browser console (F12) for JS errors
Check Flask terminal for Python errors
Test both WP and Question reports

10.2 Debugging Tips

Frontend not working:

Check browser console (F12) for errors
Verify URL is http://localhost:5000 (not 5500)
Hard refresh: Ctrl+F5 (clears cache)
Check Network tab: look for failed requests (red)

Backend errors:

Check Flask terminal output
Look for stack traces
Test API with curl (isolate backend from frontend)
Check temp/ folder for generated files

Report generation fails:

Check internet connection (needs to reach www.itu.int)
Verify dates are valid (start < end)
Check Flask timeout (default 5 min)
Look at ITU website (might be down)

10.3 File Locations Reference

# If you need to find something:
Backend logic       → utils/*.py
Form HTML          → templates/index.html
Form submission JS → static/js/app.js (line ~48: form.addEventListener)
Validation         → utils/validators.py
Report generation  → utils/report_generator.py
Flask routes       → app.py (line ~32: @app.route)
Configuration      → config.py
CSS styles         → static/css/style.css

10.4 Git Status

Branch: feature/web-interface

Untracked files:

?? .gitignore                    # Root level
?? auto-doc-latex/web-app/       # Entire web-app folder

To commit:

git add .
git commit -m "feat: add web interface for ITU-T report generation

- Flask application with WP and Question report generation
- Modern frontend with Bootstrap 5
- Input validation (client and server side)
- Automatic file cleanup
- Complete documentation (README + USAGE + PROJECT_STATUS)
"

To merge (later):

git checkout main
git merge feature/web-interface
git push origin main

10.5 Next Session Checklist

When resuming work:

Verify Flask is NOT running: taskkill /F /IM python.exe
Activate venv
Navigate to auto-doc-latex/web-app
Start Flask: python app.py
Open browser: http://localhost:5000
Test report generation (2-5 min per test)
Verify downloaded ZIP structure
Upload to Overleaf and compile to PDF (final validation)

11. Questions to Resolve

Should we keep the documentType field?
- Currently: Removed from frontend, not used in validation
- Original scripts expect it in JSON
- Decision needed: Remove completely or add to validation?
Timeout value:
- Current: 5 minutes
- Is this enough for all report types?
- Should we make it configurable per report type?
Error messages:
- Current: Show raw Python exceptions in dev mode
- Should we sanitize error messages for users?
- Keep technical details in logs only?
CSV upload feature:
- Planned but not implemented
- Is it needed for MVP?
- How to handle large CSVs?

12. Contact & Resources

Documentation:

Technical: auto-doc-latex/web-app/README.md
User guide: auto-doc-latex/web-app/USAGE.md
This document: auto-doc-latex/web-app/PROJECT_STATUS.md

Test:

Backend: python test_setup.py
Frontend: Open http://localhost:5000 and test manually

Logs:

Flask: Terminal output or flask_test.log
Browser: F12 → Console tab

End of Document
Last Updated: 2026-05-26 22:30
Status: Ready for final testing and validation

Appendix: Quick Reference Commands

# Start application
cd auto-doc-latex/web-app
python app.py

# Test backend
python test_setup.py

# Stop Flask
# Windows: taskkill /F /IM python.exe
# Linux/Mac: pkill -f "python app.py"

# View logs
tail -f flask_test.log

# Test API
curl -X POST http://127.0.0.1:5000/api/generate-wp-report \
  -H "Content-Type: application/json" \
  -d '{"group":17,"workingParty":1,"place":"Geneva","start":"2026/06/03","end":"2026/06/11","sessions":1}'

# Check Flask status
curl -s http://127.0.0.1:5000/ | grep "ITU-T"