Spaces:
Sleeping
Sleeping
Codex: upload Offline Form Pilot Space
Browse files- .gitignore +14 -0
- CODEX_BUILD_LOG.md +33 -0
- DEPLOYMENT.md +65 -0
- FIELD_NOTES.md +27 -0
- README.md +45 -5
- app.py +236 -0
- examples/anonymized_trace.jsonl +1 -0
- formpilot/__init__.py +5 -0
- formpilot/engine.py +328 -0
- formpilot/model_assist.py +73 -0
- requirements.txt +5 -0
- tests/test_engine.py +32 -0
.gitignore
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
.venv/
|
| 2 |
+
__pycache__/
|
| 3 |
+
*.pyc
|
| 4 |
+
.pytest_cache/
|
| 5 |
+
.ruff_cache/
|
| 6 |
+
|
| 7 |
+
traces/*.json
|
| 8 |
+
traces/*.jsonl
|
| 9 |
+
exports/*.json
|
| 10 |
+
exports/*.csv
|
| 11 |
+
*.log
|
| 12 |
+
|
| 13 |
+
.env
|
| 14 |
+
.DS_Store
|
CODEX_BUILD_LOG.md
ADDED
|
@@ -0,0 +1,33 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Codex Build Log
|
| 2 |
+
|
| 3 |
+
This project is being built with OpenAI Codex as the coding agent for the Build Small Hackathon OpenAI/Codex track.
|
| 4 |
+
|
| 5 |
+
## Build Principles
|
| 6 |
+
|
| 7 |
+
- Keep commits small and Codex-attributed.
|
| 8 |
+
- Keep the app review-first: never auto-submit forms.
|
| 9 |
+
- Make model behavior inspectable through structured outputs and traces.
|
| 10 |
+
- Prefer a small-model-friendly workflow over a generic chatbot.
|
| 11 |
+
|
| 12 |
+
## Timeline
|
| 13 |
+
|
| 14 |
+
### 2026-06-06
|
| 15 |
+
|
| 16 |
+
- User selected the form-filling concept.
|
| 17 |
+
- Codex initialized a fresh standalone project under `projects/offline-form-pilot`.
|
| 18 |
+
- Codex added the initial README, build log, requirements, and git hygiene files.
|
| 19 |
+
- Codex implemented the first Gradio app, local structured matcher, optional Hugging Face small-model assist, trace export, and engine tests.
|
| 20 |
+
- Codex ran `pytest` in the project venv and a direct app smoke check.
|
| 21 |
+
- Codex created and pushed the public GitHub repo: https://github.com/rishabhsai/offline-form-pilot
|
| 22 |
+
- Codex confirmed the repo is public and added Hugging Face deployment notes plus a synthetic anonymized trace example.
|
| 23 |
+
- Codex attempted to create the official `build-small-hackathon/offline-form-pilot` Space, but HF returned a 403 for missing namespace creation rights.
|
| 24 |
+
- Codex created the fallback public personal Space: https://huggingface.co/spaces/rishabhsai/offline-form-pilot
|
| 25 |
+
|
| 26 |
+
## Planned Codex Milestones
|
| 27 |
+
|
| 28 |
+
1. Scaffold repo and README.
|
| 29 |
+
2. Implement Gradio app and deterministic field matcher.
|
| 30 |
+
3. Add optional small-model backends and trace export.
|
| 31 |
+
4. Add tests and local verification.
|
| 32 |
+
5. Push public GitHub repo with Codex-attributed commits.
|
| 33 |
+
6. Deploy Hugging Face Space after HF auth is available.
|
DEPLOYMENT.md
ADDED
|
@@ -0,0 +1,65 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Deployment
|
| 2 |
+
|
| 3 |
+
## GitHub
|
| 4 |
+
|
| 5 |
+
Public repo:
|
| 6 |
+
|
| 7 |
+
https://github.com/rishabhsai/offline-form-pilot
|
| 8 |
+
|
| 9 |
+
The commit history uses Codex-attributed commit messages for the OpenAI/Codex track.
|
| 10 |
+
|
| 11 |
+
## Hugging Face Space
|
| 12 |
+
|
| 13 |
+
The local `hf` CLI is installed in the project venv. Authenticate first:
|
| 14 |
+
|
| 15 |
+
```bash
|
| 16 |
+
source .venv/bin/activate
|
| 17 |
+
hf auth login
|
| 18 |
+
```
|
| 19 |
+
|
| 20 |
+
If late org access is approved:
|
| 21 |
+
|
| 22 |
+
```bash
|
| 23 |
+
hf repos create build-small-hackathon/offline-form-pilot --type space --space-sdk gradio --exist-ok
|
| 24 |
+
hf upload build-small-hackathon/offline-form-pilot . --type space \
|
| 25 |
+
--exclude ".venv/*" \
|
| 26 |
+
--exclude "traces/*" \
|
| 27 |
+
--exclude "exports/*" \
|
| 28 |
+
--commit-message "Codex: upload Offline Form Pilot Space"
|
| 29 |
+
```
|
| 30 |
+
|
| 31 |
+
If org access is not available, use a personal Space:
|
| 32 |
+
|
| 33 |
+
```bash
|
| 34 |
+
hf repos create rishabhsai/offline-form-pilot --type space --space-sdk gradio --exist-ok
|
| 35 |
+
hf upload rishabhsai/offline-form-pilot . --type space \
|
| 36 |
+
--exclude ".venv/*" \
|
| 37 |
+
--exclude "traces/*" \
|
| 38 |
+
--exclude "exports/*" \
|
| 39 |
+
--commit-message "Codex: upload Offline Form Pilot Space"
|
| 40 |
+
```
|
| 41 |
+
|
| 42 |
+
Current personal Space:
|
| 43 |
+
|
| 44 |
+
https://huggingface.co/spaces/rishabhsai/offline-form-pilot
|
| 45 |
+
|
| 46 |
+
After deployment, update `README.md` with the Space URL and push one final Codex-attributed commit to GitHub.
|
| 47 |
+
|
| 48 |
+
## Space Secrets
|
| 49 |
+
|
| 50 |
+
For the optional small-model assist mode, add one of these secrets in the Space settings:
|
| 51 |
+
|
| 52 |
+
- `HF_TOKEN`
|
| 53 |
+
- `HUGGINGFACEHUB_API_TOKEN`
|
| 54 |
+
|
| 55 |
+
The app still runs without a token using the local structured matcher.
|
| 56 |
+
|
| 57 |
+
## Demo Checklist
|
| 58 |
+
|
| 59 |
+
1. Paste a messy form.
|
| 60 |
+
2. Paste safe sample facts.
|
| 61 |
+
3. Run local structured matcher.
|
| 62 |
+
4. Switch to small-model assist if the Space token is configured.
|
| 63 |
+
5. Show ready/review/missing fields.
|
| 64 |
+
6. Show generated questions before copying.
|
| 65 |
+
7. Download trace JSON and field CSV.
|
FIELD_NOTES.md
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Field Notes
|
| 2 |
+
|
| 3 |
+
## Question
|
| 4 |
+
|
| 5 |
+
Can a small model make form filling safer and less frustrating by turning messy paperwork into a reviewable field table instead of directly controlling a browser or submitting a form?
|
| 6 |
+
|
| 7 |
+
## Intended User
|
| 8 |
+
|
| 9 |
+
One real person who regularly deals with confusing forms or paperwork. The project should name the workflow in the demo without exposing private data.
|
| 10 |
+
|
| 11 |
+
## What The App Does
|
| 12 |
+
|
| 13 |
+
- Finds likely fields in pasted form text.
|
| 14 |
+
- Matches user-provided facts to those fields.
|
| 15 |
+
- Separates confident fills from missing or risky fields.
|
| 16 |
+
- Generates plain-English questions for the missing fields.
|
| 17 |
+
- Exports an anonymized trace for review.
|
| 18 |
+
|
| 19 |
+
## What The App Does Not Do
|
| 20 |
+
|
| 21 |
+
- It does not submit forms.
|
| 22 |
+
- It does not bypass user review.
|
| 23 |
+
- It does not claim legal, medical, financial, or immigration advice.
|
| 24 |
+
|
| 25 |
+
## Early Hypothesis
|
| 26 |
+
|
| 27 |
+
The useful small-model behavior is not "agent fills the website." The useful behavior is "agent prepares a clean checklist the person can verify quickly."
|
README.md
CHANGED
|
@@ -1,13 +1,53 @@
|
|
| 1 |
---
|
| 2 |
title: Offline Form Pilot
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
colorTo: green
|
| 6 |
sdk: gradio
|
| 7 |
sdk_version: 6.16.0
|
| 8 |
-
python_version: '3.13'
|
| 9 |
app_file: app.py
|
| 10 |
pinned: false
|
|
|
|
|
|
|
| 11 |
---
|
| 12 |
|
| 13 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
title: Offline Form Pilot
|
| 3 |
+
colorFrom: green
|
| 4 |
+
colorTo: blue
|
|
|
|
| 5 |
sdk: gradio
|
| 6 |
sdk_version: 6.16.0
|
|
|
|
| 7 |
app_file: app.py
|
| 8 |
pinned: false
|
| 9 |
+
license: mit
|
| 10 |
+
short_description: Review confusing forms before copying fields.
|
| 11 |
---
|
| 12 |
|
| 13 |
+
# Offline Form Pilot
|
| 14 |
+
|
| 15 |
+
Offline Form Pilot helps a real person fill confusing forms without handing control to an autopilot. Paste a form, paste the facts you are comfortable using, and the app produces a review table with proposed values, confidence, missing fields, and questions to ask before anything is copied.
|
| 16 |
+
|
| 17 |
+
The app is designed for the Build Small Hackathon:
|
| 18 |
+
|
| 19 |
+
- **Track:** Backyard AI.
|
| 20 |
+
- **Small model constraint:** default target is a <=4B small model.
|
| 21 |
+
- **Safety posture:** no automatic submission, no hidden form filling, no legal/financial certainty.
|
| 22 |
+
- **Codex track:** this public repo was built with Codex-attributed commits and includes `CODEX_BUILD_LOG.md`.
|
| 23 |
+
|
| 24 |
+
## Why Small Models Fit
|
| 25 |
+
|
| 26 |
+
Form help is mostly narrow extraction, matching, and clarification. A small model can do useful work when the interface forces structured fields, review, and missing-information checks.
|
| 27 |
+
|
| 28 |
+
## Local Run
|
| 29 |
+
|
| 30 |
+
```bash
|
| 31 |
+
python3 -m venv .venv
|
| 32 |
+
source .venv/bin/activate
|
| 33 |
+
pip install -r requirements.txt
|
| 34 |
+
python app.py
|
| 35 |
+
```
|
| 36 |
+
|
| 37 |
+
## Deployment
|
| 38 |
+
|
| 39 |
+
See `DEPLOYMENT.md` for the GitHub and Hugging Face Space commands.
|
| 40 |
+
|
| 41 |
+
## Submission Checklist
|
| 42 |
+
|
| 43 |
+
- Public GitHub repo linked from this README.
|
| 44 |
+
- Hugging Face Space link added after deployment.
|
| 45 |
+
- Short demo video.
|
| 46 |
+
- Social post.
|
| 47 |
+
- Field notes.
|
| 48 |
+
- Optional trace dataset with anonymized examples.
|
| 49 |
+
|
| 50 |
+
## Repository Links
|
| 51 |
+
|
| 52 |
+
- GitHub: https://github.com/rishabhsai/offline-form-pilot
|
| 53 |
+
- Hugging Face Space: https://huggingface.co/spaces/rishabhsai/offline-form-pilot
|
app.py
ADDED
|
@@ -0,0 +1,236 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import annotations
|
| 2 |
+
|
| 3 |
+
import json
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
from typing import Any
|
| 6 |
+
|
| 7 |
+
import gradio as gr
|
| 8 |
+
import pandas as pd
|
| 9 |
+
|
| 10 |
+
from formpilot.engine import analyze_form, export_trace, rows_to_csv
|
| 11 |
+
from formpilot.model_assist import DEFAULT_SMALL_MODEL, try_hf_model_assist
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
APP_DIR = Path(__file__).resolve().parent
|
| 15 |
+
TRACE_DIR = APP_DIR / "traces"
|
| 16 |
+
EXPORT_DIR = APP_DIR / "exports"
|
| 17 |
+
TRACE_DIR.mkdir(exist_ok=True)
|
| 18 |
+
EXPORT_DIR.mkdir(exist_ok=True)
|
| 19 |
+
|
| 20 |
+
SAMPLE_FORM = """Community Center Membership Form
|
| 21 |
+
|
| 22 |
+
Full name: ____________________
|
| 23 |
+
Email: ____________________
|
| 24 |
+
Phone: ____________________
|
| 25 |
+
Address: ____________________
|
| 26 |
+
Emergency contact: ____________________
|
| 27 |
+
Relationship: ____________________
|
| 28 |
+
Signature: ____________________
|
| 29 |
+
Date: ____________________
|
| 30 |
+
"""
|
| 31 |
+
|
| 32 |
+
SAMPLE_FACTS = """Full name: Jordan Lee
|
| 33 |
+
Email: jordan.lee@example.com
|
| 34 |
+
Phone: 555-0137
|
| 35 |
+
Address: 42 Maple Street, Springfield, NY 10027
|
| 36 |
+
Emergency contact: Priya Lee
|
| 37 |
+
Relationship: Sister
|
| 38 |
+
Date: June 6, 2026
|
| 39 |
+
"""
|
| 40 |
+
|
| 41 |
+
CSS = """
|
| 42 |
+
.gradio-container {
|
| 43 |
+
max-width: 1240px !important;
|
| 44 |
+
}
|
| 45 |
+
#hero {
|
| 46 |
+
padding: 8px 0 16px 0;
|
| 47 |
+
border-bottom: 1px solid #d7dde5;
|
| 48 |
+
}
|
| 49 |
+
#hero h1 {
|
| 50 |
+
font-size: 34px;
|
| 51 |
+
line-height: 1.05;
|
| 52 |
+
letter-spacing: 0;
|
| 53 |
+
margin: 0 0 8px 0;
|
| 54 |
+
}
|
| 55 |
+
#hero p {
|
| 56 |
+
color: #5b6470;
|
| 57 |
+
max-width: 860px;
|
| 58 |
+
font-size: 15px;
|
| 59 |
+
}
|
| 60 |
+
.status-ready {
|
| 61 |
+
color: #1f6f4a;
|
| 62 |
+
font-weight: 700;
|
| 63 |
+
}
|
| 64 |
+
.status-review {
|
| 65 |
+
color: #9a6500;
|
| 66 |
+
font-weight: 700;
|
| 67 |
+
}
|
| 68 |
+
.status-missing {
|
| 69 |
+
color: #a33d3d;
|
| 70 |
+
font-weight: 700;
|
| 71 |
+
}
|
| 72 |
+
.panel-note {
|
| 73 |
+
border: 1px solid #d7dde5;
|
| 74 |
+
border-radius: 8px;
|
| 75 |
+
background: #ffffff;
|
| 76 |
+
padding: 12px 14px;
|
| 77 |
+
}
|
| 78 |
+
"""
|
| 79 |
+
|
| 80 |
+
|
| 81 |
+
def _status_html(rows: list[dict[str, Any]], risks: list[str]) -> str:
|
| 82 |
+
ready = sum(1 for row in rows if row["status"] == "ready")
|
| 83 |
+
review = sum(1 for row in rows if row["status"] == "review")
|
| 84 |
+
missing = sum(1 for row in rows if row["status"] == "missing")
|
| 85 |
+
risk_items = "".join(f"<li>{risk}</li>" for risk in risks)
|
| 86 |
+
return f"""
|
| 87 |
+
<div class="panel-note">
|
| 88 |
+
<p><span class="status-ready">{ready} ready</span> · <span class="status-review">{review} review</span> · <span class="status-missing">{missing} missing</span></p>
|
| 89 |
+
<ul>{risk_items}</ul>
|
| 90 |
+
</div>
|
| 91 |
+
"""
|
| 92 |
+
|
| 93 |
+
|
| 94 |
+
def _write_export_files(payload: dict[str, Any]) -> tuple[str, str]:
|
| 95 |
+
trace_path = export_trace(payload, TRACE_DIR)
|
| 96 |
+
csv_path = EXPORT_DIR / "formpilot_latest_fields.csv"
|
| 97 |
+
csv_path.write_text(rows_to_csv(payload["rows"]), encoding="utf-8")
|
| 98 |
+
return trace_path, str(csv_path)
|
| 99 |
+
|
| 100 |
+
|
| 101 |
+
def _model_payload_to_rows(model_payload: dict[str, Any]) -> list[dict[str, Any]]:
|
| 102 |
+
rows = model_payload.get("fields", [])
|
| 103 |
+
normalized = []
|
| 104 |
+
for row in rows:
|
| 105 |
+
normalized.append(
|
| 106 |
+
{
|
| 107 |
+
"field": str(row.get("field", "")),
|
| 108 |
+
"proposed_value": str(row.get("proposed_value", "")),
|
| 109 |
+
"status": str(row.get("status", "review")),
|
| 110 |
+
"confidence": int(row.get("confidence", 0) or 0),
|
| 111 |
+
"source": str(row.get("source", "")),
|
| 112 |
+
"note": str(row.get("note", "")),
|
| 113 |
+
}
|
| 114 |
+
)
|
| 115 |
+
return normalized
|
| 116 |
+
|
| 117 |
+
|
| 118 |
+
def run_pilot(
|
| 119 |
+
form_text: str,
|
| 120 |
+
user_facts: str,
|
| 121 |
+
mode: str,
|
| 122 |
+
model_id: str,
|
| 123 |
+
) -> tuple[pd.DataFrame, str, str, str, dict[str, Any], str, str]:
|
| 124 |
+
if not form_text.strip():
|
| 125 |
+
raise gr.Error("Paste a form or request first.")
|
| 126 |
+
|
| 127 |
+
payload = analyze_form(form_text, user_facts)
|
| 128 |
+
backend_note = "Local structured matcher"
|
| 129 |
+
|
| 130 |
+
if mode == "Small model assist":
|
| 131 |
+
try:
|
| 132 |
+
model_payload = try_hf_model_assist(form_text, user_facts, model_id.strip() or DEFAULT_SMALL_MODEL)
|
| 133 |
+
model_rows = _model_payload_to_rows(model_payload)
|
| 134 |
+
if model_rows:
|
| 135 |
+
payload["rows"] = model_rows
|
| 136 |
+
payload["questions"] = model_payload.get("questions", payload["questions"])
|
| 137 |
+
payload["risk_summary"] = model_payload.get("risk_summary", payload["risk_summary"])
|
| 138 |
+
payload["copy_ready"] = "\n".join(
|
| 139 |
+
f"{row['field']}: {row['proposed_value'] or '[NEEDS USER INPUT]'}"
|
| 140 |
+
for row in model_rows
|
| 141 |
+
)
|
| 142 |
+
backend_note = f"Small model assist: {model_id.strip() or DEFAULT_SMALL_MODEL}"
|
| 143 |
+
except Exception as exc:
|
| 144 |
+
payload["risk_summary"].insert(
|
| 145 |
+
0,
|
| 146 |
+
f"Small model assist failed; used local matcher instead. Reason: {exc}",
|
| 147 |
+
)
|
| 148 |
+
|
| 149 |
+
payload["backend"] = backend_note
|
| 150 |
+
trace_path, csv_path = _write_export_files(payload)
|
| 151 |
+
|
| 152 |
+
rows = payload["rows"]
|
| 153 |
+
table = pd.DataFrame(rows, columns=["field", "proposed_value", "status", "confidence", "source", "note"])
|
| 154 |
+
questions = "\n".join(f"- {question}" for question in payload["questions"]) or "No missing-field questions detected."
|
| 155 |
+
summary = _status_html(rows, payload["risk_summary"])
|
| 156 |
+
return table, payload["copy_ready"], questions, summary, payload, trace_path, csv_path
|
| 157 |
+
|
| 158 |
+
|
| 159 |
+
def clear_outputs() -> tuple[pd.DataFrame, str, str, str, dict[str, Any], None, None]:
|
| 160 |
+
return pd.DataFrame(), "", "", "", {}, None, None
|
| 161 |
+
|
| 162 |
+
|
| 163 |
+
def build_demo() -> gr.Blocks:
|
| 164 |
+
with gr.Blocks(title="Offline Form Pilot") as demo:
|
| 165 |
+
gr.Markdown(
|
| 166 |
+
"""
|
| 167 |
+
# Offline Form Pilot
|
| 168 |
+
Paste a form and the facts you are willing to use. The app prepares a review table, missing-field questions, and copy-ready text without submitting anything.
|
| 169 |
+
""",
|
| 170 |
+
elem_id="hero",
|
| 171 |
+
)
|
| 172 |
+
|
| 173 |
+
with gr.Row():
|
| 174 |
+
with gr.Column(scale=5):
|
| 175 |
+
form_text = gr.Textbox(
|
| 176 |
+
label="Form or request text",
|
| 177 |
+
value=SAMPLE_FORM,
|
| 178 |
+
lines=12,
|
| 179 |
+
max_lines=18,
|
| 180 |
+
)
|
| 181 |
+
user_facts = gr.Textbox(
|
| 182 |
+
label="User facts",
|
| 183 |
+
value=SAMPLE_FACTS,
|
| 184 |
+
lines=10,
|
| 185 |
+
max_lines=16,
|
| 186 |
+
)
|
| 187 |
+
with gr.Column(scale=3):
|
| 188 |
+
mode = gr.Radio(
|
| 189 |
+
label="Analysis mode",
|
| 190 |
+
choices=["Local structured matcher", "Small model assist"],
|
| 191 |
+
value="Local structured matcher",
|
| 192 |
+
)
|
| 193 |
+
model_id = gr.Textbox(label="Small model id", value=DEFAULT_SMALL_MODEL)
|
| 194 |
+
run_btn = gr.Button("Prepare form review", variant="primary")
|
| 195 |
+
clear_btn = gr.Button("Clear outputs")
|
| 196 |
+
gr.Markdown(
|
| 197 |
+
"""
|
| 198 |
+
Human review is required. Do not paste secrets unless you are comfortable with the selected backend.
|
| 199 |
+
""",
|
| 200 |
+
elem_classes=["panel-note"],
|
| 201 |
+
)
|
| 202 |
+
|
| 203 |
+
summary = gr.HTML()
|
| 204 |
+
table = gr.Dataframe(
|
| 205 |
+
label="Review table",
|
| 206 |
+
headers=["field", "proposed_value", "status", "confidence", "source", "note"],
|
| 207 |
+
wrap=True,
|
| 208 |
+
interactive=False,
|
| 209 |
+
)
|
| 210 |
+
with gr.Row():
|
| 211 |
+
copy_ready = gr.Textbox(label="Copy-ready draft", lines=10, buttons=["copy"])
|
| 212 |
+
questions = gr.Textbox(label="Questions before copying", lines=10, buttons=["copy"])
|
| 213 |
+
with gr.Accordion("Trace and exports", open=False):
|
| 214 |
+
raw_json = gr.JSON(label="Trace JSON")
|
| 215 |
+
trace_file = gr.File(label="Download trace JSON")
|
| 216 |
+
csv_file = gr.File(label="Download field CSV")
|
| 217 |
+
|
| 218 |
+
run_btn.click(
|
| 219 |
+
run_pilot,
|
| 220 |
+
inputs=[form_text, user_facts, mode, model_id],
|
| 221 |
+
outputs=[table, copy_ready, questions, summary, raw_json, trace_file, csv_file],
|
| 222 |
+
api_name="prepare_form_review",
|
| 223 |
+
)
|
| 224 |
+
clear_btn.click(
|
| 225 |
+
clear_outputs,
|
| 226 |
+
outputs=[table, copy_ready, questions, summary, raw_json, trace_file, csv_file],
|
| 227 |
+
)
|
| 228 |
+
|
| 229 |
+
return demo
|
| 230 |
+
|
| 231 |
+
|
| 232 |
+
demo = build_demo()
|
| 233 |
+
|
| 234 |
+
|
| 235 |
+
if __name__ == "__main__":
|
| 236 |
+
demo.launch(css=CSS)
|
examples/anonymized_trace.jsonl
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"case_id":"community_center_demo","input_type":"pasted_text","form_fields":["Full name","Email","Phone","Address","Emergency contact","Relationship","Signature","Date"],"available_facts":["full name","email","phone","address","emergency contact","relationship","date"],"outcome":{"ready":7,"review":0,"missing":1},"missing_fields":["Signature"],"note":"Synthetic trace for demo and Sharing is Caring badge planning."}
|
formpilot/__init__.py
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Offline Form Pilot package."""
|
| 2 |
+
|
| 3 |
+
from formpilot.engine import analyze_form
|
| 4 |
+
|
| 5 |
+
__all__ = ["analyze_form"]
|
formpilot/engine.py
ADDED
|
@@ -0,0 +1,328 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Structured form analysis for Offline Form Pilot."""
|
| 2 |
+
|
| 3 |
+
from __future__ import annotations
|
| 4 |
+
|
| 5 |
+
import csv
|
| 6 |
+
import io
|
| 7 |
+
import json
|
| 8 |
+
import re
|
| 9 |
+
from dataclasses import asdict, dataclass
|
| 10 |
+
from datetime import datetime, timezone
|
| 11 |
+
from difflib import SequenceMatcher
|
| 12 |
+
from pathlib import Path
|
| 13 |
+
from typing import Any
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
COMMON_FIELDS = [
|
| 17 |
+
"full name",
|
| 18 |
+
"first name",
|
| 19 |
+
"last name",
|
| 20 |
+
"date of birth",
|
| 21 |
+
"email",
|
| 22 |
+
"phone",
|
| 23 |
+
"address",
|
| 24 |
+
"city",
|
| 25 |
+
"state",
|
| 26 |
+
"postal code",
|
| 27 |
+
"zip code",
|
| 28 |
+
"country",
|
| 29 |
+
"employer",
|
| 30 |
+
"school",
|
| 31 |
+
"student id",
|
| 32 |
+
"account number",
|
| 33 |
+
"policy number",
|
| 34 |
+
"emergency contact",
|
| 35 |
+
"relationship",
|
| 36 |
+
"signature",
|
| 37 |
+
"date",
|
| 38 |
+
]
|
| 39 |
+
|
| 40 |
+
SENSITIVE_TERMS = {
|
| 41 |
+
"ssn",
|
| 42 |
+
"social security",
|
| 43 |
+
"passport",
|
| 44 |
+
"bank",
|
| 45 |
+
"routing",
|
| 46 |
+
"account",
|
| 47 |
+
"card",
|
| 48 |
+
"credit",
|
| 49 |
+
"medical",
|
| 50 |
+
"diagnosis",
|
| 51 |
+
"tax",
|
| 52 |
+
"visa",
|
| 53 |
+
"immigration",
|
| 54 |
+
}
|
| 55 |
+
|
| 56 |
+
STOPWORDS = {
|
| 57 |
+
"the",
|
| 58 |
+
"a",
|
| 59 |
+
"an",
|
| 60 |
+
"your",
|
| 61 |
+
"you",
|
| 62 |
+
"of",
|
| 63 |
+
"for",
|
| 64 |
+
"and",
|
| 65 |
+
"or",
|
| 66 |
+
"to",
|
| 67 |
+
"in",
|
| 68 |
+
"on",
|
| 69 |
+
"with",
|
| 70 |
+
"please",
|
| 71 |
+
"enter",
|
| 72 |
+
"provide",
|
| 73 |
+
}
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
@dataclass(frozen=True)
|
| 77 |
+
class FieldMatch:
|
| 78 |
+
field: str
|
| 79 |
+
proposed_value: str
|
| 80 |
+
status: str
|
| 81 |
+
confidence: int
|
| 82 |
+
source: str
|
| 83 |
+
note: str
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
def normalize_label(text: str) -> str:
|
| 87 |
+
"""Normalize labels for fuzzy matching."""
|
| 88 |
+
|
| 89 |
+
text = text.lower()
|
| 90 |
+
text = re.sub(r"[^a-z0-9]+", " ", text)
|
| 91 |
+
text = re.sub(r"\s+", " ", text).strip()
|
| 92 |
+
return text
|
| 93 |
+
|
| 94 |
+
|
| 95 |
+
def _tokens(text: str) -> set[str]:
|
| 96 |
+
return {token for token in normalize_label(text).split() if token not in STOPWORDS}
|
| 97 |
+
|
| 98 |
+
|
| 99 |
+
def parse_user_facts(raw_facts: str) -> dict[str, str]:
|
| 100 |
+
"""Parse key-value facts from pasted user notes."""
|
| 101 |
+
|
| 102 |
+
facts: dict[str, str] = {}
|
| 103 |
+
free_lines: list[str] = []
|
| 104 |
+
for line in raw_facts.splitlines():
|
| 105 |
+
cleaned = line.strip().strip("-*")
|
| 106 |
+
if not cleaned:
|
| 107 |
+
continue
|
| 108 |
+
match = re.match(r"^([^:=]{2,60})\s*[:=]\s*(.+)$", cleaned)
|
| 109 |
+
if match:
|
| 110 |
+
key = normalize_label(match.group(1))
|
| 111 |
+
value = match.group(2).strip()
|
| 112 |
+
facts[key] = value
|
| 113 |
+
else:
|
| 114 |
+
free_lines.append(cleaned)
|
| 115 |
+
|
| 116 |
+
inferred = infer_facts_from_free_text("\n".join(free_lines))
|
| 117 |
+
for key, value in inferred.items():
|
| 118 |
+
facts.setdefault(key, value)
|
| 119 |
+
return facts
|
| 120 |
+
|
| 121 |
+
|
| 122 |
+
def infer_facts_from_free_text(text: str) -> dict[str, str]:
|
| 123 |
+
"""Extract a small set of common facts from unstructured text."""
|
| 124 |
+
|
| 125 |
+
facts: dict[str, str] = {}
|
| 126 |
+
email = re.search(r"[\w.+-]+@[\w.-]+\.[a-zA-Z]{2,}", text)
|
| 127 |
+
if email:
|
| 128 |
+
facts["email"] = email.group(0)
|
| 129 |
+
|
| 130 |
+
phone = re.search(r"(?:\+?\d[\d .()-]{7,}\d)", text)
|
| 131 |
+
if phone:
|
| 132 |
+
facts["phone"] = phone.group(0).strip()
|
| 133 |
+
|
| 134 |
+
zip_code = re.search(r"\b\d{5}(?:-\d{4})?\b", text)
|
| 135 |
+
if zip_code:
|
| 136 |
+
facts["zip code"] = zip_code.group(0)
|
| 137 |
+
facts["postal code"] = zip_code.group(0)
|
| 138 |
+
|
| 139 |
+
dob = re.search(r"\b(?:dob|date of birth)\s*[:=]?\s*([A-Za-z0-9, /.-]{6,20})", text, re.I)
|
| 140 |
+
if dob:
|
| 141 |
+
facts["date of birth"] = dob.group(1).strip()
|
| 142 |
+
|
| 143 |
+
return facts
|
| 144 |
+
|
| 145 |
+
|
| 146 |
+
def detect_fields(form_text: str) -> list[str]:
|
| 147 |
+
"""Find likely form fields from pasted form text."""
|
| 148 |
+
|
| 149 |
+
candidates: list[str] = []
|
| 150 |
+
|
| 151 |
+
for line in form_text.splitlines():
|
| 152 |
+
cleaned = line.strip()
|
| 153 |
+
if not cleaned:
|
| 154 |
+
continue
|
| 155 |
+
cleaned = re.sub(r"\s+", " ", cleaned)
|
| 156 |
+
|
| 157 |
+
label_match = re.match(r"^([A-Za-z][A-Za-z0-9 /'().,-]{1,70})\s*[:_]{1,}\s*(?:\[\s*\])?\s*$", cleaned)
|
| 158 |
+
if label_match:
|
| 159 |
+
candidates.append(label_match.group(1))
|
| 160 |
+
continue
|
| 161 |
+
|
| 162 |
+
bracket_match = re.match(r"^([A-Za-z][A-Za-z0-9 /'().,-]{1,70})\s*\[\s*\]\s*$", cleaned)
|
| 163 |
+
if bracket_match:
|
| 164 |
+
candidates.append(bracket_match.group(1))
|
| 165 |
+
continue
|
| 166 |
+
|
| 167 |
+
inline_match = re.match(r"^([A-Za-z][A-Za-z0-9 /'().,-]{1,45})\s*:\s+_{2,}", cleaned)
|
| 168 |
+
if inline_match:
|
| 169 |
+
candidates.append(inline_match.group(1))
|
| 170 |
+
|
| 171 |
+
lowered_form = normalize_label(form_text)
|
| 172 |
+
for common in COMMON_FIELDS:
|
| 173 |
+
if common in lowered_form:
|
| 174 |
+
candidates.append(common)
|
| 175 |
+
|
| 176 |
+
unique: list[str] = []
|
| 177 |
+
seen: set[str] = set()
|
| 178 |
+
for candidate in candidates:
|
| 179 |
+
label = normalize_label(candidate)
|
| 180 |
+
if len(label) < 2 or label in seen:
|
| 181 |
+
continue
|
| 182 |
+
seen.add(label)
|
| 183 |
+
unique.append(candidate.strip(" :_"))
|
| 184 |
+
return unique
|
| 185 |
+
|
| 186 |
+
|
| 187 |
+
def match_field(field: str, facts: dict[str, str]) -> FieldMatch:
|
| 188 |
+
"""Match one field label to available user facts."""
|
| 189 |
+
|
| 190 |
+
field_norm = normalize_label(field)
|
| 191 |
+
field_tokens = _tokens(field)
|
| 192 |
+
best_key = ""
|
| 193 |
+
best_score = 0.0
|
| 194 |
+
|
| 195 |
+
for key in facts:
|
| 196 |
+
key_tokens = _tokens(key)
|
| 197 |
+
overlap = len(field_tokens & key_tokens) / max(1, len(field_tokens | key_tokens))
|
| 198 |
+
ratio = SequenceMatcher(None, field_norm, key).ratio()
|
| 199 |
+
score = max(overlap, ratio * 0.85)
|
| 200 |
+
if score > best_score:
|
| 201 |
+
best_key = key
|
| 202 |
+
best_score = score
|
| 203 |
+
|
| 204 |
+
if not best_key:
|
| 205 |
+
return _missing_match(field, "No matching user fact found.")
|
| 206 |
+
|
| 207 |
+
value = facts[best_key]
|
| 208 |
+
sensitive = is_sensitive_field(field)
|
| 209 |
+
if best_score >= 0.86 or field_norm == best_key:
|
| 210 |
+
confidence = 95 if not sensitive else 84
|
| 211 |
+
status = "review" if sensitive else "ready"
|
| 212 |
+
note = "Strong label match."
|
| 213 |
+
elif best_score >= 0.58:
|
| 214 |
+
confidence = int(best_score * 100)
|
| 215 |
+
status = "review"
|
| 216 |
+
note = f"Possible match from '{best_key}'."
|
| 217 |
+
else:
|
| 218 |
+
return _missing_match(field, "No close enough user fact found.")
|
| 219 |
+
|
| 220 |
+
if sensitive:
|
| 221 |
+
note += " Sensitive field: verify manually before copying."
|
| 222 |
+
|
| 223 |
+
return FieldMatch(
|
| 224 |
+
field=field,
|
| 225 |
+
proposed_value=value,
|
| 226 |
+
status=status,
|
| 227 |
+
confidence=confidence,
|
| 228 |
+
source=best_key,
|
| 229 |
+
note=note,
|
| 230 |
+
)
|
| 231 |
+
|
| 232 |
+
|
| 233 |
+
def _missing_match(field: str, note: str) -> FieldMatch:
|
| 234 |
+
return FieldMatch(
|
| 235 |
+
field=field,
|
| 236 |
+
proposed_value="",
|
| 237 |
+
status="missing",
|
| 238 |
+
confidence=0,
|
| 239 |
+
source="",
|
| 240 |
+
note=note,
|
| 241 |
+
)
|
| 242 |
+
|
| 243 |
+
|
| 244 |
+
def is_sensitive_field(field: str) -> bool:
|
| 245 |
+
label = normalize_label(field)
|
| 246 |
+
return any(term in label for term in SENSITIVE_TERMS)
|
| 247 |
+
|
| 248 |
+
|
| 249 |
+
def questions_for_missing(matches: list[FieldMatch]) -> list[str]:
|
| 250 |
+
"""Generate plain-English follow-up questions for missing fields."""
|
| 251 |
+
|
| 252 |
+
questions = []
|
| 253 |
+
for match in matches:
|
| 254 |
+
if match.status == "missing":
|
| 255 |
+
questions.append(f"What should go in '{match.field}'?")
|
| 256 |
+
return questions
|
| 257 |
+
|
| 258 |
+
|
| 259 |
+
def _risk_summary(matches: list[FieldMatch]) -> list[str]:
|
| 260 |
+
risks = []
|
| 261 |
+
missing = sum(1 for match in matches if match.status == "missing")
|
| 262 |
+
review = sum(1 for match in matches if match.status == "review")
|
| 263 |
+
sensitive = sum(1 for match in matches if is_sensitive_field(match.field))
|
| 264 |
+
if missing:
|
| 265 |
+
risks.append(f"{missing} field(s) still need information.")
|
| 266 |
+
if review:
|
| 267 |
+
risks.append(f"{review} field(s) should be reviewed before copying.")
|
| 268 |
+
if sensitive:
|
| 269 |
+
risks.append(f"{sensitive} sensitive field(s) detected.")
|
| 270 |
+
if not risks:
|
| 271 |
+
risks.append("All detected fields have proposed values, but user review is still required.")
|
| 272 |
+
return risks
|
| 273 |
+
|
| 274 |
+
|
| 275 |
+
def analyze_form(form_text: str, user_facts: str, use_demo_fields: bool = True) -> dict[str, Any]:
|
| 276 |
+
"""Analyze form text and facts into reviewable outputs."""
|
| 277 |
+
|
| 278 |
+
fields = detect_fields(form_text)
|
| 279 |
+
if not fields and use_demo_fields:
|
| 280 |
+
fields = ["Full name", "Email", "Phone", "Address", "Date", "Signature"]
|
| 281 |
+
|
| 282 |
+
facts = parse_user_facts(user_facts)
|
| 283 |
+
matches = [match_field(field, facts) for field in fields]
|
| 284 |
+
rows = [asdict(match) for match in matches]
|
| 285 |
+
return {
|
| 286 |
+
"created_at": datetime.now(timezone.utc).isoformat(),
|
| 287 |
+
"fields": fields,
|
| 288 |
+
"facts": facts,
|
| 289 |
+
"rows": rows,
|
| 290 |
+
"questions": questions_for_missing(matches),
|
| 291 |
+
"risk_summary": _risk_summary(matches),
|
| 292 |
+
"copy_ready": copy_ready_text(matches),
|
| 293 |
+
}
|
| 294 |
+
|
| 295 |
+
|
| 296 |
+
def copy_ready_text(matches: list[FieldMatch]) -> str:
|
| 297 |
+
"""Create a conservative copy-ready field list."""
|
| 298 |
+
|
| 299 |
+
lines = []
|
| 300 |
+
for match in matches:
|
| 301 |
+
value = match.proposed_value if match.proposed_value else "[NEEDS USER INPUT]"
|
| 302 |
+
flag = " REVIEW" if match.status == "review" else ""
|
| 303 |
+
lines.append(f"{match.field}: {value}{flag}")
|
| 304 |
+
return "\n".join(lines)
|
| 305 |
+
|
| 306 |
+
|
| 307 |
+
def export_trace(payload: dict[str, Any], directory: Path | str = "traces") -> str:
|
| 308 |
+
"""Write one anonymizable JSON trace and return its path."""
|
| 309 |
+
|
| 310 |
+
out_dir = Path(directory)
|
| 311 |
+
out_dir.mkdir(parents=True, exist_ok=True)
|
| 312 |
+
stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
|
| 313 |
+
path = out_dir / f"formpilot_trace_{stamp}.json"
|
| 314 |
+
path.write_text(json.dumps(payload, indent=2), encoding="utf-8")
|
| 315 |
+
return str(path)
|
| 316 |
+
|
| 317 |
+
|
| 318 |
+
def rows_to_csv(rows: list[dict[str, Any]]) -> str:
|
| 319 |
+
"""Serialize rows for quick export."""
|
| 320 |
+
|
| 321 |
+
output = io.StringIO()
|
| 322 |
+
writer = csv.DictWriter(
|
| 323 |
+
output,
|
| 324 |
+
fieldnames=["field", "proposed_value", "status", "confidence", "source", "note"],
|
| 325 |
+
)
|
| 326 |
+
writer.writeheader()
|
| 327 |
+
writer.writerows(rows)
|
| 328 |
+
return output.getvalue()
|
formpilot/model_assist.py
ADDED
|
@@ -0,0 +1,73 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Optional small-model assist for form analysis."""
|
| 2 |
+
|
| 3 |
+
from __future__ import annotations
|
| 4 |
+
|
| 5 |
+
import json
|
| 6 |
+
import os
|
| 7 |
+
from typing import Any
|
| 8 |
+
|
| 9 |
+
|
| 10 |
+
DEFAULT_SMALL_MODEL = os.getenv("FORMPILOT_MODEL", "openbmb/MiniCPM5-1B")
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
def build_model_prompt(form_text: str, user_facts: str) -> str:
|
| 14 |
+
"""Prompt a small model to return conservative form-fill JSON."""
|
| 15 |
+
|
| 16 |
+
return f"""You are helping prepare a form for human review. Do not submit anything.
|
| 17 |
+
|
| 18 |
+
Return only JSON with this schema:
|
| 19 |
+
{{
|
| 20 |
+
"fields": [
|
| 21 |
+
{{
|
| 22 |
+
"field": "field label",
|
| 23 |
+
"proposed_value": "value or empty string",
|
| 24 |
+
"status": "ready|review|missing",
|
| 25 |
+
"confidence": 0,
|
| 26 |
+
"source": "fact used or empty",
|
| 27 |
+
"note": "short reason"
|
| 28 |
+
}}
|
| 29 |
+
],
|
| 30 |
+
"questions": ["questions for missing fields"],
|
| 31 |
+
"risk_summary": ["review warnings"]
|
| 32 |
+
}}
|
| 33 |
+
|
| 34 |
+
Rules:
|
| 35 |
+
- Use only the user facts.
|
| 36 |
+
- If a value is absent, mark missing.
|
| 37 |
+
- Sensitive fields must be review, not ready.
|
| 38 |
+
- Never invent account numbers, IDs, dates, signatures, addresses, or legal facts.
|
| 39 |
+
|
| 40 |
+
FORM:
|
| 41 |
+
{form_text}
|
| 42 |
+
|
| 43 |
+
USER FACTS:
|
| 44 |
+
{user_facts}
|
| 45 |
+
"""
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
def try_hf_model_assist(form_text: str, user_facts: str, model_id: str = DEFAULT_SMALL_MODEL) -> dict[str, Any]:
|
| 49 |
+
"""Call a small Hugging Face model and parse its JSON response."""
|
| 50 |
+
|
| 51 |
+
try:
|
| 52 |
+
from huggingface_hub import InferenceClient
|
| 53 |
+
except ImportError as exc:
|
| 54 |
+
raise RuntimeError("huggingface_hub is not installed.") from exc
|
| 55 |
+
|
| 56 |
+
token = os.getenv("HF_TOKEN") or os.getenv("HUGGINGFACEHUB_API_TOKEN")
|
| 57 |
+
client = InferenceClient(model=model_id, token=token)
|
| 58 |
+
prompt = build_model_prompt(form_text, user_facts)
|
| 59 |
+
response = client.text_generation(
|
| 60 |
+
prompt,
|
| 61 |
+
max_new_tokens=700,
|
| 62 |
+
temperature=0.1,
|
| 63 |
+
return_full_text=False,
|
| 64 |
+
)
|
| 65 |
+
return _parse_json_response(str(response))
|
| 66 |
+
|
| 67 |
+
|
| 68 |
+
def _parse_json_response(raw: str) -> dict[str, Any]:
|
| 69 |
+
start = raw.find("{")
|
| 70 |
+
end = raw.rfind("}")
|
| 71 |
+
if start == -1 or end == -1 or end <= start:
|
| 72 |
+
raise ValueError("Model did not return a JSON object.")
|
| 73 |
+
return json.loads(raw[start : end + 1])
|
requirements.txt
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
gradio>=6.16.0
|
| 2 |
+
pandas>=2.2.0
|
| 3 |
+
pydantic>=2.0.0
|
| 4 |
+
huggingface_hub>=0.32.0
|
| 5 |
+
pytest>=8.0.0
|
tests/test_engine.py
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from formpilot.engine import analyze_form, detect_fields, parse_user_facts
|
| 2 |
+
|
| 3 |
+
|
| 4 |
+
def test_detect_fields_from_underlined_form() -> None:
|
| 5 |
+
form = """
|
| 6 |
+
Full name: ____________
|
| 7 |
+
Email: ________________
|
| 8 |
+
Emergency contact: ____
|
| 9 |
+
"""
|
| 10 |
+
assert detect_fields(form) == ["Full name", "Email", "Emergency contact"]
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
def test_parse_user_facts_key_values_and_free_text() -> None:
|
| 14 |
+
facts = parse_user_facts(
|
| 15 |
+
"""
|
| 16 |
+
Name: Jordan Lee
|
| 17 |
+
jordan.lee@example.com
|
| 18 |
+
Phone: 555-0137
|
| 19 |
+
"""
|
| 20 |
+
)
|
| 21 |
+
assert facts["name"] == "Jordan Lee"
|
| 22 |
+
assert facts["email"] == "jordan.lee@example.com"
|
| 23 |
+
assert facts["phone"] == "555-0137"
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
def test_analyze_form_marks_missing_fields() -> None:
|
| 27 |
+
payload = analyze_form("Full name: ____\nEmail: ____\nSignature: ____", "Full name: Jordan Lee")
|
| 28 |
+
rows = {row["field"]: row for row in payload["rows"]}
|
| 29 |
+
assert rows["Full name"]["status"] == "ready"
|
| 30 |
+
assert rows["Email"]["status"] == "missing"
|
| 31 |
+
assert rows["Signature"]["status"] == "missing"
|
| 32 |
+
assert payload["questions"]
|