| --- |
| license: apache-2.0 |
| language: |
| - en |
| base_model: distilbert/distilbert-base-uncased |
| pipeline_tag: text-classification |
| tags: |
| - document-classification |
| - document-ai |
| - pii-detection |
| - redaction |
| - distilbert |
| library_name: transformers |
| --- |
| |
| # FR-Docs-v1 |
|
|
| Document type recognition, classification, and sensitive-content detection. FR-Docs identifies uploaded documents (Word, PDF, text), classifies them into **30 business categories** across 8 groups, flags sensitive spans for redaction, and returns an LLM routing suggestion, so downstream systems can assign each document to the most suitable model automatically. |
|
|
| Built by [Neural Arc Inc.](https://he2.ai) as the document intelligence layer for the Helium AI workspace. |
|
|
| ## What is in this repo |
|
|
| - **Fine-tuned DistilBERT classifier** (67M parameters): 31 labels (30 categories plus `other`), trained with the Hugging Face Trainer. |
| - **`pipeline/` folder**: the full FR-Docs pipeline source, including format detection and text extraction (PyMuPDF, python-docx), scanned-PDF detection, a three-layer sensitive-content scanner (regex, gazetteer, zero-shot GLiNER), redaction helpers, a FastAPI service with a browser tester UI, and training scripts for retraining or upgrading the base model. |
|
|
| ## Categories |
|
|
| Legal: contract, nda, court_filing, patent, compliance_filing. Financial: invoice, receipt, purchase_order, financial_statement, tax_document, insurance_document. HR: cv_resume, offer_letter, policy_document. Product: product_spec, datasheet, user_manual. Sales and Marketing: proposal, case_study, press_release, marketing_collateral. Operations: sop, report, meeting_minutes, form, presentation. Correspondence: email_thread, letter, memo. Technical: research_paper. Fallback: other. |
| |
| ## Usage |
| |
| Classification only: |
| |
| ```python |
| from transformers import pipeline |
| |
| clf = pipeline("text-classification", model="FahrenheitResearch/FR-Docs-v1") |
| print(clf("INVOICE #INV-2026-0142 ... TOTAL DUE: $27,435. Payment terms: Net 30.")) |
| # [{'label': 'invoice', 'score': ...}] |
| ``` |
| |
| Full pipeline (extraction, classification, sensitive-span detection, routing): |
| |
| ```bash |
| hf download FahrenheitResearch/FR-Docs-v1 --include "pipeline/*" --local-dir fr-docs |
| cd fr-docs/pipeline |
| pip install -r requirements.txt |
| FR_DOCS_SENSITIVITY=strict uvicorn fr_docs.api:app --port 8080 |
| # open http://localhost:8080 for the drag-and-drop tester |
| ``` |
| |
| ## Sensitive-content detection |
| |
| Three stacked detectors return character-offset spans for redaction: **regex** (emails, phones, credit cards with Luhn validation, SSN, PAN, Aadhaar, GSTIN, IBAN, passports, vehicle registrations, IPs, URLs, money amounts, salaries, account numbers, dates of birth, equity stakes), a **gazetteer** of organization-specific brand, product, and client names (editable YAML, no retraining), and optional **zero-shot NER** via GLiNER for person names, companies, and addresses. A `strict` profile additionally flags all dates, standalone number sequences, and reference IDs. |
| |
| ## Training and limitations |
| |
| v1 was trained on a fully synthetic templated corpus (7,750 documents, 250 per class). It performs well on clearly structured business documents but has not yet been trained on real-world data; expect degraded accuracy on unusual layouts, informal writing, and domains far from the templates. Scanned documents without a text layer are detected and deferred rather than classified. The training scripts in `pipeline/training/` support retraining on your own labeled JSONL with any Hugging Face base model, including ModernBERT for long documents. |
| |
| ## License |
| |
| Apache 2.0. |
| |
