Professional model card with full documentation

README.md

@@ -8,84 +8,269 @@ tags:
 - family-law
 - ohio
 - modernbert
-- mlm
+- masked-lm
+- domain-adapted
+- encoder
 base_model: answerdotai/ModernBERT-base
 datasets:
 - Roderick3rd/OhioCustodyBERT-corpus
 pipeline_tag: fill-mask
+library_name: transformers
 ---
 
 # OhioCustodyBERT
 
-A domain-adapted language model for **Ohio family law and child custody** cases, fine-tuned from [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base).
+<p align="center">
+  <em>A domain-adapted legal language model for Ohio family law and child custody proceedings</em>
+</p>
 
-## Overview
+## Table of Contents
+
+- [Model Summary](#model-summary)
+- [Usage](#usage)
+- [Training Data](#training-data)
+- [Training Details](#training-details)
+- [Results](#results)
+- [Intended Use & Limitations](#intended-use--limitations)
+- [Roadmap](#roadmap)
+- [License](#license)
+- [Citation](#citation)
+
+---
+
+## Model Summary
+
+OhioCustodyBERT is a **domain-adapted bidirectional encoder** fine-tuned from [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) on a curated corpus of Ohio family law and child custody case analyses. It inherits ModernBERT's architectural improvements — Rotary Positional Embeddings (RoPE), Local-Global Alternating Attention, Flash Attention, and 8,192-token context — while specializing in the legal domain of domestic relations.
 
 | | |
 |---|---|
-| **Base Model** | answerdotai/ModernBERT-base (149.7M params) |
+| **Base Model** | [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) (149.7M params) |
+| **Architecture** | Encoder-only, Pre-Norm Transformer, GeGLU, RoPE |
+| **Context Length** | Up to 8,192 tokens |
 | **Domain** | Ohio family law, child custody, domestic relations |
-| **Training** | Masked Language Modeling (MLM) on custody case corpus |
-| **Hardware** | 2× NVIDIA Tesla T4 (Kaggle) |
-| **Corpus** | 864 curated custody case analyses |
+| **Task** | Masked Language Modeling (MLM) |
+| **Training Hardware** | 2× NVIDIA Tesla T4 (Kaggle, 32GB VRAM total) |
 | **Developer** | [Phoenix Intelligence LLC](https://github.com/rodneymullins) |
+| **Version** | v0.1 (proof-of-concept) |
 
-## Training Data
-
-Fine-tuned on the [OhioCustodyBERT-corpus](https://huggingface.co/datasets/Roderick3rd/OhioCustodyBERT-corpus), which contains structured analyses of Ohio custody cases including:
+OhioCustodyBERT is designed to understand the specialized vocabulary, reasoning patterns, and statutory references found in Ohio custody proceedings — including best-interest factors under ORC § 3109.04, guardian ad litem recommendations, shared parenting plans, and modification standards.
 
-- **Case excerpts** from Ohio appellate decisions (e.g., *In re M.J.C.*, *Davis v. Davis*)
-- **Best-interest factor analysis** under Ohio Revised Code § 3109.04
-- **Guardian ad litem recommendations**
-- **Custody modification standards**
-- **Shared parenting plan evaluations**
+---
 
 ## Usage
 
+Install transformers ≥ 4.48.0:
+
+```bash
+pip install -U transformers>=4.48.0
+```
+
+For best performance, install Flash Attention:
+
+```bash
+pip install flash-attn
+```
+
+### Fill-Mask (MLM)
+
 ```python
 from transformers import pipeline
 
 fill = pipeline("fill-mask", model="Roderick3rd/OhioCustodyBERT")
 
-# Predict custody terms
-result = fill("The court granted [MASK] custody to the mother.")
-for r in result[:3]:
-    print(f"{r['token_str']:15s} ({r['score']:.3f})")
+# Predict custody-specific legal terms
+results = fill("The court granted [MASK] custody to the mother.")
+for r in results[:5]:
+    print(f"  {r['token_str']:20s} (score: {r['score']:.4f})")
 ```
 
-## Use Cases
+### As a Feature Extractor
+
+```python
+from transformers import AutoTokenizer, AutoModel
+import torch
+
+tokenizer = AutoTokenizer.from_pretrained("Roderick3rd/OhioCustodyBERT")
+model = AutoModel.from_pretrained("Roderick3rd/OhioCustodyBERT")
+
+text = "The guardian ad litem recommended supervised visitation based on the best interest of the child."
+inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)
 
-- **Legal research**: Auto-complete and suggest custody-relevant legal terminology
-- **Document analysis**: Enhanced understanding of family law documents
-- **Case prediction**: Feature extraction for custody outcome modeling
-- **RAG pipelines**: Better embeddings for custody-related retrieval
+with torch.no_grad():
+    outputs = model(**inputs)
+
+# Use [CLS] embedding for classification or retrieval
+cls_embedding = outputs.last_hidden_state[:, 0, :]
+print(f"Embedding shape: {cls_embedding.shape}")  # [1, 768]
+```
+
+### Fine-Tuning for Downstream Tasks
+
+OhioCustodyBERT can be further fine-tuned for:
+
+- **Custody outcome classification** — predict custody type from case facts
+- **Legal NER** — extract parties, statutes, dates, custody terms
+- **Semantic search** — retrieve relevant precedent from case databases
+- **Document classification** — categorize motions, orders, GAL reports
+
+Fine-tuning follows standard BERT recipes. See [Hugging Face fine-tuning tutorial](https://huggingface.co/docs/transformers/training).
+
+---
+
+## Training Data
+
+Fine-tuned on the [OhioCustodyBERT-corpus](https://huggingface.co/datasets/Roderick3rd/OhioCustodyBERT-corpus), containing **864 curated training examples** and **95 validation examples** of structured Ohio custody case analyses.
+
+### Corpus Contents
+
+| Category | Description |
+|----------|-------------|
+| **Case Law** | Excerpts from Ohio appellate custody decisions (e.g., *In re M.J.C.*, *Davis v. Davis*) |
+| **Best-Interest Analysis** | Factor-by-factor analysis under Ohio Revised Code § 3109.04(F)(1) |
+| **GAL Reports** | Guardian ad litem recommendation patterns and reasoning |
+| **Modification Standards** | Change-of-circumstances analysis for custody modifications |
+| **Shared Parenting** | Shared parenting plan evaluations and court reasoning |
+| **Statutory References** | ORC § 3109.04, § 3109.051, § 3109.052, and related provisions |
+
+### Data Format
+
+Each example is structured as a system-prompted analysis:
+- **System prompt**: Legal research assistant context for Ohio family law
+- **Case content**: Extracted custody-relevant passages with analytical commentary
+
+### Planned Corpus Expansion (v0.2+)
+
+| Source | Status | Estimated Size |
+|--------|--------|----------------|
+| CourtListener (6.7M opinions) | 🔄 Downloading | ~50,000 family law cases |
+| GovInfo US Code (family law titles) | ✅ Downloaded | Federal statutes |
+| LeXFiles (19B tokens) | ✅ Downloaded | Broad legal foundation |
+| Ohio Revised Code (lawriter.net) | ⏳ Planned | Ohio-specific statutes |
+| Ohio Supreme Court slip opinions | ⏳ Planned | Latest case law |
+
+---
 
 ## Training Details
 
-- **Epochs**: 3
-- **Batch size**: 4 per device × 32 gradient accumulation = 128 effective
-- **Learning rate**: 5e-5 with cosine schedule
-- **Precision**: bf16 mixed precision
-- **Gradient checkpointing**: Enabled
+### Configuration
+
+| Parameter | Value |
+|-----------|-------|
+| **Epochs** | 3 |
+| **Per-device batch size** | 4 |
+| **Gradient accumulation** | 32 (effective batch: 128) |
+| **Learning rate** | 5e-5 |
+| **LR scheduler** | Cosine with 10-step warmup |
+| **Weight decay** | 0.01 |
+| **Precision** | bf16 mixed precision |
+| **Gradient checkpointing** | Enabled |
+| **MLM probability** | 15% |
+| **Max sequence length** | 512 tokens |
+
+### Training Infrastructure
+
+- **Hardware**: 2× NVIDIA Tesla T4 (15.6 GB VRAM each) on Kaggle
+- **Training time**: ~7 minutes
+- **Framework**: Hugging Face Transformers + Accelerate
+
+---
+
+## Results
+
+### Training Metrics
+
+| Step | Training Loss |
+|------|---------------|
+| 5 | 1.0149 |
+| 10 | 0.9345 |
+| 15 | 0.8611 |
+| 20 | 0.8364 |
+| **Final** | **0.9089** |
+
+Loss decreased **17.4%** from step 5 to step 20, indicating the model is learning custody-specific language patterns. With only 864 examples and 3 epochs, this is a proof-of-concept demonstrating the training pipeline works end-to-end.
+
+### Expected Improvements (v0.2)
+
+- **10-100x more training data** from CourtListener + GovInfo family law filtering
+- **More epochs** with larger dataset
+- **Downstream task evaluation** on custody classification benchmarks
+- **Comparison** against base ModernBERT and legal-specific models (LegalBERT, SaulLM)
+
+---
+
+## Intended Use & Limitations
+
+### Intended Use
+
+- **Legal research augmentation**: Enhanced understanding of custody terminology for RAG pipelines
+- **Document analysis**: Feature extraction for custody case classification systems
+- **Academic research**: Study of judicial reasoning patterns in family law
+- **Legal tech prototyping**: Foundation for custody-aware NLP applications
+
+### Limitations
 
-## Limitations
+⚠️ **This is v0.1 — a proof-of-concept model.**
 
-- Trained on Ohio-specific case law; may not generalize to other jurisdictions
-- Small corpus (864 examples) — best used as a starting point for further fine-tuning
-- Not a substitute for legal advice
+- **Small corpus**: 864 training examples is insufficient for robust legal understanding. Production use requires the expanded corpus (v0.2+).
+- **Ohio-specific**: Trained on Ohio case law; may not generalize to other states' custody standards, which vary significantly.
+- **Not legal advice**: This model does not provide legal advice and should not be relied upon for legal decision-making.
+- **MLM only**: Currently trained for masked language modeling. Task-specific fine-tuning (classification, NER, QA) is needed for downstream applications.
+- **Bias**: May reflect biases present in the training data, including historical patterns in judicial decision-making.
+
+### Ethical Considerations
+
+Family law involves sensitive matters affecting children and families. Any application of this model in production systems should:
+- Include human oversight for all outputs
+- Be transparent about AI involvement
+- Not be used to predict outcomes in active custody proceedings without legal professional review
+- Comply with applicable rules of professional conduct
+
+---
+
+## Roadmap
+
+| Version | Target | Key Changes |
+|---------|--------|-------------|
+| **v0.1** ✅ | Feb 2026 | Proof-of-concept, 864 examples, MLM training pipeline |
+| **v0.2** | Mar 2026 | 10,000+ examples from CourtListener + GovInfo, improved loss |
+| **v0.3** | Apr 2026 | Task fine-tuning: custody outcome classification |
+| **v1.0** | Q2 2026 | Production-ready with evaluation benchmarks |
+
+---
 
 ## License
 
-Apache 2.0
+Released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0), consistent with the base ModernBERT model.
+
+---
 
 ## Citation
 
 ```bibtex
 @misc{ohiocustodybert2026,
-  title={OhioCustodyBERT: Domain-Adapted Language Model for Ohio Family Law},
-  author={Phoenix Intelligence LLC},
+  title={OhioCustodyBERT: A Domain-Adapted Language Model for Ohio Family Law and Child Custody},
+  author={Mullins, Roderick and Phoenix Intelligence LLC},
   year={2026},
   publisher={Hugging Face},
-  url={https://huggingface.co/Roderick3rd/OhioCustodyBERT}
+  howpublished={\url{https://huggingface.co/Roderick3rd/OhioCustodyBERT}},
 }
 ```
+
+### Base Model Citation
+
+```bibtex
+@misc{modernbert,
+  title={Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference},
+  author={Benjamin Warner and Antoine Chaffin and Benjamin Clavié and Orion Weller and Oskar Hallström and Said Taghadouini and Alexis Gallagher and Raja Biswas and Faisal Ladhak and Tom Aarsen and Nathan Cooper and Griffin Adams and Jeremy Howard and Iacopo Poli},
+  year={2024},
+  eprint={2412.13663},
+  archivePrefix={arXiv},
+  primaryClass={cs.CL},
+}
+```
+
+---
+
+<p align="center">
+  <em>Built by <a href="https://github.com/rodneymullins">Phoenix Intelligence LLC</a> • Part of the Antigravity AI Infrastructure</em>
+</p>