Update model card: Legal Agent Council architecture with Qwopus-27B judge, corrected corpus stats (469 opinions, 916 audited SFT pairs)

Browse files

Files changed (1) hide show

README.md +54 -325

README.md CHANGED Viewed

@@ -4,364 +4,93 @@ language:
 - en
 tags:
 - legal
 - custody
 - family-law
-- ohio
 - modernbert
-- masked-lm
-- domain-adapted
-- encoder
-base_model: answerdotai/ModernBERT-base
 datasets:
 - Roderick3rd/OhioCustodyBERT-corpus
-pipeline_tag: fill-mask
-library_name: transformers
 ---
 # OhioCustodyBERT
-<p align="center">
-  <em>A domain-adapted legal language model for Ohio family law and child custody proceedings</em>
-</p>
-## Table of Contents
-- [Model Summary](#model-summary)
-- [Usage](#usage)
-- [Training Data](#training-data)
-- [Training Details](#training-details)
-- [Results](#results)
-- [Intended Use & Limitations](#intended-use--limitations)
-- [Roadmap](#roadmap)
-- [License](#license)
-- [Citation](#citation)
----
-## Model Summary
-OhioCustodyBERT is a **domain-adapted bidirectional encoder** fine-tuned from [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) on a curated corpus of Ohio family law and child custody case analyses. It inherits ModernBERT's architectural improvements — Rotary Positional Embeddings (RoPE), Local-Global Alternating Attention, Flash Attention, and 8,192-token context — while specializing in the legal domain of domestic relations.
 | | |
 |---|---|
-| **Base Model** | [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) (149.7M params) |
-| **Architecture** | Encoder-only, Pre-Norm Transformer, GeGLU, RoPE |
-| **Context Length** | Up to 8,192 tokens |
-| **Domain** | Ohio family law, child custody, domestic relations |
-| **Task** | Masked Language Modeling (MLM) |
-| **Training Hardware** | 2× NVIDIA Tesla T4 (Kaggle, 32GB VRAM total) |
-| **Developer** | [Phoenix Intelligence LLC](https://github.com/rodneymullins) |
-| **Version** | v0.1 (proof-of-concept) |
-OhioCustodyBERT is designed to understand the specialized vocabulary, reasoning patterns, and statutory references found in Ohio custody proceedings — including best-interest factors under ORC § 3109.04, guardian ad litem recommendations, shared parenting plans, and modification standards.
----
-## Usage
-Install transformers ≥ 4.48.0:
-```bash
-pip install -U transformers>=4.48.0
-```
-For best performance, install Flash Attention:
-```bash
-pip install flash-attn
-```
-### Fill-Mask (MLM)
-```python
-from transformers import pipeline
-fill = pipeline("fill-mask", model="Roderick3rd/OhioCustodyBERT")
-# Predict custody-specific legal terms
-results = fill("The court granted [MASK] custody to the mother.")
-for r in results[:5]:
-    print(f"  {r['token_str']:20s} (score: {r['score']:.4f})")
-```
-### As a Feature Extractor
-```python
-from transformers import AutoTokenizer, AutoModel
-import torch
-tokenizer = AutoTokenizer.from_pretrained("Roderick3rd/OhioCustodyBERT")
-model = AutoModel.from_pretrained("Roderick3rd/OhioCustodyBERT")
-text = "The guardian ad litem recommended supervised visitation based on the best interest of the child."
-inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)
-with torch.no_grad():
-    outputs = model(**inputs)
-# Use [CLS] embedding for classification or retrieval
-cls_embedding = outputs.last_hidden_state[:, 0, :]
-print(f"Embedding shape: {cls_embedding.shape}")  # [1, 768]
-```
-### Fine-Tuning for Downstream Tasks
-OhioCustodyBERT can be further fine-tuned for:
-- **Custody outcome classification** — predict custody type from case facts
-- **Legal NER** — extract parties, statutes, dates, custody terms
-- **Semantic search** — retrieve relevant precedent from case databases
-- **Document classification** — categorize motions, orders, GAL reports
-Fine-tuning follows standard BERT recipes. See [Hugging Face fine-tuning tutorial](https://huggingface.co/docs/transformers/training).
----
-## Training Data
-Fine-tuned on the [OhioCustodyBERT-corpus](https://huggingface.co/datasets/Roderick3rd/OhioCustodyBERT-corpus), containing **864 curated training examples** and **95 validation examples** of structured Ohio custody case analyses.
-### Corpus Contents
-| Category | Description |
-|----------|-------------|
-| **Case Law** | Excerpts from Ohio appellate custody decisions (e.g., *In re M.J.C.*, *Davis v. Davis*) |
-| **Best-Interest Analysis** | Factor-by-factor analysis under Ohio Revised Code § 3109.04(F)(1) |
-| **GAL Reports** | Guardian ad litem recommendation patterns and reasoning |
-| **Modification Standards** | Change-of-circumstances analysis for custody modifications |
-| **Shared Parenting** | Shared parenting plan evaluations and court reasoning |
-| **Statutory References** | ORC § 3109.04, § 3109.051, § 3109.052, and related provisions |
-### Data Format
-Each example is structured as a system-prompted analysis:
-- **System prompt**: Legal research assistant context for Ohio family law
-- **Case content**: Extracted custody-relevant passages with analytical commentary
-### Planned Corpus Expansion (v0.2+)
-| Source | Status | Estimated Size |
-|--------|--------|----------------|
-| CourtListener (6.7M opinions) | 🔄 Downloading | ~50,000 family law cases |
-| GovInfo US Code (family law titles) | ✅ Downloaded | Federal statutes |
-| LeXFiles (19B tokens) | ✅ Downloaded | Broad legal foundation |
-| Ohio Revised Code (lawriter.net) | ⏳ Planned | Ohio-specific statutes |
-| Ohio Supreme Court slip opinions | ⏳ Planned | Latest case law |
----
-## Training Details
-### Configuration
-| Parameter | Value |
-|-----------|-------|
-| **Epochs** | 3 |
-| **Per-device batch size** | 4 |
-| **Gradient accumulation** | 32 (effective batch: 128) |
-| **Learning rate** | 5e-5 |
-| **LR scheduler** | Cosine with 10-step warmup |
-| **Weight decay** | 0.01 |
-| **Precision** | bf16 mixed precision |
-| **Gradient checkpointing** | Enabled |
-| **MLM probability** | 15% |
-| **Max sequence length** | 512 tokens |
-### Training Infrastructure
-- **Hardware**: 2× NVIDIA Tesla T4 (15.6 GB VRAM each) on Kaggle
-- **Training time**: ~7 minutes
-- **Framework**: Hugging Face Transformers + Accelerate
----
-## Results
-### Training Metrics
-| Step | Training Loss |
-|------|---------------|
-| 5 | 1.0149 |
-| 10 | 0.9345 |
-| 15 | 0.8611 |
-| 20 | 0.8364 |
-| **Final** | **0.9089** |
-Loss decreased **17.4%** from step 5 to step 20, indicating the model is learning custody-specific language patterns. With only 864 examples and 3 epochs, this is a proof-of-concept demonstrating the training pipeline works end-to-end.
-### Expected Improvements (v0.2)
-- **10-100x more training data** from CourtListener + GovInfo family law filtering
-- **More epochs** with larger dataset
-- **Downstream task evaluation** on custody classification benchmarks
-- **Comparison** against base ModernBERT and legal-specific models (LegalBERT, SaulLM)
----
-## Intended Use & Limitations
-### Intended Use
-- **Legal research augmentation**: Enhanced understanding of custody terminology for RAG pipelines
-- **Document analysis**: Feature extraction for custody case classification systems
-- **Academic research**: Study of judicial reasoning patterns in family law
-- **Legal tech prototyping**: Foundation for custody-aware NLP applications
-### Limitations
-⚠️ **This is v0.1 — a proof-of-concept model.**
-- **Small corpus**: 864 training examples is insufficient for robust legal understanding. Production use requires the expanded corpus (v0.2+).
-- **Ohio-specific**: Trained on Ohio case law; may not generalize to other states' custody standards, which vary significantly.
-- **Not legal advice**: This model does not provide legal advice and should not be relied upon for legal decision-making.
-- **MLM only**: Currently trained for masked language modeling. Task-specific fine-tuning (classification, NER, QA) is needed for downstream applications.
-- **Bias**: May reflect biases present in the training data, including historical patterns in judicial decision-making.
-### Ethical Considerations
-Family law involves sensitive matters affecting children and families. Any application of this model in production systems should:
-- Include human oversight for all outputs
-- Be transparent about AI involvement
-- Not be used to predict outcomes in active custody proceedings without legal professional review
-- Comply with applicable rules of professional conduct
----
-## Roadmap
-| Version | Target | Key Changes |
-|---------|--------|-------------|
-| **v0.1** ✅ | Feb 2026 | Proof-of-concept, 864 examples, MLM training pipeline |
-| **v0.2** | Mar 2026 | 10,000+ examples from CourtListener + GovInfo, improved loss |
-| **v0.3** | Apr 2026 | Task fine-tuning: custody outcome classification |
-| **v1.0** | Q2 2026 | Production-ready with evaluation benchmarks |
----
-## Ohio Family Law Reform: A Timeline
-Understanding the legislative and judicial history behind Ohio's custody framework is essential context for this model's domain.
-| Year | Event |
-|------|-------|
-| **1999** | The **Ohio Task Force on Family Law and Children** is created by the 122nd General Assembly. |
-| **2001** | The task force releases its final report to the legislature and Chief Justice Thomas J. Moyer, outlining goals for reforming Ohio's family law via proposed changes to **ORC Chapter 3109**. This leads Chief Justice Moyer to create the **Advisory Committee on Children, Family, and the Courts** (now the Advisory Committee on Children and Families). |
-| **2005** | The advisory committee issues its report and recommendations, including many of the same proposals to amend ORC Chapter 3109 as the 2001 task force. The **Subcommittee on Family Law Reform Implementation (FLRI)** is created. |
-| **2009** | The Supreme Court adopts **Guardian ad Litem standards, Sup.R. 48**. |
-| **2010** | The Supreme Court promulgates **Uniform Domestic Relations Forms 1–5**. |
-| **2013** | The Supreme Court promulgates **Uniform Domestic Relations Forms 6–28**. |
-| **2014** | The Supreme Court hosts the **Domestic Relations Summit**, an interdisciplinary training. The court also adopts **parent coordination guidelines, Sup.R. 16.60** (formerly Sup.R. 90.05). |
-| **2016** | The Supreme Court adopts **Sup.R. 44(C)(2)(h)(i–ix)** requiring confidential information be placed in a separate family file. |
-| **2022** | The Supreme Court adopts **custody evaluator standards, Sup.R. 91**. |
-| **2023** | The Supreme Court adopts **neutral evaluation guidelines, Sup.R. 16.50**. |
-| **2024** | **SB325** is introduced in the 135th General Assembly — the implementation of the original proposals for a significantly revised Chapter 3109, creating a new statutory scheme for family law in Ohio. The bill stalls in the Senate Judiciary Committee. |
-| **2025** | **SB174** is introduced in the 136th General Assembly (reintroduction of SB325). The bill is amended to address concerns and **passes the Ohio Senate in November**. |
-| **2026** | **SB174** is pending in the House Judiciary Committee where conversations continue around revisions to improve the bill's language. |
-> **Why this matters for OhioCustodyBERT:** The evolving statutory framework means Ohio family law is a moving target. Our model must be continuously updated as SB174 progresses and new case law interprets the changing standards. The transition from the current ORC § 3109.04 best-interest factors to the proposed revised Chapter 3109 will fundamentally reshape custody analysis in Ohio.
-### SB174 Key Provisions
-The bill implements the original task force's Goal 1: *"Establishing and maintaining a parent-child relationship is of fundamental importance to the welfare of a child."* Key changes include:
-- **Terminology reform**: Eliminates "residential parent" and "custodial parent" labels to reduce adversarial dynamics
-- **Parenting responsibilities** (§3109.04(A)(20)): Replaces "custody" framework with parenting responsibility allocation
-- **Expanded best-interest factors**: 26 enumerated factors (up from 9 + 5) under §3109.0430
-- **Maximizing parenting time**: Courts must consider maximizing time with each parent when in the child's best interest (§3109.044)
-- **Written findings required**: Courts must provide written findings of fact when rejecting joint parenting plans (§3109.046)
-- **Modification standards**: Changes may be granted due to change in circumstances of *either* parent (§3109.0419)
-- **Anti-interference provisions**: Specific procedures for motions alleging interference with parenting time (§§3109.0491-3109.0493)
-- **Attorney fee sanctions**: Courts may assess fees for bad-faith or frivolous modification motions (§3109.0420)
-- **Public policy declaration**: "(I)t is the public policy of this chapter, when it is in the child's best interest (1) to foster and continue the relationship between the child and each parent; (2) for the child's parents to have substantial, meaningful, and developmentally appropriate parenting time" (§3109.401)
-### Sources
-- **Ohio State Bar Association** — ["Kids Come First: Family Law Reform in Ohio"](https://www.ohiobar.org/member-tools-benefits/practice-resources/practice-library-search/practice-library/2026-ohio-lawyer/family-law-reform-in-ohio/), *Ohio Lawyer*, February 2026. Co-authored by OSBA, Ohio Judicial Conference, and Ohio Domestic Violence Network.
-- **Ohio Legislature** — [Senate Bill 174](https://www.legislature.ohio.gov/legislation/136/sb174), 136th General Assembly
-- **Supreme Court of Ohio** — [2001 Task Force Report: "Family Law Reform: Minimizing Conflict, Maximizing Families"](https://www.supremecourt.ohio.gov/docs/JCS/taskforce/report_final.pdf)
-- **Supreme Court of Ohio** — [2005 Advisory Committee Report and Recommendations](https://www.supremecourt.ohio.gov/docs/JCS/taskforce/FLCreport_2005.pdf)
-- **Supreme Court of Ohio** — [Planning for Parenting Time: Ohio's Guide for Parents Living Apart](https://www.supremecourt.ohio.gov/docs/Publications/JCS/parentingGuide.pdf)
-- **Supreme Court of Ohio** — [2002 GAL Standards Task Force Report](https://www.supremecourt.ohio.gov/docs/Publications/GAL/FinalReport.pdf)
 ### Reference Materials
-The following research and legal documents inform the model's training domain:
-1. **ECPH Encyclopedia of Psychology** (Zhang Kan) — Psychological foundations of custody evaluation
-2. **Effects on Children and Improving Relationships: Overnight Access Research** (Kelly & Lamb) — Seminal research on overnight visitation impact on child development
-3. **GAL Report Analysis** — Guardian ad Litem reporting patterns and recommendation structures
-4. **Gifted Child Memorandum** — Special considerations for gifted children in custody proceedings
-## Acknowledgments & Methodology Credits
-Our domain-adaptive pre-training approach draws inspiration from several key works:
-### AI4Bharat / IndicBERT
-Our training methodology follows the **domain-adaptive MLM fine-tuning** paradigm demonstrated by [AI4Bharat's IndicBERT](https://huggingface.co/ai4bharat/indic-bert) project. Key parallels:
-| IndicBERT | OhioCustodyBERT |
-|-----------|-----------------|
-| Pre-trained ALBERT on 8.9B tokens across 12 Indian languages | Fine-tuned ModernBERT on Ohio custody case corpus |
-| Masked Language Modeling (MLM) objective | Masked Language Modeling (MLM) objective |
-| Domain-specific corpus curation | Domain-specific corpus curation (legal/custody) |
-| Curriculum learning phases | Planned multi-phase training (v0.2+) |
-| Efficient training with limited compute | Trained on free-tier Kaggle 2×T4 GPUs |
-IndicBERT demonstrated that **focused domain adaptation on a well-curated corpus** can outperform larger general-purpose models — a principle we apply to the legal domain. Their curriculum learning approach (foundation → high-resource → low-resource) will inform our planned training phases:
-1. **Phase 1** (current): Broad legal language modeling on custody cases
-2. **Phase 2** (planned): Expanded corpus with CourtListener + GovInfo federal law
-3. **Phase 3** (planned): Ohio-specific statutes and recent slip opinions
-```bibtex
-@inproceedings{kakwani2020indicnlpsuite,
-    title={{IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages}},
-    author={Divyanshu Kakwani and Anoop Kunchukuttan and Satber Gopi and Gokul N.C. and Avik Bhattacharyya and Mitesh M. Khapra and Pratyush Kumar},
-    year={2020},
-    booktitle={Findings of EMNLP},
-}
-```
-### ModernBERT (Answer.AI)
-Our base model, [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base), provides the architectural foundation with modern improvements including RoPE, Flash Attention, and 8,192-token context. See citation in the [Citation](#citation) section.
-## License
-Released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0), consistent with the base ModernBERT model.
----
 ## Citation
 ```bibtex
 @misc{ohiocustodybert2026,
-  title={OhioCustodyBERT: A Domain-Adapted Language Model for Ohio Family Law and Child Custody},
-  author={Mullins, Roderick and Phoenix Intelligence LLC},
   year={2026},
-  publisher={Hugging Face},
-  howpublished={\url{https://huggingface.co/Roderick3rd/OhioCustodyBERT}},
 }
 ```
-### Base Model Citation
-```bibtex
-@misc{modernbert,
-  title={Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference},
-  author={Benjamin Warner and Antoine Chaffin and Benjamin Clavié and Orion Weller and Oskar Hallström and Said Taghadouini and Alexis Gallagher and Raja Biswas and Faisal Ladhak and Tom Aarsen and Nathan Cooper and Griffin Adams and Jeremy Howard and Iacopo Poli},
-  year={2024},
-  eprint={2412.13663},
-  archivePrefix={arXiv},
-  primaryClass={cs.CL},
-}
-```
----
-<p align="center">
-  <em>Built by <a href="https://github.com/rodneymullins">Phoenix Intelligence LLC</a> • Part of the Antigravity AI Infrastructure</em>
-</p>

 - en
 tags:
 - legal
+- ohio
 - custody
 - family-law
+- parental-alienation
+- bert
 - modernbert
 datasets:
 - Roderick3rd/OhioCustodyBERT-corpus
+pipeline_tag: text-classification
 ---
 # OhioCustodyBERT
+**Ohio Family Law & Custody Specialist** — Fine-tuned ModernBERT for Ohio family law, child custody, and parental alienation analysis.
+## Model Overview
 | | |
 |---|---|
+| **Base Model** | ModernBERT-base (149.7M params) |
+| **Domain** | Ohio family law, child custody, parental alienation |
+| **Training** | Kaggle 2×T4, 864 examples, 3 epochs, bf16 |
+| **Loss** | 1.01 → 0.84 (final eval: 0.91) |
+| **Version** | v0.1 (v0.2 in progress) |
+## Part of the Legal Agent Council
+OhioCustodyBERT is one of **4 models** in a multi-agent legal council architecture. All models are fine-tuned on the **identical curated Ohio family law corpus**, enabling consensus-based legal analysis where different architectures cross-validate each other.
+| Model | Role in Council |
+|---|---|
+| **OhioCustodyBERT** ← this model | Retriever + Verifier (classification, NER, fact-checking) |
+| Qwen3.5-9B (LoRA) | Specialist Agents (statute, case law, adversary) |
+| Qwen3-30B-A3B (LoRA) | Procedure Agent (sparse expert routing) |
+| Qwen3.5-27B-Claude-Opus-Distilled (Q8_0) | Synthesis Judge (Claude reasoning distilled, 28 GB) |
+**Principle**: Same data, different architectures = meaningful consensus. When all 4 models agree, confidence is high. When they disagree, the disagreement reveals where the law is ambiguous or the facts need closer examination.
+## Training Corpus (v0.2 — March 2026)
+### Court Opinions — 469 deduplicated, full-text opinions (14.9 MB)
+| Source | Count | Key Cases |
+|---|---|---|
+| Ohio Court of Appeals | 149 | PA, custody, GAL, DV, contempt, visitation |
+| Ohio Supreme Court | 199 | Shearer, Felton, In re Murray, In re Bonfield |
+| US Supreme Court | 121 | Santosky v. Kramer, Stanley v. Illinois, Troxel v. Granville |
+### SFT Training Data (audited March 10, 2026)
+| Dataset | Records | Status |
+|---|---|---|
+| v19 curated pairs | 916 train / 76 valid | ✅ Audited |
+| Opinion-generated pairs | ~1,400 (3 per opinion) | 🔄 Generating |
+| Phase 2 strategy pairs | 45 | ✅ Ready |
 ### Reference Materials
+- **941 records (4.0 MB)**: Black's Law Dictionary (591), OH Task Force on Family Law, Custody Evaluator Toolkit, GAL Task Force, Parenting Time Guide
+- **ORC Family Law (323 records, 1.2 MB)**: §3109, §2919, §2151, §3111, §3127, §3105
+- **55 PA references**: Academic papers, case analyses, legal guides
+## Intended Use
+- **Classification**: Custody outcome prediction, legal issue spotting
+- **NER**: Legal entity recognition (judges, statutes, case names)
+- **Retrieval**: Semantic search over Ohio family law corpus
+- **Fact-checking**: Citation verification for LLM-generated legal text
+- **Council agent**: BERT Retriever + Verifier roles in Legal Agent Council
+## Limitations
+- Trained primarily on Ohio law — may not generalize to other jurisdictions
+- v0.1 had limited training examples (864); v0.2 will substantially expand this
+- Not a substitute for legal counsel
+- Classification boundaries reflect Ohio statutory framework (R.C. 3109.04)
 ## Citation
 ```bibtex
 @misc{ohiocustodybert2026,
+  title={OhioCustodyBERT: Domain-Specific Legal Language Model for Ohio Family Law},
+  author={Roderick Mullins},
   year={2026},
+  howpublished={HuggingFace: Roderick3rd/OhioCustodyBERT}
 }
 ```
+## Related Resources
+- **Corpus**: [Roderick3rd/OhioCustodyBERT-corpus](https://huggingface.co/datasets/Roderick3rd/OhioCustodyBERT-corpus)
+- **Kaggle**: [ohio-legal-corpus](https://www.kaggle.com/datasets/phoenixintelligence/ohio-legal-corpus), [pa-references](https://www.kaggle.com/datasets/phoenixintelligence/pa-references)
+- **Council Architecture**: 6 agents, 4 models, sequential swap on M4 Pro 48GB