Add IndicBERT methodology credit and acknowledgments
Browse files
README.md
CHANGED
|
@@ -238,6 +238,42 @@ Family law involves sensitive matters affecting children and families. Any appli
|
|
| 238 |
|
| 239 |
---
|
| 240 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 241 |
## License
|
| 242 |
|
| 243 |
Released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0), consistent with the base ModernBERT model.
|
|
|
|
| 238 |
|
| 239 |
---
|
| 240 |
|
| 241 |
+
## Acknowledgments & Methodology Credits
|
| 242 |
+
|
| 243 |
+
Our domain-adaptive pre-training approach draws inspiration from several key works:
|
| 244 |
+
|
| 245 |
+
### AI4Bharat / IndicBERT
|
| 246 |
+
|
| 247 |
+
Our training methodology follows the **domain-adaptive MLM fine-tuning** paradigm demonstrated by [AI4Bharat's IndicBERT](https://huggingface.co/ai4bharat/indic-bert) project. Key parallels:
|
| 248 |
+
|
| 249 |
+
| IndicBERT | OhioCustodyBERT |
|
| 250 |
+
|-----------|-----------------|
|
| 251 |
+
| Pre-trained ALBERT on 8.9B tokens across 12 Indian languages | Fine-tuned ModernBERT on Ohio custody case corpus |
|
| 252 |
+
| Masked Language Modeling (MLM) objective | Masked Language Modeling (MLM) objective |
|
| 253 |
+
| Domain-specific corpus curation | Domain-specific corpus curation (legal/custody) |
|
| 254 |
+
| Curriculum learning phases | Planned multi-phase training (v0.2+) |
|
| 255 |
+
| Efficient training with limited compute | Trained on free-tier Kaggle 2×T4 GPUs |
|
| 256 |
+
|
| 257 |
+
IndicBERT demonstrated that **focused domain adaptation on a well-curated corpus** can outperform larger general-purpose models — a principle we apply to the legal domain. Their curriculum learning approach (foundation → high-resource → low-resource) will inform our planned training phases:
|
| 258 |
+
|
| 259 |
+
1. **Phase 1** (current): Broad legal language modeling on custody cases
|
| 260 |
+
2. **Phase 2** (planned): Expanded corpus with CourtListener + GovInfo federal law
|
| 261 |
+
3. **Phase 3** (planned): Ohio-specific statutes and recent slip opinions
|
| 262 |
+
|
| 263 |
+
```bibtex
|
| 264 |
+
@inproceedings{kakwani2020indicnlpsuite,
|
| 265 |
+
title={{IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages}},
|
| 266 |
+
author={Divyanshu Kakwani and Anoop Kunchukuttan and Satber Gopi and Gokul N.C. and Avik Bhattacharyya and Mitesh M. Khapra and Pratyush Kumar},
|
| 267 |
+
year={2020},
|
| 268 |
+
booktitle={Findings of EMNLP},
|
| 269 |
+
}
|
| 270 |
+
```
|
| 271 |
+
|
| 272 |
+
### ModernBERT (Answer.AI)
|
| 273 |
+
|
| 274 |
+
Our base model, [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base), provides the architectural foundation with modern improvements including RoPE, Flash Attention, and 8,192-token context. See citation in the [Citation](#citation) section.
|
| 275 |
+
|
| 276 |
+
|
| 277 |
## License
|
| 278 |
|
| 279 |
Released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0), consistent with the base ModernBERT model.
|