Roderick3rd commited on
Commit
4fd67f9
·
verified ·
1 Parent(s): 1654304

Add IndicBERT methodology credit and acknowledgments

Browse files
Files changed (1) hide show
  1. README.md +36 -0
README.md CHANGED
@@ -238,6 +238,42 @@ Family law involves sensitive matters affecting children and families. Any appli
238
 
239
  ---
240
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
241
  ## License
242
 
243
  Released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0), consistent with the base ModernBERT model.
 
238
 
239
  ---
240
 
241
+ ## Acknowledgments & Methodology Credits
242
+
243
+ Our domain-adaptive pre-training approach draws inspiration from several key works:
244
+
245
+ ### AI4Bharat / IndicBERT
246
+
247
+ Our training methodology follows the **domain-adaptive MLM fine-tuning** paradigm demonstrated by [AI4Bharat's IndicBERT](https://huggingface.co/ai4bharat/indic-bert) project. Key parallels:
248
+
249
+ | IndicBERT | OhioCustodyBERT |
250
+ |-----------|-----------------|
251
+ | Pre-trained ALBERT on 8.9B tokens across 12 Indian languages | Fine-tuned ModernBERT on Ohio custody case corpus |
252
+ | Masked Language Modeling (MLM) objective | Masked Language Modeling (MLM) objective |
253
+ | Domain-specific corpus curation | Domain-specific corpus curation (legal/custody) |
254
+ | Curriculum learning phases | Planned multi-phase training (v0.2+) |
255
+ | Efficient training with limited compute | Trained on free-tier Kaggle 2×T4 GPUs |
256
+
257
+ IndicBERT demonstrated that **focused domain adaptation on a well-curated corpus** can outperform larger general-purpose models — a principle we apply to the legal domain. Their curriculum learning approach (foundation → high-resource → low-resource) will inform our planned training phases:
258
+
259
+ 1. **Phase 1** (current): Broad legal language modeling on custody cases
260
+ 2. **Phase 2** (planned): Expanded corpus with CourtListener + GovInfo federal law
261
+ 3. **Phase 3** (planned): Ohio-specific statutes and recent slip opinions
262
+
263
+ ```bibtex
264
+ @inproceedings{kakwani2020indicnlpsuite,
265
+ title={{IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages}},
266
+ author={Divyanshu Kakwani and Anoop Kunchukuttan and Satber Gopi and Gokul N.C. and Avik Bhattacharyya and Mitesh M. Khapra and Pratyush Kumar},
267
+ year={2020},
268
+ booktitle={Findings of EMNLP},
269
+ }
270
+ ```
271
+
272
+ ### ModernBERT (Answer.AI)
273
+
274
+ Our base model, [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base), provides the architectural foundation with modern improvements including RoPE, Flash Attention, and 8,192-token context. See citation in the [Citation](#citation) section.
275
+
276
+
277
  ## License
278
 
279
  Released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0), consistent with the base ModernBERT model.