MIGRATION_TO_DEBERTA.md

Migration from Hierarchical BERT to DeBERTa-base

Summary

Successfully migrated the codebase from using BERT-base-uncased to DeBERTa-base (microsoft/deberta-base).

Changes Made

1. Configuration (config.py)

• Changed model name: bert_model_name from "bert-base-uncased" to "microsoft/deberta-base"
• Updated documentation: References to "Legal-BERT" updated to "Legal-DeBERTa"

2. Model Architecture (model.py)

• Updated imports and docstrings: Changed references from BERT to DeBERTa
• Modified forward pass: DeBERTa doesn't have pooler_output like BERT. Changed to use last_hidden_state[:, 0, :] (CLS token) instead
• Updated both model classes:
  • FullyLearningBasedLegalBERT: Now uses DeBERTa
  • HierarchicalLegalBERT: Now uses DeBERTa hierarchically
• Fixed tokenizer: Default model changed to "microsoft/deberta-base"
• Dynamic hidden size: Model now gets hidden size from config (still 768 for DeBERTa-base)

3. Training Scripts (train.py, trainer.py)

• Updated documentation and print statements to reference DeBERTa instead of BERT

Key Technical Differences

BERT vs DeBERTa

Feature	BERT	DeBERTa
Model	bert-base-uncased	microsoft/deberta-base
Hidden Size	768	768
Pooler Output	✅ Available	❌ Not available
CLS Token	outputs.pooler_output	outputs.last_hidden_state[:, 0, :]
Attention	Standard	Disentangled attention

Why DeBERTa?

1. Improved Performance: DeBERTa uses disentangled attention mechanism
2. Better Context Understanding: Position-aware attention
3. State-of-the-art: Generally outperforms BERT on many benchmarks

No Breaking Changes

• ✅ Model architecture remains the same (hierarchical structure intact)
• ✅ Training pipeline unchanged
• ✅ All multi-task heads (classification, severity, importance) work as before
• ✅ Loss functions and optimization unchanged
• ✅ Data loading and preprocessing unchanged

Next Steps

Before Training

1. Ensure transformers library is up to date:

   pip install --upgrade transformers
   

2. The first training run will download DeBERTa-base model (~360MB)

Training

Simply run your existing training command:

python train.py --epochs 20 --batch-size 16

The model will automatically:

• Download DeBERTa-base from Hugging Face
• Use the hierarchical architecture with DeBERTa as encoder
• Save checkpoints with DeBERTa weights

Model Compatibility

• Old BERT checkpoints will NOT be compatible with new DeBERTa model
• You'll need to retrain from scratch
• This is expected and necessary when changing the base encoder

Files Modified

1. ✅ config.py - Model name and documentation
2. ✅ model.py - Model architecture and forward pass
3. ✅ train.py - Training script documentation
4. ✅ trainer.py - Trainer documentation

Files NOT Modified (still work as-is)

• data_loader.py - No changes needed
• evaluate.py - Works with new model
• inference.py - Works with new model
• risk_discovery.py - Independent of encoder choice
• All other utility files

Performance Expectations

DeBERTa should provide:

• Similar or better accuracy on risk classification
• Better handling of legal text nuances
• Potentially faster convergence during training