Update README.md

README.md

@@ -17,11 +17,11 @@ base_model:
 - answerdotai/ModernBERT-base
 ---
 
-# Themida-ModernBERT Legal 21G: A Greek Legal Language Model with Advanced Optimization
+# GEM-ModernBERT Legal: A Greek Legal Language Model with Advanced Optimization
 
 ## Model Description
 
-**Themida-ModernBERT Legal 21G** is a ModernBERT-base model pre-trained from scratch on a strategically curated 21GB corpus of Greek legal, parliamentary, and governmental text. This model leverages ModernBERT's cutting-edge architectural innovations including **Flash Attention 2**, **StableAdamW optimizer**, **1024-token context length**, and **advanced memory optimization** techniques to deliver superior performance on Greek legal document understanding tasks.
+**GEM-ModernBERT Legal** is a ModernBERT-base model pre-trained from scratch on a strategically curated 21GB corpus of Greek legal, parliamentary, and governmental text. This model leverages ModernBERT's cutting-edge architectural innovations including **Flash Attention 2**, **StableAdamW optimizer**, **1024-token context length**, and **advanced memory optimization** techniques to deliver superior performance on Greek legal document understanding tasks.
 
 Building upon our proven **quality-based data repetition strategy**, this model incorporates ModernBERT's state-of-the-art training methodology with **30% masking probability**, **trapezoidal learning rate scheduling**, and **optimized batch sizing** for enhanced convergence and performance. The model is specifically designed to handle longer legal documents with its extended 1024-token context window while maintaining computational efficiency through advanced optimization techniques.
 
@@ -37,8 +37,8 @@ from transformers import pipeline
 # Load the model
 fill_mask = pipeline(
     "fill-mask",
-    model="novelcore/themida-modernbert-legal-21GB-1024",
-    tokenizer="novelcore/themida-modernbert-legal-21GB-1024"
+    model="novelcore/gem-modernbert-hq-legal",
+    tokenizer="novelcore/gem-modernbert-hq-legal"
 )
 
 # Example from a legal context with longer sequence support
@@ -55,8 +55,8 @@ For downstream tasks:
 from transformers import AutoTokenizer, AutoModelForSequenceClassification
 
 # For legal document classification with extended context
-tokenizer = AutoTokenizer.from_pretrained("novelcore/themida-modernbert-legal-21GB-1024")
-model = AutoModelForSequenceClassification.from_pretrained("novelcore/themida-modernbert-legal-21GB-1024")
+tokenizer = AutoTokenizer.from_pretrained("novelcore/gem-modernbert-hq-legal")
+model = AutoModelForSequenceClassification.from_pretrained("novelcore/gem-modernbert-hq-legal")
 
 # The model supports up to 1024 tokens for longer legal documents
 ```
@@ -203,138 +203,4 @@ Consistent with our previous models:
 1. **Highest quality sources** (legal dictionaries) repeated 4x
 2. **Medium-high quality sources** (court reports) repeated 3x  
 3. **Medium quality sources** (EU legal texts) repeated 2x
-4. **Lower quality sources** used once for diversity
-
-### Training Efficiency Improvements
-
-- **Faster Training**: 97h vs 146h (DeBERTa) despite longer sequences
-- **Better Convergence**: Optimized batch sizing and learning rate
-- **Memory Efficiency**: Advanced CUDA memory management
-- **Stable Training**: StableAdamW and conservative hyperparameters
-
-## Evaluation Results
-
-Comprehensive performance comparison across our model family:
-
-| Model | Architecture | Context | Training Loss | Eval Loss | Training Time | Vocab Size |
-| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
-| `Themida-ModernBERT Legal 21G` | ModernBERT-base | 1024 | 0.7648 | 0.7751 | 97h 9m | 50K |
-| `Themida-DeBERTa Legal 21G` | DeBERTa-base | 512 | 0.7913 | 0.7314 | 146h 13m | 128K |
-| `Themida-RoBERTa Legal 21G` | RoBERTa-base | 512 | 0.617 | 0.573 | 66h 39m | 50K |
-
-*Performance variations reflect different architectural designs and optimization strategies. Downstream task evaluations will be updated as results become available.*
-
-## Architecture Comparison: ModernBERT Advantages
-
-### Over RoBERTa
-- **2x Longer Context**: 1024 vs 512 tokens for complete document processing
-- **Flash Attention 2**: Memory-efficient processing of longer sequences
-- **Advanced Optimizer**: StableAdamW vs standard AdamW
-- **Optimized MLM**: 30% vs 15% masking probability
-
-### Over DeBERTa  
-- **Faster Training**: 97h vs 146h with comparable context understanding
-- **Memory Efficiency**: Better optimization for large-scale training
-- **Stable Convergence**: Conservative hyperparameters with reliable results
-- **Modern Optimizations**: Latest attention and memory management techniques
-
-### Unique ModernBERT Features
-- **Extended Context Processing**: Handle complete legal documents
-- **Memory Optimization**: Advanced CUDA memory management
-- **Training Stability**: StableAdamW with element-wise epsilon mode
-- **Attention Efficiency**: Flash Attention 2 with custom optimizations
-
-## Intended Uses
-
-### Primary Use Cases
-- **Long-form legal document analysis** (up to 1024 tokens)
-- **Complete contract processing** without truncation
-- **Parliamentary debate analysis** with full context
-- **Legal precedent identification** across extended text
-- **Regulatory compliance checking** with comprehensive document coverage
-- **Legal question answering** with enhanced context understanding
-
-### Enhanced Capabilities
-- **Full legal articles** processing without chunking
-- **Extended court decisions** analysis with complete reasoning
-- **Complex legislative texts** with multiple cross-references
-- **Parliamentary proceedings** with speaker continuity
-- **Legal research** with comprehensive document context
-
-### Optimal Use Cases for 1024-token Context
-- **Complete legal contracts** (most fit within 1024 tokens)
-- **Court decision summaries** with full reasoning
-- **Parliamentary speeches** and debates
-- **Legal article analysis** without truncation
-- **Regulatory text processing** with full context
-
-## Performance Advantages
-
-### Speed and Efficiency
-- **35% Faster Training**: 97h vs 146h (DeBERTa) with longer contexts
-- **Memory Optimization**: Advanced CUDA allocation for large batches
-- **Flash Attention 2**: Efficient processing of 1024-token sequences
-- **Stable Convergence**: Reliable training with conservative settings
-
-### Quality Improvements  
-- **Extended Context**: 2x longer sequences capture complete documents
-- **Better Representations**: 30% MLM probability for enhanced learning
-- **Stable Training**: StableAdamW optimizer with element-wise epsilon
-- **Optimized Architecture**: Modern attention mechanisms and memory management
-
-## Limitations and Considerations
-
-- The model may reflect biases present in Greek legal and governmental texts
-- Quality-based repetition may amplify biases from higher-quality sources
-- **Higher memory requirements** for inference due to 1024-token context
-- **Longer processing time** for extended sequences compared to 512-token models
-- Performance may degrade on informal or colloquial Greek text
-- Limited knowledge of legal concepts post-training data cutoff
-- Optimized specifically for Greek legal domain
-
-## Technical Specifications
-
-- **Model Size**: ~139M parameters
-- **Architecture**: ModernBERT-base with Flash Attention 2
-- **Context Length**: 1024 tokens (2x standard BERT models)
-- **Training Time**: 97 hours 9 minutes on 8x H100 80GB GPUs
-- **Effective Dataset Size**: 21.12GB (with quality-based repetition)
-- **Vocabulary Size**: 50,373 tokens
-- **Memory Requirements**: Optimized for H100 GPUs with advanced allocation
-- **Inference Speed**: Efficient with Flash Attention 2 optimizations
-
-## Deployment Recommendations
-
-### Hardware Requirements
-- **GPU Memory**: Minimum 24GB for inference with long sequences
-- **Optimal Hardware**: H100, A100, or modern GPUs with Flash Attention support
-- **Memory Configuration**: Use provided CUDA memory optimization settings
-
-### Performance Tuning
-- **Enable Flash Attention 2** for optimal performance
-- **Use BFloat16** precision for H100/A100 GPUs
-- **Configure memory allocation** using provided PYTORCH_CUDA_ALLOC_CONF
-- **Batch sizing**: Adjust based on available GPU memory
-
-## Model Card Authors
-
-[Your Name / Your Organization's Name]
-
-## Citation
-
-If you use this model in your research, please cite it as follows:
-
-```bibtex
-@misc{your_name_2025_themida_modernbert_21g,
-  author = {[Your Name/Organization]},
-  title = {Themida-ModernBERT Legal 21G: A Greek Legal Language Model with Advanced Optimization},
-  year = {2025},
-  publisher = {Hugging Face},
-  journal = {Hugging Face Hub},
-  howpublished = {\url{https://huggingface.co/novelcore/themida-modernbert-legal-21GB-1024}},
-}
-```
-
-## Acknowledgments
-
-We thank the Greek government institutions for making their legal texts publicly available, enabling the creation of this specialized language model. Special recognition goes to Answer.AI for the ModernBERT architecture and the open-source community for Flash Attention 2 and StableAdamW optimizations. This model represents the culmination of our research into optimal training strategies for Greek legal language understanding, combining proven data curation techniques with cutting-edge architectural innovations.
+4. **Lower quality sources** used once for diversity