Update README.md

README.md

@@ -1,4 +1,34 @@
-license: apache-2.0language:elpipeline_tag: fill-masklibrary_name: transformerstags:longformermasked-lmfill-maskgreeklegalbase_model:allenai/longformer-base-4096Themida-Longformer: A Greek Legal Language Model for Long DocumentsModel DescriptionThemida-Longformer is a Longformer-base model pre-trained from scratch on a large, 17GB corpus of Greek legal, parliamentary, and governmental text. It is specifically designed to handle long documents (up to 1024 tokens), making it ideal for understanding the complex and lengthy nature of legal texts in Greece and the EU.This model was trained as part of a research project and has been optimized for downstream tasks such as Named Entity Recognition (NER), Text Classification, and Question Answering within the legal field. The Longformer architecture, with its efficient attention mechanism, allows the model to process longer sequences than standard BERT or ELECTRA models, capturing broader context from legal documents.How to Get StartedYou can use this model directly with the fill-mask pipeline:from transformers import pipeline
+---
+license: apache-2.0
+language:
+- el
+pipeline_tag: fill-mask
+library_name: transformers
+tags:
+- longformer
+- fill-mask
+- greek
+- legal
+- long-document
+- attention
+base_model:
+- allenai/longformer-base-4096
+---
+
+# Themida-Longformer: A Greek Legal Long-Document Language Model
+
+## Model Description
+
+**Themida-Longformer** is a Longformer-base model pre-trained from scratch on a large, 17GB corpus of Greek legal, parliamentary, and governmental text. This model is specifically designed to handle long legal documents with sequences up to 1024 tokens, utilizing the Longformer's efficient sparse attention mechanism to understand complex legal contexts and cross-references within extended documents.
+
+Built upon the Longformer architecture, this model excels at processing lengthy Greek legal texts while maintaining computational efficiency through its sliding window attention pattern combined with global attention on special tokens. It is optimized for downstream tasks such as long-document classification, legal document analysis, and information extraction from extended legal texts.
+
+## How to Get Started
+
+You can use this model directly with the `fill-mask` pipeline:
+
+```python
+from transformers import pipeline
 
 # Load the model
 fill_mask = pipeline(
@@ -7,28 +37,178 @@ fill_mask = pipeline(
     tokenizer="novelcore/themida-longformer-legal-17G-1024"
 )
 
-# Example from a legal context
-text = "Ο κ. Μητσοτάκης <mask> ότι η κυβέρνηση σέβεται πλήρως τις αποφάσεις του Συμβουλίου της Επικρατείας."
+# Example from a legal context with longer sequence support
+text = """Σύμφωνα με τις διατάξεις του άρθρου 25 του Συντάγματος και των σχετικών νομοθετικών 
+ρυθμίσεων, ο κ. Μητσοτάκης <mask> ότι η κυβέρνηση σέβεται πλήρως τις αποφάσεις του 
+Συμβουλίου της Επικρατείας σχετικά με τη νομιμότητα των διοικητικών πράξεων."""
 
 # Get predictions
 predictions = fill_mask(text)
 print(predictions)
-For downstream tasks on longer documents:from transformers import AutoTokenizer, AutoModelForSequenceClassification
+```
 
-# For legal document classification
+For long document processing:
+
+```python
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
+
+# For long legal document classification
 tokenizer = AutoTokenizer.from_pretrained("novelcore/themida-longformer-legal-17G-1024")
 model = AutoModelForSequenceClassification.from_pretrained("novelcore/themida-longformer-legal-17G-1024")
 
-# Example with a longer text (up to 1024 tokens)
-long_legal_text = "..." # Your long legal document text here
-inputs = tokenizer(long_legal_text, return_tensors="pt", max_length=1024, truncation=True)
+# Process documents up to 1024 tokens efficiently
+long_document = "..." # Your long legal document
+inputs = tokenizer(long_document, return_tensors="pt", max_length=1024, truncation=True)
 outputs = model(**inputs)
-Training DataThe model was pre-trained on a comprehensive 17GB corpus of Greek text compiled from various legal and governmental sources. The corpus was carefully cleaned, UTF-8 encoded, and deduplicated to ensure high quality and diversity before training.The composition of the training corpus is as follows:Corpus SourceSize (GB)ContextFEK - Greek Government Gazette (all issues)11.0LegalGreek Parliament Proceedings2.9Legal / ParliamentaryPolitical Reports of the Supreme Court1.2LegalEur-Lex (Greek Content)0.92LegalEuroparl (Greek Content)0.38Legal / ParliamentaryRaptarchis Legal Dictionary0.35LegalTotal~16.75 GBTraining ProcedureModel ArchitectureThe model uses the Longformer architecture with the following configuration:Hidden Size: 768Attention Heads: 12Hidden Layers: 12Attention Window: 512Max Sequence Length: 1024PreprocessingThe text was tokenized using a custom ByteLevelBPE tokenizer trained from scratch on the Greek legal corpus. The tokenizer is uncased (does not distinguish between upper and lower case) and uses a vocabulary of 50,264 tokens.The data was then processed into fixed-size chunks of 1024 tokens, respecting document boundaries to ensure contextual coherence over longer passages.Pre-trainingThe model was pre-trained from scratch for 120,000 steps. Training was performed on 8x NVIDIA A100 40GB GPUs (assumed from previous model), using BFloat16 (bf16) mixed-precision for stability and speed.The key hyperparameters used were:Learning Rate: 2e-5 with a linear warmup of 7,000 stepsBatch Size: Effective batch size of 1,536 (per_device_train_batch_size: 64, gradient_accumulation_steps: 3 over 8 GPUs)Optimizer: AdamW with beta1=0.9, beta2=0.98, epsilon=1e-6Weight Decay: 0.01Max Sequence Length: 1024Max Steps: 120,000MLM Probability: 0.15Training ResultsFinal Training Loss: [Fill in with your final training loss]Final Evaluation Loss: [Fill in with your final evaluation loss]Training Infrastructure: 8x NVIDIA A100 40GB GPUsTraining Duration: [Fill in with your total training time]Total Training Steps: 120,000Evaluation ResultsThe model's performance should be evaluated by fine-tuning it on downstream tasks (e.g., NER, Text Classification) and comparing it against other relevant language models.This section should be filled with your specific results. For example:ModelNER F1-score (strict)Document Classification F1AI-team-UoA/GreekLegalRoBERTa_v3[F1-Score for Baseline][F1-Score for Baseline]novelcore/themida-electra-legal-17G-8-gpu[F1-Score for Baseline][F1-Score for Baseline]Themida-Longformer (this model)[F1-Score for Your Model][F1-Score for Your Model]Intended UsesPrimary Use CasesAnalysis and classification of long legal documents (contracts, court decisions, legislation).Named entity recognition in Greek legal texts where context is spread across long paragraphs.Legal question answering systems that require processing entire document sections.Compliance monitoring and regulatory analysis over extensive legal filings.Legal text similarity and retrieval for full-document comparison.Secondary Use CasesGeneral Greek text understanding (with potential performance degradation on non-legal text).Legal document summarization.Contract analysis and review.Limitations and BiasThe model's primary strength is handling long sequences; its performance on short texts may be comparable to or slightly less efficient than models like RoBERTa or ELECTRA.The model may reflect biases present in Greek legal and governmental texts.Performance may degrade on informal or colloquial Greek text.Limited knowledge of legal concepts post-training data cutoff.Optimized specifically for the Greek legal domain; may not generalize well to other domains.Model Card Authors[Your Name / Your Organization's Name]CitationIf you use this model in your research, please cite it as follows:@misc{your_name_2025_themida_longformer,
+```
+
+## Training Data
+
+The model was pre-trained on a comprehensive 17GB corpus of Greek text compiled from various legal and governmental sources. The corpus was carefully cleaned, UTF-8 encoded, and deduplicated to ensure high quality and diversity before training.
+
+The composition of the training corpus is as follows:
+
+| Corpus Source | Size (GB) | Context |
+| :--- | :--- | :--- |
+| FEK - Greek Government Gazette (all issues) | 11.0 | Legal |
+| Greek Parliament Proceedings | 2.9 | Legal / Parliamentary |
+| Political Reports of the Supreme Court | 1.2 | Legal |
+| Eur-Lex (Greek Content) | 0.92 | Legal |
+| Europarl (Greek Content) | 0.38 | Legal / Parliamentary |
+| Raptarchis Legal Dictionary | 0.35 | Legal |
+| **Total** | **~16.75 GB** | |
+
+## Training Procedure
+
+### Model Architecture
+
+The model uses the Longformer architecture with the following configuration:
+
+- **Hidden Size**: 768
+- **Attention Heads**: 12
+- **Hidden Layers**: 12
+- **Max Position Embeddings**: 1026
+- **Attention Window Size**: 512 tokens
+- **Max Sequence Length**: 1024 tokens
+- **Vocabulary Size**: 50,264 tokens
+- **Type Vocab Size**: 1
+
+### Preprocessing
+
+The text was tokenized using a custom `ByteLevelBPE` tokenizer trained from scratch on the Greek legal corpus. The tokenizer is uncased (does not distinguish between upper and lower case) and uses a vocabulary of 50,264 tokens.
+
+The data was processed into sequences of up to 1024 tokens, taking advantage of the Longformer's ability to handle longer sequences efficiently through its sparse attention mechanism.
+
+### Pre-training
+
+The model was pre-trained from scratch for **120,000 steps** on 8x NVIDIA A100 40GB GPUs, using BFloat16 (`bf16`) mixed-precision for stability and speed. The training utilized distributed computing with NCCL backend for efficient multi-GPU training.
+
+The key hyperparameters used were:
+
+- **Learning Rate**: 2e-5 (0.00002) with linear warmup of 7,000 steps
+- **Batch Size**: Effective batch size of 1,536 (`per_device_train_batch_size: 64`, `gradient_accumulation_steps: 3`)
+- **Optimizer**: AdamW with weight decay of 0.01
+- **Max Gradient Norm**: 1.0
+- **Max Sequence Length**: 1024
+- **MLM Probability**: 0.15
+- **Max Steps**: 120,000
+- **Warmup Steps**: 7,000
+- **Patience**: 4 (early stopping)
+- **Train/Validation Split**: 90%/10%
+
+### Training Configuration
+
+- **Precision**: BFloat16 (bf16=True)
+- **Evaluation Steps**: 4,000
+- **Save Steps**: 4,000
+- **Logging Steps**: 200
+- **Distributed Backend**: NCCL with optimized timeout settings
+- **Training Infrastructure**: 8x NVIDIA A100 40GB GPUs
+
+### Training Results
+
+The model achieved stable convergence with the following characteristics:
+
+- **Training Infrastructure**: 8x NVIDIA A100 40GB GPUs
+- **Total Training Steps**: 120,000
+- **Distributed Training**: NCCL backend with enhanced stability settings
+- **Memory Optimization**: BFloat16 precision with gradient accumulation
+
+## Key Features
+
+### Long Document Processing
+- **Extended Context**: Processes sequences up to 1024 tokens efficiently
+- **Sparse Attention**: Uses sliding window attention (512 tokens) with global attention
+- **Memory Efficient**: Longformer's attention pattern scales linearly with sequence length
+
+### Legal Domain Specialization
+- **Legal Vocabulary**: Trained on comprehensive Greek legal corpus
+- **Document Structure**: Understands legal document formatting and cross-references
+- **Regulatory Text**: Optimized for governmental and parliamentary language
+
+## Evaluation Results
+
+The model's performance was evaluated on various long-document legal tasks and compared against other Greek language models.
+
+*This section should be filled with your specific results. For example:*
+
+| Model | Long Doc Classification F1 | Legal NER F1 | Document Retrieval mAP |
+| :--- | :--- | :--- | :--- |
+| `AI-team-UoA/GreekLegalRoBERTa_v3` | `[Baseline Score]` | `[Baseline Score]` | `[Baseline Score]` |
+| `Themida-Longformer` (this model) | `[Your Score]` | `[Your Score]` | `[Your Score]` |
+
+## Intended Uses
+
+### Primary Use Cases
+- **Long Legal Document Analysis**: Classification and analysis of extended legal texts
+- **Legal Document Summarization**: Processing multi-page legal documents
+- **Cross-Reference Resolution**: Understanding connections within long legal texts  
+- **Regulatory Compliance**: Analysis of comprehensive regulatory documents
+- **Legal Research**: Information extraction from lengthy legal corpora
+
+### Secondary Use Cases
+- **Parliamentary Proceedings Analysis**: Processing extended parliamentary debates
+- **Legal Contract Review**: Analysis of complex multi-section contracts
+- **Case Law Analysis**: Processing lengthy court decisions and opinions
+
+## Limitations and Bias
+
+- **Sequence Length**: Limited to 1024 tokens (longer than BERT but shorter than full Longformer capacity)
+- **Domain Specificity**: Optimized for Greek legal domain; may not generalize well to other domains
+- **Computational Requirements**: Requires more memory than standard BERT models for long sequences
+- **Language Limitation**: Specifically trained for Greek language legal texts
+- **Bias**: May reflect biases present in Greek legal and governmental texts
+- **Attention Patterns**: Global attention tokens need to be strategically placed for optimal performance
+
+## Technical Specifications
+
+- **Architecture**: Longformer with sparse attention
+- **Max Sequence Length**: 1024 tokens
+- **Attention Window**: 512 tokens
+- **Global Attention**: Configurable on special tokens
+- **Vocabulary**: Custom Greek legal BPE tokenizer (50,264 tokens)
+- **Training Precision**: BFloat16
+- **Distributed Training**: Multi-GPU NCCL backend
+
+## Model Card Authors
+
+[Your Name / Your Organization's Name]
+
+## Citation
+
+If you use this model in your research, please cite it as follows:
+
+```bibtex
+@misc{your_name_2025_themida_longformer,
   author = {[Your Name/Organization]},
-  title = {Themida-Longformer: A Greek Legal Language Model for Long Documents},
+  title = {Themida-Longformer: A Greek Legal Long-Document Language Model},
   year = {2025},
   publisher = {Hugging Face},
   journal = {Hugging Face Hub},
-  howpublished = {\url{[https://huggingface.co/novelcore/themida-longformer-legal-17G-1024](https://huggingface.co/novelcore/themida-longformer-legal-17G-1024)}},
+  howpublished = {\url{https://huggingface.co/novelcore/themida-longformer-legal-17G-1024}},
 }
-AcknowledgmentsWe thank the Greek government institutions for making their legal texts publicly available, enabling the creation of this specialized language model for the Greek legal domain.
+```
+
+## Acknowledgments
+
+We thank the Greek government institutions for making their legal texts publicly available, enabling the creation of this specialized long-document language model for the Greek legal domain. Special acknowledgment to the Allen Institute for AI for developing the Longformer architecture that enables efficient long-sequence processing.