Update README.md

README.md

@@ -1,199 +1,202 @@
 ---
+license: apache-2.0
+language:
+- el
+pipeline_tag: fill-mask
 library_name: transformers
-tags: []
+tags:
+- electra
+- fill-mask
+- greek
+- legal
+- discriminator
+- generator
+base_model:
+- google/electra-base-discriminator
 ---
 
-# Model Card for Model ID
+# Themida-ELECTRA v2: A Greek Legal Language Model
 
-<!-- Provide a quick summary of what the model is/does. -->
+## Model Description
 
+**Themida-ELECTRA v2** is an improved ELECTRA-base model pre-trained from scratch on a large, 17GB corpus of Greek legal, parliamentary, and governmental text. This second version incorporates refined training hyperparameters for enhanced performance and stability. It is designed for understanding the complex vocabulary and context of the legal domain in Greece and the EU.
 
+This model was trained as part of a research project and has been optimized for downstream tasks such as Named Entity Recognition (NER), Text Classification, and Question Answering within the legal field. The ELECTRA architecture provides more efficient pre-training compared to masked language models like BERT by using a generator-discriminator approach.
 
-## Model Details
+## How to Get Started
 
-### Model Description
+You can use this model directly with the `fill-mask` pipeline:
 
-<!-- Provide a longer summary of what this model is. -->
+```python
+from transformers import pipeline
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+# Load the model
+fill_mask = pipeline(
+    "fill-mask",
+    model="novelcore/themida-electra-legal-17G-8-gpu-v2",
+    tokenizer="novelcore/themida-electra-legal-17G-8-gpu-v2"
+)
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+# Example from a legal context
+text = "Ο κ. Μητσοτάκης <mask> ότι η κυβέρνηση σέβεται πλήρως τις αποφάσεις του Συμβουλίου της Επικρατείας."
 
-### Model Sources [optional]
+# Get predictions
+predictions = fill_mask(text)
+print(predictions)
 
-<!-- Provide the basic links for the model. -->
+# Get predictions
+predictions = fill_mask(text)
+[{'score': 0.20120874047279358,
+  'token': 4014,
+  'token_str': ' ειπε',
+  'sequence': ' ο κ . μητσοτακης ειπε οτι η κυβερνηση σεβεται πληρως τις αποφασεις του συμβουλιου της επικρατειας .'},
+ {'score': 0.19406235218048096,
+  'token': 12702,
+  'token_str': ' δηλωσε',
+  'sequence': ' ο κ . μητσοτακης δηλωσε οτι η κυβερνηση σεβεται πληρως τις αποφασεις του συμβουλιου της επικρατειας .'},
+ {'score': 0.18023167550563812,
+  'token': 11151,
+  'token_str': ' δηλωνει',
+  'sequence': ' ο κ . μητσοτακης δηλωνει οτι η κυβερνηση σεβεται πληρως τις αποφασεις του συμβουλιου της επικρατειας .'},
+ {'score': 0.08440685272216797,
+  'token': 8534,
+  'token_str': ' υποστηριζει',
+  'sequence': ' ο κ . μητσοτακης υποστηριζει οτι η κυβερνηση σεβεται πληρως τις αποφασεις του συμβουλιου της επικρατειας .'},
+ {'score': 0.05247046798467636,
+  'token': 3523,
+  'token_str': ' λεει',
+  'sequence': ' ο κ . μητσοτακης λεει οτι η κυβερνηση σεβεται πληρως τις αποφασεις του συμβουλιου της επικρατειας .'}]
+```
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+For downstream tasks:
 
-## Uses
+```python
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+# For legal document classification
+tokenizer = AutoTokenizer.from_pretrained("novelcore/themida-electra-legal-17G-8-gpu-v2")
+model = AutoModelForSequenceClassification.from_pretrained("novelcore/themida-electra-legal-17G-8-gpu-v2")
+```
 
-### Direct Use
+## Training Data
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+The model was pre-trained on a comprehensive 17GB corpus of Greek text compiled from various legal and governmental sources. The corpus was carefully cleaned, UTF-8 encoded, and deduplicated to ensure high quality and diversity before training.
 
-[More Information Needed]
+The composition of the training corpus is as follows:
 
-### Downstream Use [optional]
+| Corpus Source | Size (GB) | Context |
+| :--- | :--- | :--- |
+| FEK - Greek Government Gazette (all issues) | 11.0 | Legal |
+| Greek Parliament Proceedings | 2.9 | Legal / Parliamentary |
+| Political Reports of the Supreme Court | 1.2 | Legal |
+| Eur-Lex (Greek Content) | 0.92 | Legal |
+| Europarl (Greek Content) | 0.38 | Legal / Parliamentary |
+| Raptarchis Legal Dictionary | 0.35 | Legal |
+| **Total** | **~16.75 GB** | |
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+## Training Procedure
 
-[More Information Needed]
+### Model Architecture
 
-### Out-of-Scope Use
+The model uses the ELECTRA architecture with the following configuration:
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+- **Discriminator Hidden Size**: 768
+- **Discriminator Attention Heads**: 12
+- **Discriminator Hidden Layers**: 12
+- **Generator Size Fraction**: 0.25 (192 hidden size generator)
 
-[More Information Needed]
+### Preprocessing
 
-## Bias, Risks, and Limitations
+The text was tokenized using a custom `ByteLevelBPE` tokenizer trained from scratch on the Greek legal corpus. The tokenizer is uncased (does not distinguish between upper and lower case) and uses a vocabulary of 50,264 tokens.
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+The data was then processed into fixed-size chunks of 512 tokens, respecting document boundaries to ensure contextual coherence.
 
-[More Information Needed]
+### Pre-training
 
-### Recommendations
+The model was pre-trained from scratch for **200,000 steps** on 8x NVIDIA A100 40GB GPUs, using BFloat16 (`bf16`) mixed-precision for stability and speed. This second version incorporates improved hyperparameters for enhanced convergence and performance.
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+The key hyperparameters used were:
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+- **Learning Rate**: 1e-4 (0.0001) with a linear warmup of 12,000 steps
+- **Batch Size**: Effective batch size of 3,840 (`per_device_train_batch_size: 60`, `gradient_accumulation_steps: 8`)
+- **Optimizer**: AdamW with `beta1=0.9`, `beta2=0.98`, `epsilon=1e-6`
+- **Weight Decay**: 0.01
+- **Max Sequence Length**: 512
+- **Max Steps**: 200,000
+- **Warmup Steps**: 12,000
+- **Generator Loss Weight**: 50.0
+- **Discriminator Loss Weight**: 50.0
 
-## How to Get Started with the Model
+### Training Results
 
-Use the code below to get started with the model.
+The model achieved the following performance metrics:
 
-[More Information Needed]
+- **Final Training Loss**: 0.0056
+- **Final Evaluation Loss**: 0.0054
+- **Training Infrastructure**: 8x NVIDIA A100 40GB GPUs
+- **Total Training Steps**: 200,000
 
-## Training Details
+### Improvements in v2
 
-### Training Data
+This second version incorporates the following improvements over the initial model:
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+- **Reduced Learning Rate**: Lowered from 8e-4 to 1e-4 for more stable convergence
+- **Extended Training**: Increased from 120,000 to 200,000 steps for better performance
+- **Enhanced Warmup**: Extended warmup period from 6,000 to 12,000 steps for smoother training initialization
 
-[More Information Needed]
+## Evaluation Results
 
-### Training Procedure
+The model's performance was evaluated by fine-tuning it on downstream Named Entity Recognition (NER) tasks and comparing it against other legal language models.
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+*This section should be filled with your specific results. For example:*
 
-#### Preprocessing [optional]
+| Model | NER F1-score (strict) |
+| :--- | :--- |
+| `AI-team-UoA/GreekLegalRoBERTa_v3` | `[F1-Score for Baseline]` |
+| `Themida-ELECTRA v1` | `[F1-Score for v1]` |
+| `Themida-ELECTRA v2` (this model) | `[F1-Score for v2]` |
 
-[More Information Needed]
+## Intended Uses
 
+### Primary Use Cases
+- Legal document analysis and classification
+- Named entity recognition in Greek legal texts
+- Legal question answering systems
+- Compliance monitoring and regulatory analysis
+- Legal text similarity and retrieval
 
-#### Training Hyperparameters
+### Secondary Use Cases
+- General Greek text understanding (with potential performance degradation)
+- Legal document summarization
+- Contract analysis and review
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+## Limitations and Bias
 
-#### Speeds, Sizes, Times [optional]
+- The model may reflect biases present in Greek legal and governmental texts
+- Performance may degrade on informal or colloquial Greek text
+- Limited knowledge of legal concepts post-training data cutoff
+- Optimized specifically for Greek legal domain; may not generalize well to other domains
+- The ELECTRA architecture may require different fine-tuning approaches compared to BERT-like models
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+## Model Card Authors
 
-[More Information Needed]
+[Your Name / Your Organization's Name]
 
-## Evaluation
+## Citation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+If you use this model in your research, please cite it as follows:
 
-### Testing Data, Factors & Metrics
+```bibtex
+@misc{your_name_2025_themida_electra_v2,
+  author = {[Your Name/Organization]},
+  title = {Themida-ELECTRA v2: A Greek Legal Language Model},
+  year = {2025},
+  publisher = {Hugging Face},
+  journal = {Hugging Face Hub},
+  howpublished = {\url{https://huggingface.co/[Your Username]/[Your Model Name]}},
+}
+```
 
-#### Testing Data
+## Acknowledgments
 
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
-
-## Model Card Contact
-
-[More Information Needed]
+We thank the Greek government institutions for making their legal texts publicly available, enabling the creation of this specialized language model for the Greek legal domain.