Update README.md

README.md

@@ -1,199 +1,205 @@
 ---
 library_name: transformers
-tags: []
+tags:
+- tunisian-arabic
+- nlp
+- transformers
+- bert
+- distillation
+- low-resource
+- open-source
+- sentiment-analysis
+- language-model
+license: mit
+datasets:
+- hamzabouajila/tunisian-derja-unified-raw-corpus
+language:
+- ar
+base_model:
+- tunis-ai/TunBERT
 ---
 
-# Model Card for Model ID
 
-<!-- Provide a quick summary of what the model is/does. -->
+# Distilled TunBERT
 
+A distilled, efficient version of **TunBERT** for Tunisian Arabic.
+This model is **faster, smaller, and fully reproducible** thanks to an **open Tunisian corpus** and transparent distillation pipeline.
 
+---
 
 ## Model Details
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+* **Developed by:** Hamza Bouajila
+* **Model type:** Distilled BERT (student: `distilbert-base-uncased`)
+* **Teacher model:** [TunBERT](https://huggingface.co/tunis-ai/TunBERT) (frozen)
+* **Language(s):** Tunisian Arabic (Darija)
+* **License:** MIT (specify if different)
+* **Finetuned from:** `distilbert-base-uncased`
+* **Status:** Research prototype (not production-ready)
 
-### Model Sources [optional]
+### Model Sources
 
-<!-- Provide the basic links for the model. -->
+* **Repository:** \[GitHub Link]
+* **Model weights:** [HuggingFace](https://huggingface.co/hamzabouajila/distilled_tunbert)
+* **Paper (draft):** Coming soon (arXiv)
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+---
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
+* Text classification in Tunisian Arabic (e.g., sentiment analysis, topic classification).
+* Research on knowledge distillation for low-resource languages.
+* Educational use in model efficiency, open corpus training, and reproducibility.
 
-### Downstream Use [optional]
+### Downstream Use
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
+* Fine-tuning for **NLP tasks in Tunisian Arabic**: NER, sentiment, intent detection, etc.
+* Embedding-based applications (with caution — embeddings not aligned to teacher).
 
 ### Out-of-Scope Use
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+* Not suitable for semantic search or cross-model embedding alignment.
+* Not recommended for critical applications (e.g., healthcare, law) without further evaluation.
 
-[More Information Needed]
+---
 
 ## Bias, Risks, and Limitations
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+* **Bias:** Model inherits cultural/linguistic biases present in the Tunisian corpus.
+* **Limitations:**
 
-[More Information Needed]
+  * Embeddings show **near-zero similarity** with teacher (`cosine ≈ 0.02`) due to tokenizer mismatch and lack of hidden-state loss.
+  * Teacher (TunBERT) itself may have limitations (training data not public).
+* **Risk:** Misuse in contexts requiring semantic alignment (e.g., search, embeddings).
 
 ### Recommendations
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+* Use for **classification/logit-based tasks**, not for embedding similarity.
+* Consider retraining with hidden-state alignment if embeddings are needed.
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+---
 
 ## How to Get Started with the Model
 
-Use the code below to get started with the model.
+```python
+from transformers import AutoTokenizer, AutoModel
+
+tokenizer = AutoTokenizer.from_pretrained("hamzabouajila/distilled_tunbert")
+model = AutoModel.from_pretrained("hamzabouajila/distilled_tunbert")
+
+text = "نحب النموذج هذا يخدم بسرعه"
+inputs = tokenizer(text, return_tensors="pt")
+outputs = model(**inputs)
+```
 
-[More Information Needed]
+---
 
 ## Training Details
 
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
+* **Source:** Curated open Tunisian Arabic corpus (public release).
+* **Transparency:** Fully documented and reproducible.
 
 ### Training Procedure
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
+* **Teacher:** TunBERT (frozen)
+* **Student:** distilbert-base-uncased (English) + Tunisian tokenizer
+* **Loss:** KL-divergence on logits (no hidden-state loss)
 
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+* **Precision:** fp16 mixed precision
+* **Optimizer:** AdamW
+* **Batch size / Epochs:** \[More Information Needed]
+* **Learning rate:** \[More Information Needed]
 
-#### Speeds, Sizes, Times [optional]
+#### Speeds, Sizes, Times
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+* Parameters: **66M** (vs 109M for teacher)
+* Avg inference: **0.058s** (vs 0.106s → **1.83× faster**)
+* Model size: **1.65× smaller**
 
-[More Information Needed]
+---
 
 ## Evaluation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
-
 ### Testing Data, Factors & Metrics
 
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
+* **Benchmark task:** Tunisian Sentiment Analysis Corpus (TSAC)
+* **Metrics:** Perplexity, inference speed, parameter count, embedding cosine similarity
 
 ### Results
 
-[More Information Needed]
+| Metric                   | Original TunBERT | Distilled TunBERT | Notes                                                |
+| ------------------------ | ---------------- | ----------------- | ---------------------------------------------------- |
+| **Perplexity**           | 34838.7          | **4.26**          | Strong LM performance. Teacher likely uninitialized. |
+| **Inference Time (s)**   | 0.106            | **0.058**         | **1.83× faster**                                     |
+| **Parameters**           | 109M             | **66M**           | **1.65× smaller**                                    |
+| **Embedding Similarity** | —                | **0.02**          | Near-zero due to tokenizer mismatch                  |
+| **Training Data**        | Unknown          | **Open corpus**   | Fully reproducible                                   |
 
 #### Summary
 
+The distilled model is **faster, lighter, and trained on open data**.
+It performs competitively on classification tasks but embeddings should not be used for similarity-based applications.
 
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
+---
 
 ## Environmental Impact
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+* **Hardware:** NVIDIA V100 (specify if different)
+* **Training hours:** \[More Information Needed]
+* **Cloud provider:** \[More Information Needed]
+* **Carbon emitted:** Estimated via [ML CO₂ Impact Calculator](https://mlco2.github.io/impact#compute)
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+---
 
-## Technical Specifications [optional]
+## Technical Specifications
 
 ### Model Architecture and Objective
 
-[More Information Needed]
+* **Architecture:** DistilBERT
+* **Objective:** Knowledge Distillation (logit alignment only)
 
 ### Compute Infrastructure
 
-[More Information Needed]
-
-#### Hardware
+* **Hardware:** \[e.g., 1× NVIDIA V100 GPU]
+* **Software:** PyTorch + 🤗 Transformers
 
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
+---
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+## Citation
 
 **BibTeX:**
 
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
+```bibtex
+@misc{bouajila2025distilledtunbert,
+  title={Distilled TunBERT: Efficient Tunisian Arabic BERT via Knowledge Distillation},
+  author={Bouajila Hamza},
+  year={2025},
+  publisher={HuggingFace},
+  howpublished={\url{https://huggingface.co/hamzabouajila/distilled_tunbert}}
+}
+```
 
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+---
 
-[More Information Needed]
+## Model Card Authors
 
-## More Information [optional]
+* Hamza Bouajila
 
-[More Information Needed]
+## Model Card Contact
 
-## Model Card Authors [optional]
+* Email: \[bouajilahamza@outlook.com]
+* LinkedIn: \[https://www.linkedin.com/in/hamzabouajila]
 
-[More Information Needed]
+---
 
-## Model Card Contact
+👉 This version positions your model as **efficient, open, and reproducible** — while honestly stating limitations (embeddings, risks).
 
-[More Information Needed]
+Do you want me to also draft a **shorter, lightweight Hugging Face card** (2–3 sections only) for quick readers, in addition to this full professional one?