Copy from GemMaroc/GemMaroc-27b-it

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,155 @@
+---
+library_name: transformers
+tags:
+- MoroccanArabic
+- Darija
+- GemMaroc
+- DarijaLLM
+- conversational
+pipeline_tag: text-generation
+datasets:
+- GemMaroc/TULU-3-50k-darija-english
+language:
+- ar
+- ary
+- en
+base_model:
+- google/gemma-3-27b-it
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+# GemMaroc‑27B
+
+Unlocking **Moroccan Darija** proficiency in a state‑of‑the‑art large language model, trained with a *minimal‑data, green‑AI* recipe that preserves Gemma‑27B’s strong reasoning abilities while adding fluent Darija generation.
+
+---
+
+## Model at a glance
+
+|                     | Details                                                                                                                       |
+| ------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| **Model ID**        | `AbderrahmanSkiredj1/GemMaroc-27b-it`                                                                                         |
+| **Base model**      | [`google/gemma-3-27b`](https://huggingface.co/google/gemma-3-27b)                                                             |
+| **Architecture**    | Decoder‑only Transformer (Gemma 3)                                                                                            |
+| **Parameters**      | 27 billion                                                                                                                    |
+| **Context length**  | 2 048 tokens                                                                                                                  |
+| **Training regime** | Supervised fine‑tuning (LoRA → merged) on 50 K high‑quality Darija/English instructions TULU‑50K slice |
+| **Compute budget**  | 48 GPU·h (8 × H100‑80GB × 6 h) – ≈ 26 kWh / 10 kg CO₂e                                                                        |
+| **License**         | Apache 2.0                                                                                                                    |
+
+---
+
+## Why another Darija model?
+
+* **Inclusive AI** > 36 million speakers of Moroccan Arabic remain underserved by open LLMs.
+* **Quality‑over‑quantity** A carefully curated 50 K instruction set surfaces Darija competence without sacrificing cross‑lingual reasoning.
+* **Green AI** GemMaroc achieves Atlas‑Chat‑level Darija scores using < 2 % of the energy.
+
+---
+
+## Benchmark summary
+
+| Model            | Darija MMLU | Darija HellaSwag | GSM8K @5   | HellaSwag (EN) |
+| ---------------- | ----------- | ---------------- | ---------- | -------------- |
+| Atlas‑Chat‑27B   | **61.9 %**  | 48.4 %           | 82.0 %     | 77.8 %         |
+| **GemMaroc‑27B** | 61.6 %      | **60.5 %**       | **84.2 %** | **79.3 %**     |
+
+<sub>Zero‑shot accuracy; full table in the paper.</sub>
+
+---
+
+## Quick start
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
+
+model_id = "AbderrahmanSkiredj1/GemMaroc-27b-it"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model     = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype="auto",
+    device_map="auto"
+)
+
+pipe = pipeline(
+    "text-generation",
+    model=model,
+    tokenizer=tokenizer,
+    device_map="auto",
+    max_new_tokens=1024,
+    temperature=0.7,
+    repetition_penalty=1.2,
+    no_repeat_ngram_size=3,
+)
+
+messages = [
+    {"role": "user", "content": "شنو هي نظرية ‘butterfly effect’؟ فسّرها بدارجة ونقّط مثال بسيط."}
+]
+
+prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+print(pipe(prompt)[0]["generated_text"][len(prompt):])
+```
+
+### Chat template (Gemma 3 format)
+
+The tokenizer provides a baked‑in Jinja template that starts with a **begin‑of‑sequence** token (`<bos>`), then alternates user/model turns, each wrapped by `<start_of_turn>` … `<end_of_turn>` markers. When you set `add_generation_prompt=True` it ends after the opening model tag so the model can continue:
+
+```
+<bos><start_of_turn>user
+{user message}<end_of_turn>
+<start_of_turn>model
+```
+
+The assistant will keep generating tokens until it decides to emit `<end_of_turn>`.
+
+```python
+prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
+```
+
+No manual token juggling required—the call above handles BOS, turn delimiters, and newline placement automatically.
+
+---
+
+Pre‑quantised checkpoints will be published under the same repo tags (`gemmaroc‑27b‑awq‑int4`, `gemmaroc‑27b‑gguf‑q4_k_m`).
+
+---
+
+## Training recipe (one‑paragraph recap)
+
+1. **Data** Translate a 44 K reasoning slice of TULU 50K into Darija, keeping 20 % English for cross‑lingual robustness.
+2. **LoRA SFT** Rank 16, α = 32, 3 epochs, bf16, context 2 048.
+3. **Merge & push** Merge LoRA into base weights (`peft.merge_and_unload`), convert to safetensors, upload.
+
+---
+
+## Limitations & ethical considerations
+
+* Sentiment and abstractive summarisation still trail state‑of‑the‑art.
+* Tokeniser is unchanged; rare Darija spellings may fragment.
+* Model may inherit societal biases present in pre‑training data.
+* No RLHF / RLAIF safety alignment yet – apply a moderation layer in production.
+
+---
+
+## Citation
+
+If you use GemMaroc in your work, please cite:
+
+```bibtex
+@misc{skiredj2025gemmarocunlockingdarijaproficiency,
+      title={GemMaroc: Unlocking Darija Proficiency in LLMs with Minimal Data}, 
+      author={Abderrahman Skiredj and Ferdaous Azhari and Houdaifa Atou and Nouamane Tazi and Ismail Berrada},
+      year={2025},
+      eprint={2505.17082},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2505.17082}, 
+}
+
+
+```

config.json

@@ -0,0 +1,62 @@
+{
+  "architectures": [
+    "Gemma3ForConditionalGeneration"
+  ],
+  "boi_token_index": 255999,
+  "eoi_token_index": 256000,
+  "eos_token_id": [
+    1,
+    106
+  ],
+  "image_token_index": 262144,
+  "initializer_range": 0.02,
+  "mm_tokens_per_image": 256,
+  "model_type": "gemma3",
+  "text_config": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_logit_softcapping": null,
+    "cache_implementation": "hybrid",
+    "final_logit_softcapping": null,
+    "head_dim": 128,
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 5376,
+    "initializer_range": 0.02,
+    "intermediate_size": 21504,
+    "max_position_embeddings": 131072,
+    "model_type": "gemma3_text",
+    "num_attention_heads": 32,
+    "num_hidden_layers": 62,
+    "num_key_value_heads": 16,
+    "query_pre_attn_scalar": 168,
+    "rms_norm_eps": 1e-06,
+    "rope_local_base_freq": 10000.0,
+    "rope_scaling": {
+      "factor": 8.0,
+      "rope_type": "linear"
+    },
+    "rope_theta": 1000000.0,
+    "sliding_window": 1024,
+    "sliding_window_pattern": 6,
+    "torch_dtype": "bfloat16",
+    "use_cache": true,
+    "vocab_size": 262208
+  },
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.50.0.dev0",
+  "vision_config": {
+    "attention_dropout": 0.0,
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "image_size": 896,
+    "intermediate_size": 4304,
+    "layer_norm_eps": 1e-06,
+    "model_type": "siglip_vision_model",
+    "num_attention_heads": 16,
+    "num_channels": 3,
+    "num_hidden_layers": 27,
+    "patch_size": 14,
+    "torch_dtype": "bfloat16",
+    "vision_use_head": false
+  }
+}

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "bos_token_id": 2,
+  "cache_implementation": "hybrid",
+  "do_sample": true,
+  "eos_token_id": [
+    1,
+    106
+  ],
+  "pad_token_id": 0,
+  "top_k": 64,
+  "top_p": 0.95,
+  "transformers_version": "4.50.0.dev0"
+}

model-00001-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e6be3cbab588533a4b00f2d7f0409327fa0a9608dd698e2f18102b930ee39e0
+size 3797563736

model-00002-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:464e049a45839e53faa1329063614ddebd5877e30e749a30c303c748f2376cc7
+size 3996872248

model-00003-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d9677954e3c2caea91856d7454330d5cc2cb9eff41c628facef7899167cbfcb
+size 3897739432

model-00004-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1d66a4d1c4195942c3e05fa7f79b7654b3e3981e48336eab6daa60ac7439e5aa
+size 3897783032

model-00005-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:14d51a1e62d9d8cee5f6b2762b9aeb5ec842f126f7c92681333d1275cb49b4a6
+size 3897783048

model-00006-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7e599fe743df2d6983646121e7f91faaeb710852b137cf0c85f0e13e6dc86478
+size 3996872312

model-00007-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b876f51177e09bf1753a7e2fba6a728e3d3e00eba3917f0569a18aadbc9e854
+size 3897739488

model-00008-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c8070843e45521ff2f3e6b5026ab6db39b485c8a433128d3e099eec7ffeb2e9
+size 3897783032

model-00009-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:af4e30f25df5d0647ce86533bfef534b2e03fd6d7e2c068b6a36d1e8303fd3b7
+size 3897783048

model-00010-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ee8a6eecce1fb7823d8662291dd4dd20ba611cc409399714b76b7a3437137f7
+size 3996872312

model-00011-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d211b9fa4d49423d485049ff1268cd2620fc50f517987acf3f91b0994f2da52b
+size 3897739488

model-00012-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:595acef257e9c18b27d9fcc21603b89af1859aad97acb53ea21462639ec36209
+size 3897783032

model-00013-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4a55fdc43e97d92d377998f28f6834c6cbc4ddbbefbf3bce2fe9d6a32220161d
+size 3897783048

model-00014-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:24591d035bb0003b1d89041429da29359ef2693e8775c04416b68bea9a701606
+size 3996883168

preprocessor_config.json

@@ -0,0 +1,29 @@
+{
+  "do_convert_rgb": null,
+  "do_normalize": true,
+  "do_pan_and_scan": null,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "Gemma3ImageProcessor",
+  "image_seq_length": 256,
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "pan_and_scan_max_num_crops": null,
+  "pan_and_scan_min_crop_size": null,
+  "pan_and_scan_min_ratio_to_activate": null,
+  "processor_class": "Gemma3Processor",
+  "resample": 2,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "height": 896,
+    "width": 896
+  }
+}

processor_config.json

@@ -0,0 +1,4 @@
+{
+  "image_seq_length": 256,
+  "processor_class": "Gemma3Processor"
+}

special_tokens_map.json

@@ -0,0 +1,33 @@
+{
+  "boi_token": "<start_of_image>",
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eoi_token": "<end_of_image>",
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "image_token": "<image_soft_token>",
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4667f2089529e8e7657cfb6d1c19910ae71ff5f28aa7ab2ff2763330affad795
+size 33384568