| --- |
| library_name: transformers |
| tags: |
| - chemistry |
| - biology |
| - finance |
| - legal |
| - music |
| - art |
| - code |
| - climate |
| - medical |
| - agent |
| - text-generation-inference |
| - Duchifat-2 |
| - conversational |
| - chat |
| - SFT |
| license: apache-2.0 |
| language: |
| - he |
| - en |
| base_model: |
| - Raziel1234/Duchifat-2 |
| pipeline_tag: text-generation |
| metrics: |
| - brier_score |
| --- |
| # 🕊️ Duchifat-2.2-Instruct |
|
|
| **Duchifat-2.2-Instruct** is a fine-tuned version of the original **Duchifat-2 base model**. While this specific version is an optimized Instruct/Chat model, the underlying base architecture and weights were developed and trained from scratch by **Raziel**. |
|
|
| ## 🚀 Lineage & Development |
| - **Base Model (Duchifat-2):** Built and pre-trained from scratch on **3.27 Billion tokens** (50/50 Hebrew-English C4 dataset). It features 136M parameters and was designed to establish a native Hebrew reasoning foundation. |
| - **Version 2.2 (Instruct):** A refined fine-tuned version (SFT) designed to transform the base capabilities into a quirky, safe, and highly responsive conversational agent. |
|
|
| ### Key Features: |
| * **Native Hebrew Foundation:** Unlike models that adapt English weights, Duchifat was born in Hebrew using the **DictaLM tokenizer**, ensuring high efficiency and natural linguistic flow. |
| * **Compact Power:** At only 136M parameters, it delivers impressive performance while remaining small enough for edge deployment and low-latency applications. |
| * **Quirky & Human-like:** The SFT process focused on giving the model a distinct personality—witty and engaging rather than robotic. |
| * **Safety Integrated:** Built-in guardrails ensure the model remains professional and refuses to engage with profanity or offensive prompts. |
|
|
| ## 📊 Benchmark Results (Zero-Shot) |
| Tested using manual prompt formatting to accurately reflect real-world chat performance. |
|
|
| | Task | Version | Filter | n-shot | Metric | Value | Stderr | |
| | :--- | :--- | :--- | :--- | :--- | :--- | :--- | |
| | **piqa** | 1 | none | 0 | acc | **0.70** | ± 0.1528 | |
| | **piqa** | 1 | none | 0 | acc_norm | **0.70** | ± 0.1528 | |
| | **hellaswag** | 1 | none | 0 | acc | **0.40** | ± 0.1633 | |
| | **hellaswag** | 1 | none | 0 | acc_norm | **0.40** | ± 0.1633 | |
| | **winogrande** | 1 | none | 0 | acc | **0.40** | ± 0.1633 | |
| | **arc_easy** | 1 | none | 0 | acc | **0.10** | ± 0.1000 | |
| | **arc_easy** | 1 | none | 0 | acc_norm | **0.10** | ± 0.1000 | |
| |
| ## 🛠️ Technical Specifications |
| * **Parameters:** 136M |
| * **Base Pre-training Data:** 3.27B tokens (C4 Hebrew/English) |
| * **Tokenizer:** DictaLM (Hebrew optimized) |
| * **Context Window:** 1024 tokens |
| |
| ## 💡 How to Use |
| Use the following instruction format to trigger the Instruct-tuned behavior: |
| |
| Prompt Template: |
| ``` |
| <|instruction|> |
| {user_query} |
| <|assistant|> |
| ``` |
| |
| ## Example Usage: |
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
| model_id = "razielAI/Duchifat-2.2-Instruct" |
| tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) |
| model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).to("cuda") |
| |
| prompt = "<|instruction|>\nשלום!\n<|assistant|>\n" |
| inputs = tokenizer(prompt, return_tensors="pt").to("cuda") |
|
|
| output = model.generate(**inputs, max_new_tokens=256, temperature=0.7, do_sample=True) |
| print(tokenizer.decode(output[0], skip_special_tokens=True)) |
| ``` |
| |
| ## ⚠️ Limitations |
| Duchifat-2.2 is a lightweight model. It excels at conversational tasks, social media content, and short-form text generation. It is not designed for complex mathematical proofs or extensive coding sessions. |
| |
| ## 🕊️ About the Duchifat Project |
| The Duchifat (Hoopoe) project is dedicated to creating efficient, open-source AI with a native understanding of the Hebrew language and culture. |
