| | --- |
| | language: |
| | - en |
| | license: apache-2.0 |
| | tags: |
| | - text-generation |
| | - instruction-tuning |
| | - multi-task |
| | - reasoning |
| | - email |
| | - summarization |
| | - chat |
| | - peft |
| | - lora |
| | - qwen |
| | - deepseek |
| | base_model: deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B |
| | datasets: |
| | - HuggingFaceTB/smoltalk |
| | - snoop2head/enron_aeslc_emails |
| | - lucadiliello/STORIES |
| | - abisee/cnn_dailymail |
| | - wiki40b |
| | model_type: causal-lm |
| | inference: true |
| | library_name: peft |
| | pipeline_tag: text-generation |
| | --- |
| | |
| | # 🧠 Deepseek-R1-multitask-lora |
| |
|
| | **Author:** Gilbert Akham |
| | **License:** Apache-2.0 |
| | **Base model:** [`deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B`](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B) |
| | **Adapter type:** LoRA (PEFT) |
| | **Capabilities:** Multi-task generalization & reasoning |
| |
|
| | --- |
| |
|
| | # 🚀 What It Can Do |
| |
|
| | This multitask fine-tuned model handles a broad set of natural language and reasoning-based tasks, such as: |
| |
|
| | ✉️ Email & message writing — generate clear, friendly, or professional communications. |
| |
|
| | 📖 Story & creative writing — craft imaginative narratives, poems, and dialogues. |
| |
|
| | 💬 Conversational chat — maintain coherent, context-aware conversations. |
| |
|
| | 💡 Explanations & tutoring — explain technical or abstract topics simply. |
| |
|
| | 🧩 Reasoning & logic tasks — provide step-by-step answers for analytical questions. |
| |
|
| | 💻 Code generation & explanation — write and explain Python or general programming code. |
| |
|
| | 🌍 Translation & summarization — translate between multiple languages or condense information. |
| |
|
| | The model’s multi-domain training (based on datasets like SmolTalk, Everyday Conversations, and reasoning-rich samples) makes it suitable for assistants, chatbots, content generators, or educational tools. |
| | --- |
| |
|
| | ## 🧩 Training Details |
| |
|
| | | Parameter | Value | |
| | |------------|-------| |
| | | Base model | `deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B` | |
| | | Adapter | LoRA (r=8, alpha=32, dropout=0.1) | |
| | | Max sequence length | 1024 | |
| | | Learning rate | 3e-5 (cosine decay) | |
| | | Optimizer | `adamw_8bit` | |
| | | Grad Accumulation | 4 | |
| | | Precision | 4-bit quantized, FP16 compute | |
| | | Steps | 12k total (best @ ~8.2k) | |
| | | Training time | ~2.5h on A4000 | |
| | | Frameworks | 🤗 Transformers, PEFT, TRL, BitsAndBytes | |
| |
|
| | --- |
| |
|
| | ## 🧠 Reasoning Capability |
| |
|
| | Thanks to integration of **SmolTalk** and diverse multi-task prompts, the model learns: |
| | - **Chain-of-thought style reasoning** |
| | - **Conversational grounding** |
| | - **Multi-step logical inferences** |
| | - **Instruction following** across domains |
| |
|
| | Example: |
| | ```text |
| | ### Task: Explain reasoning |
| | |
| | ### Input: |
| | If a train leaves City A at 3 PM and arrives at City B at 6 PM, covering 180 km, what is its average speed? |
| | |
| | ### Output: |
| | The train travels 180 km in 3 hours. |
| | Average speed = 180 ÷ 3 = 60 km/h. |
| | |
