README.md

---
license: apache-2.0
pipeline_tag: text-generation
tags:
- unsloth
datasets:
- constructai/Ling-v2.6-Flash-Distilled-15K
---

# 💥 Qweling3.5-0.8B

![image](https://cdn-uploads.huggingface.co/production/uploads/6968f14c2b9883902ed723d6/M0FQFhDBNIil1jmgyW19B.png)

**📄 Overview**

| | |
|---|---|
| **Model Name** | constructai/Qweling3.5-0.8B |
| **Base Model** | Qwen3.5-0.8B-Base |
| **Dataset** | constructai/Ling-v2.6-Flash-Distilled-15K |
| **Training Type** | Supervised Fine-Tuning (SFT) |
| **Parameters** | 0.9B |
| **Framework** | Unsloth + LoRA |
| **Hardware** | NVIDIA T4 16GB |

---

**🎯 Intended Use**

This model is designed for **step‑by‑step reasoning tasks** where the answer requires logical decomposition before the final response. It is optimized for:

- **Educational applications** — explaining "why" and "how" questions
- **On‑device assistants** — runs on mobile, Raspberry Pi, or CPU‑only environments
- **Fast prototyping** — small footprint (0.9B parameters), low latency
- **Reasoning distillation research** — studying how small models learn from large ones (Ling → Qwen)

**Not recommended for:** multimodal tasks, non‑reasoning chat (e.g., creative writing), or production systems requiring 100% factual accuracy.

---

**⚠️ Limitations & Intended Use**

Intended Use:

* Educational & Reasoning tasks — explaining step‑by‑step logic (math, science, common sense)

* On‑device assistants — runs on CPU, Raspberry Pi, mobile (small footprint, fast inference)

* Research baseline — for studying SFT‑only reasoning without RLHF/DPO

* Distillation experiments — testing how well small models learn from large (Ling → Qwen)

Limitations:

* Size matters — 0.9B parameters, so complex or multi‑hop reasoning may still fail

* No multimodal — text only; images, video, audio are not supported

* Factual accuracy — may hallucinate or give incorrect answers; always verify critical outputs

* Domain restricted — trained on **15,000** reasoning examples (2.5 epochs); general chat or creative writing may be suboptimal

* Training data bias — inherits biases from `constructai/Ling-v2.6-Flash-Distilled-15K` dataset; not safety‑filtered for harmful content

* Hardware specific — optimised for T4/consumer GPUs; very slow on CPU without quantisation

---

# Train details

To begin with, I have trained `Qwen3.5-0.8B-Base` using **unsloth and LoRA** on the `constructai/Ling-v2.6-Flash-Distilled-15K` dataset using **2.5 epochs**, everything seemed to go fine, but during the first test of `constructai/Qweling3.5-0.8B`, I discovered a **problem** that it occurred due to an error in my code. I had to **train** the adapters on **one epoch** and then I achieved the best **result**! You can test the model yourself using this code:
```python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "constructai/Qweling3.5-0.8B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")

def ask(question):
    prompt = f"<|im_start|>user\n{question}\nAnswer concisely:<|im_end|>\n<|im_start|>assistant\n"
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs, max_new_tokens=1024, temperature=0.1, do_sample=True)
    answer = tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True)
    return answer

test_questions = [
    "On one branch there are 2 monkeys. On two such branches there are 4 monkeys. Now answer: How many on 3 branches?",
]

for q in test_questions:
    print(f"Q: {q}")
    print(f"A: {ask(q)}\n{'-'*50}")
```

---

**🙏 Acknowledgements**

This project would not have been possible without the open‑source community and the following resources:

* [Qwen Team](https://huggingface.co/Qwen) (Alibaba Cloud) — for releasing the Qwen3.5-0.8B-Base model under Apache 2.0, a perfect balance of size and intelligence.

* [Unsloth AI](https://huggingface.co/unsloth) — for making fine‑tuning on consumer hardware fast and memory‑efficient.

* [Hugging Face](https://huggingface.co/) — for the ecosystem (transformers, datasets, PEFT, Hub) that democratises LLM training.

* [Kaggle](https://www.kaggle.com) — for providing free T4 GPU runtime to run this experiment.

---

**📖 Citation**

```bibtex
@misc{Qweling3.5-0.8B,
  author = {constructai},
  title = {Qwenling3.5-0.8B: Small Reasoning Model via SFT on Ling Traces},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {https://huggingface.co/constructai/Qweling3.5-0.8B},
}
```