| --- |
| language: |
| - it |
| - en |
| license: apache-2.0 |
| tags: |
| - text-generation |
| - causal-lm |
| - bilingual |
| - italian |
| - english |
| - small-language-model |
| - trained-from-scratch |
| - quark |
| - instruct |
| - sft |
| - chat |
| library_name: transformers |
| pipeline_tag: text-generation |
| --- |
| |
| # Quark-270M-Instruct — Bilingual Chat Model |
| Quark-270M-Instruct is the **instruction-tuned** version of [Quark-270M Base](https://huggingface.co/ThingAI/Quark-270m-Base), fine-tuned for conversational use in Italian and English. Built entirely from scratch by [ThingsAI](https://things-ai.org). |
|
|
| ## Quick Start |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| import torch |
| |
| model = AutoModelForCausalLM.from_pretrained( |
| "ThingAI/Quark-270m-Instruct", |
| trust_remote_code=True, |
| torch_dtype=torch.bfloat16 |
| ).cuda() |
| model.lm_head.weight = model.embed_tokens.weight # ensure weight tying |
| |
| tokenizer = AutoTokenizer.from_pretrained("ThingAI/Quark-270m-Instruct") |
| |
| prompt = "<|user|>\nCiao, come stai?\n<|end|>\n<|assistant|>\n" |
| inputs = tokenizer(prompt, return_tensors="pt").to("cuda") |
| out = model.generate(**inputs, max_new_tokens=150, do_sample=True, temperature=0.7, top_k=40) |
| print(tokenizer.decode(out[0], skip_special_tokens=False)) |
| ``` |
|
|
| ## Chat Format |
|
|
| ``` |
| <|user|> |
| {user message} |
| <|end|> |
| <|assistant|> |
| {model response} |
| <|end|> |
| ``` |
|
|
| Multi-turn: |
|
|
| ``` |
| <|user|> |
| Ciao! |
| <|end|> |
| <|assistant|> |
| Ciao! Come posso aiutarti? |
| <|end|> |
| <|user|> |
| Cos'è l'intelligenza artificiale? |
| <|end|> |
| <|assistant|> |
| ``` |
|
|
| ## Model Details |
|
|
| | | | |
| |---|---| |
| | **Base Model** | [Quark-270M Base](https://huggingface.co/ThingAI/Quark-270m-Base) | |
| | **Parameters** | 252M (with weight tying) | |
| | **Architecture** | Decoder-only Transformer (GQA, SwiGLU, RMSNorm, RoPE) | |
| | **Vocabulary** | 65,537 tokens | |
| | **Context Length** | 2,048 tokens | |
| | **Precision** | BF16 | |
| | **Languages** | Italian, English | |
|
|
| ### Architecture |
|
|
| | | | |
| |---|---| |
| | d_model | 768 | |
| | Layers | 32 | |
| | Query Heads | 12 | |
| | KV Heads | 4 | |
| | Head Dim | 64 | |
| | FFN Dim | 2,048 | |
| | Activation | SwiGLU | |
| |
| ## Training |
| |
| ### Base Pretraining |
| |
| ~10B tokens on a bilingual mix (Italian 50%, English 43%, Code 7%) on NVIDIA B200. See [Quark-270M Base](https://huggingface.co/ThingAI/Quark-270m-Base) for details. |
| |
| ### SFT (Instruction Tuning) |
| |
| Fine-tuned on a diverse mix of conversational and instructional data: |
| |
| | Dataset | Examples | Type | |
| |---|---|---| |
| | FreedomIntelligence/alpaca-gpt4-italian | ~52,000 | Italian instructions | |
| | HuggingFaceH4/no_robots | ~9,500 | English conversations | |
| | m-a-p/CodeFeedback-Filtered-Instruction | 5,000 | Code instructions | |
| | yogeshm/text_to_bash (×80) | ~9,900 | Terminal commands | |
| | Custom chitchat (×100) | ~3,000 | Identity, greetings, basic Q&A | |
| | **Total** | **~80,000** | | |
|
|
| | | | |
| |---|---| |
| | **Hardware** | NVIDIA B200 | |
| | **Epochs** | 3 | |
| | **Learning Rate** | 2e-5 (cosine decay) | |
| | **Batch Size** | 16 × 4 = 64 effective | |
| | **Sequence Length** | 512 | |
|
|
| ## Inference Server |
|
|
| Quark-270M-Instruct powers [Things Chat](https://chat.things-ai.org) via a self-hosted FastAPI server with SSE streaming, conversation memory, web search, and content moderation. |
|
|
|
|
| ## Limitations |
|
|
| - **252M is small:** Limited factual knowledge, prone to hallucination |
| - **Mathematics:** Unreliable beyond basic arithmetic |
| - **Code:** Generates plausible but often non-functional code |
| - **Context:** 2,048 token window |
| - **No system prompt:** The model was not trained with `<|system|>` tags |
|
|
| ### Good for |
|
|
| - Self-hosted bilingual chatbot |
| - Learning about LLM training from scratch |
| - Terminal command assistance |
| - Light conversational AI |
|
|
| ### Not suited for |
|
|
| - Factual Q&A requiring accuracy |
| - Complex reasoning or math |
| - Production-grade code generation |
| - Safety-critical applications |
|
|
| ## The Quark Family |
|
|
| | Model | Parameters | Type | |
| |---|---|---| |
| | [Quark-50M](https://huggingface.co/ThingAI/Quark-50m) | 51M | Base | |
| | [Quark-135M](https://huggingface.co/ThingAI/Quark-135m) | 135M | Base | |
| | [Quark-270M Base](https://huggingface.co/ThingAI/Quark-270m-Base) | 252M | Base | |
| | **Quark-270M-Instruct** | **252M** | **Chat** | |
|
|
| ## Links |
|
|
| - 🌐 [ThingsAI](https://things-ai.org) |
| - 💬 [Things Chat](https://chat.things-ai.org) |
| - 🔤 [QuarkTokenizer](https://huggingface.co/ThingAI/QuarkTokenizer) |
| - 📊 [Open SLM Leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard) |
|
|
| --- |
|
|
| *Built from scratch by ThingsAI 🇮🇹* |
