Hugging Face's logo

ThingAI
/
ARK-270M-Base

Text Generation
Transformers
Safetensors
Italian
English
quark
causal-lm
bilingual
italian
english
small-language-model
trained-from-scratch
conversational
custom_code
Model card Files
xet
 Community
ThingsAI commited on
Commit
19f01c0
·
verified ·
1 Parent(s): aef7e9d

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +147 -0
README.md CHANGED
@@ -1,3 +1,150 @@
1
  ---
 
 
 
2
  license: apache-2.0
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - it
4
+ - en
5
  license: apache-2.0
6
+ tags:
7
+ - text-generation
8
+ - causal-lm
9
+ - bilingual
10
+ - italian
11
+ - english
12
+ - small-language-model
13
+ - trained-from-scratch
14
+ - quark
15
+ library_name: transformers
16
+ pipeline_tag: text-generation
17
  ---
18
+
19
+ # Quark-270M v0.3 — Bilingual Italian-English Language Model
20
+
21
+
22
+
23
+
24
+ ## Overview
25
+
26
+ Quark-270M-Base is the **base pretrained** model from the Quark family — a compact bilingual language model for Italian and English, built entirely from scratch by [ThingsAI](https://things-ai.org) in Prato, Italy.
27
+
28
+ This is the raw pretrained model optimized for text completion. For conversational use, see [**Quark-270M-Instruct**](https://huggingface.co/ThingAI/Quark-270m-v0.3-Instruct).
29
+
30
+ ## Model Details
31
+
32
+ | | |
33
+ |---|---|
34
+ | **Parameters** | 252M (with weight tying) |
35
+ | **Architecture** | Decoder-only Transformer |
36
+ | **Vocabulary** | 65,537 tokens (QuarkTokenizer, bilingual BPE) |
37
+ | **Context Length** | 2,048 tokens |
38
+ | **Precision** | BF16 |
39
+ | **Languages** | Italian, English |
40
+ | **License** | Apache 2.0 |
41
+
42
+ ## Architecture
43
+
44
+ | Component | Details |
45
+ |---|---|
46
+ | Model Dimension | 768 |
47
+ | Layers | 32 |
48
+ | Attention | Grouped Query Attention (GQA) |
49
+ | Query Heads | 12 |
50
+ | KV Heads | 4 (3:1 ratio) |
51
+ | Head Dimension | 64 |
52
+ | FFN Dimension | 2,048 |
53
+ | FFN Activation | SwiGLU |
54
+ | Normalization | RMSNorm (pre-norm) |
55
+ | Positional Encoding | RoPE (θ=10,000) |
56
+ | Weight Tying | embed_tokens ↔ lm_head |
57
+ | Embedding Overhead | ~19% of total params |
58
+
59
+ ### Scaling from v0.2
60
+
61
+ | | Quark-135M (v0.2) | **Quark-270M (v0.3)** |
62
+ |---|---|---|
63
+ | d_model | 576 | **768** |
64
+ | n_heads | 9 | **12** |
65
+ | n_kv_heads | 3 | **4** |
66
+ | n_layers | 30 | **32** |
67
+ | d_ff | 1,536 | **2,048** |
68
+ | Params | 144M | **252M** |
69
+ | Embedding overhead | 26% | **19%** |
70
+ | Pretrain loss | 4.63 | **~4.4** |
71
+
72
+ ## Pretraining
73
+
74
+ ### Data
75
+
76
+ Trained on **~10B tokens** from a curated bilingual mix:
77
+
78
+ | Subset | Weight | Source |
79
+ |---|---|---|
80
+ | FineWeb-2 (Italian) | 29% | `HuggingFaceFW/fineweb-2` [ita_Latn] |
81
+ | CulturaX (Italian) | 14% | `uonlp/CulturaX` [it] |
82
+ | Wikipedia (Italian) | 7% | `wikimedia/wikipedia` [20231101.it] |
83
+ | FineWeb (English) | 36% | `HuggingFaceFW/fineweb` [sample-10BT] |
84
+ | Wikipedia (English) | 7% | `wikimedia/wikipedia` [20231101.en] |
85
+ | The Stack (Code) | 7% | `bigcode/the-stack-smol` |
86
+
87
+ **Language split:** Italian 50% · English 43% · Code 7%
88
+
89
+ ### Training Configuration
90
+
91
+ | | |
92
+ |---|---|
93
+ | **Hardware** | NVIDIA B200 (RunPod) |
94
+ | **Total Tokens** | ~10B |
95
+ | **Batch Size** | 64 × 4 grad accum = 256 sequences |
96
+ | **Effective Batch** | 524,288 tokens/step |
97
+ | **Sequence Length** | 2,048 |
98
+ | **Learning Rate** | 3e-4 → 3e-5 (cosine) |
99
+ | **Warmup Steps** | 1,000 |
100
+ | **Optimizer** | AdamW (β₁=0.9, β₂=0.95) |
101
+ | **Weight Decay** | 0.1 |
102
+ | **Precision** | BF16 mixed precision |
103
+ | **Throughput** | ~281k tokens/sec |
104
+ | **Training Time** | ~10 hours |
105
+ | **Final Loss** | ~4.4 |
106
+
107
+ ### Tokenizer
108
+
109
+ [ThingAI/QuarkTokenizer](https://huggingface.co/ThingAI/QuarkTokenizer) — custom byte-level BPE, 65,536 tokens, balanced Italian + English coverage.
110
+
111
+ ## Usage
112
+
113
+ ```python
114
+ from transformers import AutoModelForCausalLM, AutoTokenizer
115
+
116
+ model = AutoModelForCausalLM.from_pretrained(
117
+ "ThingAI/Quark-270m-Instruct",
118
+ trust_remote_code=True,
119
+ torch_dtype="bfloat16"
120
+ ).cuda()
121
+
122
+ tokenizer = AutoTokenizer.from_pretrained("ThingAI/Quark-270m-Instruct")
123
+
124
+ # Text completion (base model)
125
+ inputs = tokenizer("L'Italia è un paese", return_tensors="pt").to("cuda")
126
+ out = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.7, top_k=40)
127
+ print(tokenizer.decode(out[0]))
128
+ ```
129
+
130
+ > **Note:** This is a base model — it does text completion, not instruction following. For chat/instructions, use [Quark-270M-Instruct](https://huggingface.co/ThingAI/Quark-270m-v0.3-Instruct).
131
+
132
+ ## Limitations
133
+
134
+ - **Scale:** 252M parameters limits factual knowledge and complex reasoning
135
+ - **Hallucination:** Generates plausible but often incorrect information
136
+ - **Mathematics:** Limited arithmetic capabilities
137
+ - **Code:** Can produce syntactically plausible but often non-functional code
138
+ - **Tokenizer:** Trained on limited data (14M rows), may have suboptimal compression
139
+
140
+
141
+
142
+ ## Links
143
+
144
+ - 🌐 [ThingsAI](https://things-ai.org)
145
+ - 💬 [Things Chat](https://chat.things-ai.org)
146
+ - 🔤 [QuarkTokenizer](https://huggingface.co/ThingAI/QuarkTokenizer)
147
+
148
+ ---
149
+
150
+ *Built in Italy by ThingsAI 🇮🇹*