Hugging Face's logo

ThingAI
/
ARK-270M-Base

Text Generation
Transformers
Safetensors
Italian
English
quark
causal-lm
bilingual
italian
english
small-language-model
trained-from-scratch
conversational
custom_code
Model card Files
xet
 Community
ThingsAI commited on
Commit
858ce94
·
verified ·
1 Parent(s): 4ff4f88

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +15 -38
README.md CHANGED
@@ -16,16 +16,9 @@ library_name: transformers
16
  pipeline_tag: text-generation
17
  ---
18
 
19
- # Quark-270M — Bilingual Italian-English Language Model
20
-
21
-
22
-
23
-
24
- ## Overview
25
-
26
- Quark-270M-Base is the **base pretrained** model from the Quark family — a compact bilingual language model for Italian and English, built entirely from scratch by [ThingsAI](https://things-ai.org) in Prato, Italy.
27
-
28
- This is the raw pretrained model optimized for text completion. For conversational use, see [**Quark-270M-Instruct**](https://huggingface.co/ThingAI/Quark-270m-Instruct).
29
 
30
  ## Model Details
31
 
@@ -54,20 +47,6 @@ This is the raw pretrained model optimized for text completion. For conversation
54
  | Normalization | RMSNorm (pre-norm) |
55
  | Positional Encoding | RoPE (θ=10,000) |
56
  | Weight Tying | embed_tokens ↔ lm_head |
57
- | Embedding Overhead | ~19% of total params |
58
-
59
- ### Scaling from v0.2
60
-
61
- | | Quark-135M (v0.2) | **Quark-270M (v0.3)** |
62
- |---|---|---|
63
- | d_model | 576 | **768** |
64
- | n_heads | 9 | **12** |
65
- | n_kv_heads | 3 | **4** |
66
- | n_layers | 30 | **32** |
67
- | d_ff | 1,536 | **2,048** |
68
- | Params | 144M | **252M** |
69
- | Embedding overhead | 26% | **19%** |
70
- | Pretrain loss | 4.63 | **~4.4** |
71
 
72
  ## Pretraining
73
 
@@ -90,23 +69,16 @@ Trained on **~10B tokens** from a curated bilingual mix:
90
 
91
  | | |
92
  |---|---|
93
- | **Hardware** | NVIDIA B200 (RunPod) |
94
  | **Total Tokens** | ~10B |
95
  | **Batch Size** | 64 × 4 grad accum = 256 sequences |
96
- | **Effective Batch** | 524,288 tokens/step |
97
  | **Sequence Length** | 2,048 |
98
  | **Learning Rate** | 3e-4 → 3e-5 (cosine) |
99
  | **Warmup Steps** | 1,000 |
100
  | **Optimizer** | AdamW (β₁=0.9, β₂=0.95) |
101
- | **Weight Decay** | 0.1 |
102
  | **Precision** | BF16 mixed precision |
103
  | **Throughput** | ~281k tokens/sec |
104
  | **Training Time** | ~10 hours |
105
- | **Final Loss** | ~4.4 |
106
-
107
- ### Tokenizer
108
-
109
- [ThingAI/QuarkTokenizer](https://huggingface.co/ThingAI/QuarkTokenizer) — custom byte-level BPE, 65,536 tokens, balanced Italian + English coverage.
110
 
111
  ## Usage
112
 
@@ -114,20 +86,19 @@ Trained on **~10B tokens** from a curated bilingual mix:
114
  from transformers import AutoModelForCausalLM, AutoTokenizer
115
 
116
  model = AutoModelForCausalLM.from_pretrained(
117
- "ThingAI/Quark-270m-Instruct",
118
  trust_remote_code=True,
119
  torch_dtype="bfloat16"
120
  ).cuda()
121
 
122
- tokenizer = AutoTokenizer.from_pretrained("ThingAI/Quark-270m-Instruct")
123
 
124
- # Text completion (base model)
125
  inputs = tokenizer("L'Italia è un paese", return_tensors="pt").to("cuda")
126
  out = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.7, top_k=40)
127
  print(tokenizer.decode(out[0]))
128
  ```
129
 
130
- > **Note:** This is a base model — it does text completion, not instruction following. For chat/instructions, use [Quark-270M-Instruct](https://huggingface.co/ThingAI/Quark-270m-Instruct).
131
 
132
  ## Limitations
133
 
@@ -135,9 +106,15 @@ print(tokenizer.decode(out[0]))
135
  - **Hallucination:** Generates plausible but often incorrect information
136
  - **Mathematics:** Limited arithmetic capabilities
137
  - **Code:** Can produce syntactically plausible but often non-functional code
138
- - **Tokenizer:** Trained on limited data (14M rows), may have suboptimal compression
139
 
 
140
 
 
 
 
 
 
 
141
 
142
  ## Links
143
 
@@ -147,4 +124,4 @@ print(tokenizer.decode(out[0]))
147
 
148
  ---
149
 
150
- *Built in Italy by ThingsAI 🇮🇹*
 
16
  pipeline_tag: text-generation
17
  ---
18
 
19
+ # Quark-270M Base — Bilingual Italian-English Language Model
20
+ Quark-270M Base is a compact bilingual language model for Italian and English, built entirely from scratch by [ThingsAI](https://things-ai.org). This is the raw pretrained model optimized for text completion.
21
+ For conversational use, see [**Quark-270M-Instruct**](https://huggingface.co/ThingAI/Quark-270m-Instruct).
 
 
 
 
 
 
 
22
 
23
  ## Model Details
24
 
 
47
  | Normalization | RMSNorm (pre-norm) |
48
  | Positional Encoding | RoPE (θ=10,000) |
49
  | Weight Tying | embed_tokens ↔ lm_head |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
50
 
51
  ## Pretraining
52
 
 
69
 
70
  | | |
71
  |---|---|
72
+ | **Hardware** | NVIDIA B200 |
73
  | **Total Tokens** | ~10B |
74
  | **Batch Size** | 64 × 4 grad accum = 256 sequences |
 
75
  | **Sequence Length** | 2,048 |
76
  | **Learning Rate** | 3e-4 → 3e-5 (cosine) |
77
  | **Warmup Steps** | 1,000 |
78
  | **Optimizer** | AdamW (β₁=0.9, β₂=0.95) |
 
79
  | **Precision** | BF16 mixed precision |
80
  | **Throughput** | ~281k tokens/sec |
81
  | **Training Time** | ~10 hours |
 
 
 
 
 
82
 
83
  ## Usage
84
 
 
86
  from transformers import AutoModelForCausalLM, AutoTokenizer
87
 
88
  model = AutoModelForCausalLM.from_pretrained(
89
+ "ThingAI/Quark-270m-Base",
90
  trust_remote_code=True,
91
  torch_dtype="bfloat16"
92
  ).cuda()
93
 
94
+ tokenizer = AutoTokenizer.from_pretrained("ThingAI/Quark-270m-Base")
95
 
 
96
  inputs = tokenizer("L'Italia è un paese", return_tensors="pt").to("cuda")
97
  out = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.7, top_k=40)
98
  print(tokenizer.decode(out[0]))
99
  ```
100
 
101
+ > **Note:** This is a base model for text completion. For chat and instructions, use [Quark-270M-Instruct](https://huggingface.co/ThingAI/Quark-270m-Instruct).
102
 
103
  ## Limitations
104
 
 
106
  - **Hallucination:** Generates plausible but often incorrect information
107
  - **Mathematics:** Limited arithmetic capabilities
108
  - **Code:** Can produce syntactically plausible but often non-functional code
 
109
 
110
+ ## The Quark Family
111
 
112
+ | Model | Parameters | Type |
113
+ |---|---|---|
114
+ | [Quark-50M](https://huggingface.co/ThingAI/Quark-50m) | 51M | Base |
115
+ | [Quark-135M](https://huggingface.co/ThingAI/Quark-135m) | 135M | Base |
116
+ | **Quark-270M Base** | **252M** | **Base** |
117
+ | [Quark-270M-Instruct](https://huggingface.co/ThingAI/Quark-270m-Instruct) | 252M | Chat |
118
 
119
  ## Links
120
 
 
124
 
125
  ---
126
 
127
+ *Built from scratch by ThingsAI 🇮🇹*