Update README.md

README.md

@@ -1,73 +1,138 @@
-# Quark (50M)
-
-Quark is a lightweight decoder-only language model with approximately 50 million parameters. It is designed for efficient inference on consumer hardware while maintaining reasonable language understanding and generation capabilities.
-
-## Model Description
-
-- **Architecture:** SmolLM-style (Grouped-Query Attention, SwiGLU, RMSNorm, RoPE, deep-thin)
-- **Parameters:** ~50M
-- **Context length:** 2048 tokens
-- **Vocabulary size:** 49,152 (HuggingFaceTB/cosmo2-tokenizer)
-- **Training data:** HuggingFaceTB/smollm-corpus (5B tokens total)
-  - 60% cosmopedia-v2
-  - 30% python-edu
-  - 10% fineweb-edu
-- **Hardware:** RTX 3070 (8 GB VRAM)
-- **License:** MIT
-
-## Intended Uses
-
-- Lightweight on-device chat
-- Educational experiments with small LMs
-- Fine-tuning for specific tasks (instruction following, code generation, etc.)
-
-## How to Use
+---
+language:
+- en
+- code
+license: apache-2.0
+tags:
+- smol
+- pretraining
+- instruct
+- 50M
+- causal-lm
+- gqa
+- swiglu
+- rmsnorm
+datasets:
+- HuggingFaceTB/smollm-corpus
+metrics:
+- perplexity
+model-index:
+- name: Quark-50m-Instruct
+  results: []
+pipeline_tag: text-generation
+---
+
+# Quark-50m-Instruct
+
+**Quark-50m-Instruct** is a small (≈50M parameters) decoder-only language model, fine-tuned for instruction following.
+It is built on the same architecture as the now‑abandoned “SmolLM” family and was fully pretrained on 5 billion tokens from
+[HuggingFaceTB/smollm‑corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus).
+
+- **Model type:** Causal Language Model (LLaMA‑style decoder)
+- **Architecture:** GQA · SwiGLU · RMSNorm · RoPE · Weight‑tying
+- **Pretraining tokens:** 5 B
+- **Fine‑tuning:** Instruction‑tuned (details below)
+- **Creators:** [OvercastLab](https://huggingface.co/OvercastLab) (research & development lab for ML/AI)
+- **Release date:** 22 April 2026
+
+## Model Summary
+
+Quark-50m-Instruct is designed to be an efficient assistant that can run on consumer GPUs (e.g., RTX 3070 with 8 GB VRAM)
+and even on CPU for light workloads. It is **not** competitive with large models on knowledge‑intensive tasks,
+but it excels at:
+
+- Simple conversational tasks
+- Code generation and explanation (Python)
+- Short text rewriting and summarisation
+- On‑device / edge inference
+
+The architecture closely follows the efficient‑small‑LM blueprint popularised by SmolLM:
+
+| Component   | Details                       |
+|-------------|-------------------------------|
+| Vocab size  | 49,152                        |
+| Hidden size | 384                           |
+| Layers      | 24                            |
+| Attention   | Grouped Query (6 Q heads, 2 KV heads) |
+| FFN         | SwiGLU with 1,024 intermediate |
+| Position    | RoPE (θ = 10,000)             |
+| Normalisation | RMSNorm (pre‑block)         |
+
+Total trainable parameters: **≈48 M** (with weight tying).
+
+## Uses
+
+### Direct Use
+The model can be used via the 🤗 Transformers library for standard text generation.
+It expects chat‑formatted input (see example below).
+
+### Downstream Use
+Because of the open Apache‑2.0 license, you may fine‑tune Quark-50m‑Instruct on your own data for
+domain‑specific tasks – for instance, a customer‑support bot, a code reviewer, or a story writer.
+
+### Limitations
+- Limited world knowledge (stopped at mid‑2025 pretraining data).
+- Short context window (2,048 tokens).
+- Small size means it can make more factual mistakes than larger models.
+
+## How to Get Started
 
 ```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
+from transformers import AutoTokenizer, AutoModelForCausalLM
 
 model_name = "OvercastLab/Quark-50m-Instruct"
-model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True)
-tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
-
-inputs = tokenizer("Hello, how are you?", return_tensors="pt")
-outputs = model.generate(**inputs, max_new_tokens=50)
-print(tokenizer.decode(outputs[0]))
-
-Training Details
-
-    Effective batch size: 64 sequences per step (4 micro-batches × 16 gradient accumulation)
 
-    Learning rate: 3e-4 (cosine decay to 3e-5)
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
 
-    Optimizer: AdamW (β1=0.9, β2=0.95, weight decay=0.1)
+messages = [
+    {"role": "system", "content": "You are Quark, a helpful assistant."},
+    {"role": "user", "content": "Explain group query attention in one sentence."}
+]
 
-    Precision: bfloat16
+inputs = tokenizer.apply_chat_template(
+    messages,
+    tokenize=True,
+    add_generation_prompt=True,
+    return_tensors="pt"
+).to(model.device)
 
-    Total tokens: 5 billion
+outputs = model.generate(inputs, max_new_tokens=128)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))```
+## Training Details
 
-    Training steps: ~1.2 million
+### Pretraining
 
-    Checkpoint frequency: every 2,000 steps
+The base model was pretrained from scratch on a single NVIDIA RTX 3070 (8 GB VRAM).
+Training took approximately **X days** (wall clock) and consumed about **Y kWh** (see [Environmental Impact](#environmental-impact)).
 
-Limitations
+#### Data mix
 
-    Small parameter count limits factual knowledge and reasoning depth.
+Quark‑50m was trained on exactly 5 billion tokens sampled from `HuggingFaceTB/smollm-corpus` with the following proportions:
 
-    May produce repetitive or nonsensical outputs when prompted outside its training distribution.
+| Subset            | Share | Tokens |
+|-------------------|-------|--------|
+| cosmopedia‑v2     | 60%   | 3.0 B  |
+| fineweb‑edu‑dedup | 40%   | 2.0 B  |
 
-    The base model is not instruction-tuned; use the -Instruct variant for conversational tasks.
+All data was tokenised with the official [Cosmo2 tokenizer](https://huggingface.co/HuggingFaceTB/cosmo2-tokenizer) (vocab size 49,152).
 
-Citation
+#### Hyperparameters (pretraining)
 
-If you use Quark in your work, please cite:
-bibtex
+| Parameter               | Value                      |
+|-------------------------|----------------------------|
+| Sequence length         | 2,048                      |
+| Micro‑batch size        | 4                          |
+| Gradient accumulation   | 16                         |
+| Effective batch         | 64 seqs (≈131k tokens)     |
+| Optimizer               | AdamW (β₁=0.9, β₂=0.95)   |
+| Learning rate           | 3e‑4 → 3e‑5 (cosine decay)|
+| Warmup steps            | 1,000                      |
+| Weight decay            | 0.1                        |
+| Gradient clipping       | 1.0                        |
+| Mixed precision         | bfloat16                   |
 
-@misc{quark2025,
-  author = {OvercastLab},
-  title = {Quark: A 50M Parameter Lightweight Language Model},
-  year = {2025},
-  publisher = {Hugging Face},
-  howpublished = {\url{https://huggingface.co/OvercastLab/Quark-50m-Instruct}}
-}
+### Instruction Fine‑tuning
 
+The base model was fine‑tuned on a curated set of instruction‑following data (details to be released).  
+The fine‑tuning used **LoRA** with the same sequence length and a lower learning rate (1e‑4) for a few thousand steps.