VertexAGI/quartz-micro-preview-v2

Quartz Micro Preview V2

A tiny language model trained from scratch by Vertex AGI on a single GTX 1660 Ti: a 100.09M-parameter dense Llama-architecture decoder (12 layers, hidden 768, 12 heads / 4 KV heads, SwiGLU 2048, RoPE, tied 32K byte-level BPE, 1,024 context). This is Quartz Micro Preview V2, the instruction-tuned chat model. The raw pretrained model is Quartz Micro Preview V2 Base (VertexAGI/quartz-micro-preview-v2-base).

Honest summary: at ~100M parameters this model writes short, fluent, on-topic text but often states wrong facts with confidence, is weak at maths, reasoning and code, and can repeat itself. Treat it as a small research / hobby model, not a source of truth.

Formats (one repo, three formats)

Where	Format
repo root	stock Hugging Face LlamaForCausalLM (fp32 safetensors), loads with plain transformers and plain mlx_lm
mlx/fp16, mlx/q8, mlx/q4	stock MLX (fp16, 8-bit, 4-bit)
gguf/	stock llama.cpp GGUF: f16, q8_0, q4_k_m

Every format was checked against the original training code on held-out text (perplexity; lower is better):

Format	Perplexity	Difference from original
transformers fp32 (this repo root)	27.844	0.000%
MLX fp16	27.842	0.007%
MLX 8-bit	27.857	0.045%
MLX 4-bit	28.457	2.202%
GGUF f16	27.844	0.000%
GGUF Q8_0	27.850	0.020%
GGUF Q4_K_M	27.888	0.157%

Usage

# stock transformers (no custom code, no trust_remote_code)
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("VertexAGI/quartz-micro-preview-v2")
model = AutoModelForCausalLM.from_pretrained("VertexAGI/quartz-micro-preview-v2")
messages = [{"role": "user", "content": "Why do cats purr?"}]
prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
ids = tok(prompt, add_special_tokens=False, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=400, do_sample=True, temperature=0.6, top_p=0.9, repetition_penalty=1.1)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

# stock MLX (Apple silicon)
from mlx_lm import load, generate
model, tok = load("VertexAGI/quartz-micro-preview-v2")
print(generate(model, tok, prompt=tok.apply_chat_template([{"role": "user", "content": "Why do cats purr?"}], add_generation_prompt=True), max_tokens=400))

# stock llama.cpp (GGUF in the gguf/ folder)
hf download VertexAGI/quartz-micro-preview-v2 gguf/quartz-micro-preview-v2-q8_0.gguf --local-dir .
llama-cli -m gguf/quartz-micro-preview-v2-q8_0.gguf -cnv

Instruction tuning

• Base: Quartz Micro Preview V2 Base (VertexAGI/quartz-micro-preview-v2-base).
• Data: 24,170 English conversations (9,258,981 tokens): filtered OpenAssistant OASST2 (reviewed, non-synthetic, top-ranked replies, follow-up turns kept) plus short conversational / rewriting / summarizing / constraint-following slices of smol-smoltalk. Long reasoning, maths, code and URLs were removed.
• Method: full-parameter fine-tune, loss on assistant tokens only, 2 epochs, lr 1e-4 cosine, fp32.
• Chat format: <|user|>\n{message}\n<|assistant|>\n{reply}<eos> (built into the tokenizer's chat template; no system role - put instructions in the user message).
• Suggested sampling: temperature 0.6, top-p 0.9, repetition penalty 1.1.

Evaluation (hand-written prompts, not copied from the training sets)

• 20/20 prompts answered, 17/20 replies ended cleanly with <eos>, mean distinct-4-gram ratio 0.79.
• Simple keyword fact checks: 5/15 (a deliberately easy test; misses are normal for this size).
• Assistant-token loss on held-out chat data: base 2.856 -> tuned 2.280.
• All prompts and replies are in eval_outputs.md, including the wrong ones. Machine-readable results: eval_results.json (per-prompt replies, gate checks) and eval_formats.json (all shipped formats).

Same prompts, every shipped format (greedy, repetition penalty 1.1, 400 new tokens)

Format	Ended with	Keyword fact checks	Distinct-4-gram
MLX fp16	13/21	6/15	0.57
MLX 8-bit	12/21	6/15	0.57
MLX 4-bit	9/21	5/15	0.54
GGUF f16	15/21	6/15	0.74
GGUF Q8_0	16/21	5/15	0.77
GGUF Q4_K_M	14/21	5/15	0.75

The reference run above (stock transformers fp32) scored 17/20 and 5/15. Quantizing to 8-bit changes little; MLX 4-bit loses the most (fewer replies end cleanly). The 21 prompts are the 20 evaluation prompts plus one follow-up question. The MLX columns detect a clean ending by length (an approximation), the GGUF columns by the real end-of-text marker, so compare formats within a family rather than across the two. Raw numbers: eval_results.json, eval_formats.json.

Limitations

Frequent factual mistakes and invented details; weak maths and logic; can loop on longer answers; 1,024-token context; English only; no safety tuning or refusal training, so do not deploy it unsupervised.

License

Apache-2.0 for the weights and code. Training data carries its own terms (see the dataset cards; Wikipedia and Stack Exchange are share-alike).