---
license: apache-2.0
language:
- en
base_model: VertexResearch/Vertex-0.6-35M-Base
datasets:
- HuggingFaceTB/smol-smoltalk
- VertexResearch/Vertex-0.6-35M-self-identification
pipeline_tag: text-generation
library_name: transformers
tags:
- chat
- vertex
- qwen3
---

# Vertex-0.6-35M-Instruct

The instruction-tuned chat version of
[Vertex-0.6-35M-Base](https://huggingface.co/VertexResearch/Vertex-0.6-35M-Base) —
a ≈34M-parameter Qwen3-architecture model trained from scratch on a single
RTX 4060 Laptop GPU. Uses standard **ChatML** formatting, so it works
out of the box in LM Studio, llama.cpp, Ollama, and MLX.

## Model

| | |
|---|---|
| Architecture | Qwen3 (`Qwen3ForCausalLM`) |
| Parameters | 33,924,992 (≈34M), tied embeddings |
| Context length | 1024 |
| Vocab | 32002 (32000 BPE + `<|im_start|>`, `<|im_end|>`) |
| Chat format | ChatML |
| EOS | `<|im_end|>` |

## Chat format

Standard ChatML, embedded as a `chat_template`:

```
<|im_start|>user
Hello!<|im_end|>
<|im_start|>assistant
Hi there!<|im_end|>
```

## Training

SFT on top of Vertex-0.6-35M-Base:

- **Data:** 269,072 conversations — [smol-smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk)
  (conversations ≤1024 tokens) interleaved with
  [Vertex-0.6-35M-self-identification](https://huggingface.co/datasets/VertexResearch/Vertex-0.6-35M-self-identification)
  (derived from SupraLabs/LLM-self-identification)
- ≈127M conversation tokens, 2 epochs, assistant-only loss masking
- bf16 + torch.compile, fused AdamW, lr 1e-3 cosine, 5,605 steps (~1.9h)

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "VertexResearch/Vertex-0.6-35M-Instruct"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

enc = tok.apply_chat_template(
    [{"role": "user", "content": "Who are you?"}],
    add_generation_prompt=True, return_tensors="pt", return_dict=True,
)
out = model.generate(enc["input_ids"], max_new_tokens=100)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))
# I am Vertex 0.6 35M. I was created by VertexResearch.
```

## Limitations

These models are not the most coherent yet and need more tuning: expect rambling, repetition, and inconsistent answers, especially over longer generations.

- 34M parameters: expect simple conversational ability, not reasoning,
  factual reliability, or long-form coherence.
- English + Python centric; 1024-token context; no safety tuning.