IvmeLabs/Ivme-Conversate-v2-Base

İvmeLabs · Model Card · Conversate Family

İvme-Conversate-v2-Base

Codename Apple 2 · dense decoder-only · 23,846,784 parameters

The second release in the Conversate line: a 24M parameter decoder-only base model trained from scratch, this time with a much heavier training diet. v1 produced grammatically correct sentences that did not really say anything when strung together, fluent without being about anything. v2 keeps almost the exact same architecture as v1 on purpose, and instead fixes the two things that actually mattered: far more training data, and a data mix weighted toward material that teaches a model to stay on topic across sentences.

23.8MParameters

12.85BTokens Trained

10Layers

4.75hTrain Time

01

Model Details

Architecture	Decoder-only transformer, dense (no loops, no exotic recurrence)
Parameters	23,846,784
Layers	10
Hidden dim	384
FFN	SwiGLU
Attention heads	6, full attention (no GQA)
Context length	1024 tokens
Vocab size	16,000 (custom BPE)
Positional encoding	RoPE (θ=10,000)
Normalization	RMSNorm (pre-norm)
Embeddings	Tied input/output
Biases	None

Nearly every setting above matches v1 on purpose. The point of v2 was to isolate the improvement to data and training, not to a bigger model.

Interactive architecture graph, opens in hfviewer

02

Benchmarks

Benchmarks were run with lm-evaluation-harness via a custom model adapter. Previous numbers from an internal script have been superseded by these.

Benchmark	v1	v2
WikiText-2 (byte perplexity) ↓	2.96	2.2250
BLiMP (macro-average, 68 paradigms) ↑	61.40%	75.09%
ARC-Easy (acc_norm) ↑	30.85%	39.98%
ARC-Easy (acc) ↑	—	43.56%

Every metric improved. The ARC-Easy acc_norm gain (30.85% to 39.98%) is more modest than the raw acc numbers alone suggest, but still a real improvement over v1.

BLiMP paradigm breakdown

Strong · Core Agreement

principle_A_case_1	100.00%
existential_there_quantifiers_1	99.40%
anaphor_number_agreement	98.00%
sentential_negation_npi_licensor	97.30%
determiner_noun_agreement_1	96.90%
principle_A_domain_1	95.40%

Weak · Long-Distance / Islands

wh_vs_that_with_gap_long_distance	17.20%
existential_there_quantifiers_2	30.20%
left_branch_island_echo_question	31.40%
superlative_quantifiers_2	33.20%
principle_A_reconstruction	36.10%

This pattern, strong local agreement paired with weaker long-distance syntax, is typical for models at this scale.

03

Does it actually make more sense now?

None of the benchmarks above directly test whether the model's writing holds together as connected text, which was v1's real problem. Sample output below, EMA weights, temperature 0.8, top_k 50.

Prompt: "Once upon a time, there was a"

Once upon a time, there was a wise old turtle named Timmy who lived on the coast of South America. In this magical place, no matter how big or large, people could look up at the sea and talk to each other. Timmy asked, "What do you mean, Timmy?" The turtle replied, "Well, I think you might see people talking about ocean creatures. They are like little waves that carry their voices. Sometimes they say they're too big or small to hear." After a few moments, Timmy had an idea. "Can we go on a boat-boat tour? I can't believe all the kids in the village are doing that!" As they sailed further, they saw many beautiful islands and vibrant colors. Each island had its unique culture and traditions. When they reached the top, they saw a group of kids playing and splashing around. "Wow, lookgies!" said Timmy. "They live in a big ocean full of colorful fish and

04

Training

Data mix

~12.85B tokens total, roughly 8 to 9x more than v1's 1.57B. v1 was trained Chinchilla-optimal. v2 deliberately overtrains well past that point, since the model is small and cheap to run regardless.

fineweb-edu	50%
smollm-corpus (cosmopedia-v2)	27%
dclm-baseline-1.0	8%
SimpleStories	5%
finemath (finemath-3plus)	5%

Python-Edu was originally planned as a fifth source, but the actual code text lives behind a gated dataset with no practical way to align it against the sampled subset at this scale, so it was dropped and its share redistributed across the rest.

Hyperparameters

Optimizer	Muon (body weights) + AdamW (embeddings, norms)
Muon lr	0.02
AdamW lr	3e-4
LR schedule	Warmup-Stable-Decay (WSD)
Weight decay	0.1
Gradient clipping	1.0
Batch size	192 sequences × 1024 tokens (196,608 tokens/step)
Total steps	65,376
Precision	bfloat16
Attention	PyTorch scaled_dot_product_attention (Flash Attention backend)
Compilation	torch.compile, roughly 2x throughput over eager
Final weights	EMA (β=0.999) of training trajectory

Hardware

Trained on a single NVIDIA RTX PRO 6000 Blackwell (96GB) in approximately 4.75 hours.

05

Tokenizer

Custom byte-level BPE tokenizer trained from scratch on a sample of the pretraining mix. Vocab size 16,000.

06

Inference

This model can now be loaded with AutoModelForCausalLM instead of the manual pickle-loading workflow, and weights are available as model.safetensors.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "IvmeLabs/Ivme-Conversate-v2-Base", trust_remote_code=True, dtype=torch.float32,
)
tokenizer = AutoTokenizer.from_pretrained("IvmeLabs/Ivme-Conversate-v2-Base", trust_remote_code=True)
model.eval()

inputs = tokenizer("Once upon a time, there was a", return_tensors="pt")
out = model.generate(
    **inputs, max_new_tokens=200, do_sample=True,
    temperature=0.8, top_k=50, pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(out[0], skip_special_tokens=True))

trust_remote_code=True is required (custom architecture: RoPE + SwiGLU + RMSNorm dense decoder). The original ckpt_final.pt pickle checkpoint and model/ architecture source remain in this repo unchanged for backwards compatibility.

Use left-padding (tokenizer.padding_side = "left") for batch generation. The model doesn't use an explicit attention mask over padded positions, so right-padding within a batch will give incorrect results.

Legacy inference

Here's a basic inference path in case you want to work with the pickle files directly.

import sys
import torch
from tokenizers import Tokenizer
from huggingface_hub import hf_hub_download, snapshot_download

repo_id = "IvmeLabs/Ivme-Conversate-v2-Base"
# Download just the model/ folder (architecture code) into the HF cache,
# then add it to sys.path so from model import ... works without the
# user needing to manually copy any files.
repo_local_dir = snapshot_download(repo_id, allow_patterns=["model/*"])
sys.path.append(repo_local_dir)
from model import IvmeConfig, IvmeConversateV2
ckpt_path = hf_hub_download(repo_id, "ckpt_final.pt")
tokenizer_path = hf_hub_download(repo_id, "tokenizer.json")
tokenizer = Tokenizer.from_file(tokenizer_path)
# IvmeConfig is a plain dataclass saved into the checkpoint. Trust this only
# because it's our own checkpoint, produced by our own training code.
torch.serialization.add_safe_globals([IvmeConfig])
ckpt = torch.load(ckpt_path, map_location="cuda")
cfg = ckpt["config"]
model = IvmeConversateV2(cfg)
# Use EMA weights (smoothed), not the raw training weights, for inference.
# Strip torch.compile's "_orig_mod." prefix if the checkpoint was compiled.
state_dict = ckpt["ema_state_dict"]
state_dict = {k.removeprefix("_orig_mod."): v for k, v in state_dict.items()}
model.load_state_dict(state_dict)
model.cuda().eval()
prompt = "Once upon a time, there was a"
ids = tokenizer.encode(prompt).ids
idx = torch.tensor([ids], dtype=torch.long, device="cuda")
eot_id = tokenizer.token_to_id("<|endoftext|>")
with torch.no_grad():
    for _ in range(200):
        idx_cond = idx if idx.size(1) <= cfg.context_len else idx[:, -cfg.context_len:]
        logits, _ = model(idx_cond)
        logits = logits[:, -1, :] / 0.8  # temperature
    v, _ = torch.<span class="tok-fn">topk</span>(logits, <span class="tok-num">50</span>)
    logits[logits &lt; v[:, [-<span class="tok-num">1</span>]]] = -float(<span class="tok-str">"inf"</span>)

    probs = torch.<span class="tok-fn">softmax</span>(logits, dim=-<span class="tok-num">1</span>)
    next_id = torch.<span class="tok-fn">multinomial</span>(probs, num_samples=<span class="tok-num">1</span>)
    idx = torch.<span class="tok-fn">cat</span>([idx, next_id], dim=<span class="tok-num">1</span>)

    <span class="tok-kw">if</span> next_id.item() == eot_id:
        <span class="tok-kw">break</span>

print(tokenizer.decode(idx[0].tolist()))

07

Limitations

• Base model only, not instruction tuned, will not follow instructions or answer questions
• English only
• 1024 token context window
• Weaker on long-distance syntactic dependencies than on local agreement, see BLiMP breakdown above
• No code data in the training mix (Python-Edu was dropped, see Training)
• Weak knowledge: the model has learned syntax and grammar well but falls short on knowledge and easily hallucinates. Expected to improve in upcoming variants via distillation.

08

What's next

Still on the table for a future version: distillation from a larger teacher model, and a return to the more experimental İvmetron architecture once time allows for the kind of patient debugging a genuinely novel design needs.

Check our other upcoming models on the İvmeLabs organization page.

09

Citation

@misc{ivme-conversate-v2-Base,
  author       = {IvmeLabs},
  title        = {İvme-Conversate-v2-Base},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/IvmeLabs/Ivme-Conversate-v2-Base}
}

İvmeLabs / Conversate

Apple photo by Matheus Cenali on Unsplash

Hugging Face Model Card