IvmeLabs/Ivme-Conversate-XL-v1.1-Base

İvmeLabs · Model Card · Conversate Family

İvme-Conversate-XL-v1.1-Base

Codename Large Apple 1.1 · dense decoder-only · 125.6M parameters

A 125.6M parameter dense decoder-only transformer, trained from scratch by İvmeLabs. Part of the Conversate family; see the İvmeLabs organization page for related models.

125.6MParameters

10BTokens Trained

12Layers

1024Context Length

01

Summary

This model was trained in two phases totaling 10B tokens. During evaluation of the first-phase checkpoint, a diagnostic comparison against the model's initial weights revealed that every RMSNorm scale parameter in the network, 25 tensors across 12 layers plus the output norm, had not been updated during training and remained at its default initialization value. All other parameters had trained normally.

What happened, and what we did about it

The cause was traced to the fused normalization kernel used during the first training phase, which does not appear to backpropagate correctly into its scale parameter under the configuration used.

A second training phase was run with a corrected normalization implementation, continuing from the first-phase checkpoint on the original data mixture for an additional 5B tokens at a reduced learning rate. This checkpoint is the result of that second phase.

This has not been independently verified against the exact hardware and library combination involved, and should be treated as a probable rather than confirmed cause.

02

Results

Metric	Phase 1 only	This checkpoint
BLiMP	75.71%	78.94%
ARC‑Easy (accuracy)	41.54%	44.87%
WikiText‑2 (byte‑level perplexity)	2.1639	1.9821

03

Architecture

• Layers · Hidden Size12 layers, hidden size 768
• Attention12 heads, head dimension 64
• Feed-forwardSwiGLU, intermediate dimension 3072
• Positional EncodingRotary (theta = 10000)

• NormalizationRMSNorm, pre-normalization
• EmbeddingsTied input/output, no bias terms
• Vocabulary16,000 (byte-pair encoding)
• Max Sequence Length1024 tokens

04

Training

Data. A mixture of DCLM-baseline, FineWeb-Edu, and FineMath as the primary corpus, supplemented with Wikipedia (English) and Project Gutenberg (English). The same mixture was used in both training phases.

Optimization. Two-dimensional weight matrices were optimized with Muon; embeddings and normalization parameters were optimized with AdamW. The second phase used a lower peak learning rate than the first.

Hardware. A single AMD Instinct MI300X GPU, ROCm and PyTorch, bfloat16 precision.

05

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "IvmeLabs/Ivme-Conversate-XL-v1.1-Base", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("IvmeLabs/Ivme-Conversate-XL-v1.1-Base")
inputs = tokenizer("Hello, my name is", return_tensors="pt")
outputs = model.generate(inputs["input_ids"], max_new_tokens=50)
print(tokenizer.decode(outputs[0]))

Requires trust_remote_code=True, since it uses a custom architecture defined in modeling_ivme.py within this repository rather than a built-in transformers model class. Review that file before loading the model, as is good practice for any model requiring trust_remote_code=True.

06

Checkpoints

The final checkpoint is provided at the repository root. Intermediate checkpoints from the second training phase are archived under checkpoints/.

07

Limitations

The exact cause of the normalization issue described above has not been fully confirmed. Other models trained with the same kernel and configuration may exhibit the same issue and may warrant re-evaluation.

As with any small-scale language model, this model has limited factual reliability, limited multi-step reasoning ability, and may produce plausible-sounding but incorrect output. It is not instruction-tuned and is intended as a base model for further fine-tuning or research use.

İvmeLabs / Conversate Hugging Face Model Card