İvme-Conversate-XL-v1.1-Base
A 125.6M parameter dense decoder-only transformer, trained from scratch by İvmeLabs. Part of the Conversate family; see the İvmeLabs organization page for related models.
Summary
This model was trained in two phases totaling 10B tokens. During evaluation of the first-phase checkpoint, a diagnostic comparison against the model's initial weights revealed that every RMSNorm scale parameter in the network, 25 tensors across 12 layers plus the output norm, had not been updated during training and remained at its default initialization value. All other parameters had trained normally.
What happened, and what we did about it
The cause was traced to the fused normalization kernel used during the first training phase, which does not appear to backpropagate correctly into its scale parameter under the configuration used.
A second training phase was run with a corrected normalization implementation, continuing from the first-phase checkpoint on the original data mixture for an additional 5B tokens at a reduced learning rate. This checkpoint is the result of that second phase.
Results
| Metric | Phase 1 only | This checkpoint |
|---|---|---|
| BLiMP | 75.71% | 78.94% |
| ARC‑Easy (accuracy) | 41.54% | 44.87% |
| WikiText‑2 (byte‑level perplexity) | 2.1639 | 1.9821 |
Architecture
- Layers · Hidden Size12 layers, hidden size 768
- Attention12 heads, head dimension 64
- Feed-forwardSwiGLU, intermediate dimension 3072
- Positional EncodingRotary (theta = 10000)
- NormalizationRMSNorm, pre-normalization
- EmbeddingsTied input/output, no bias terms
- Vocabulary16,000 (byte-pair encoding)
- Max Sequence Length1024 tokens
Training
Data. A mixture of DCLM-baseline, FineWeb-Edu, and FineMath as the primary corpus, supplemented with Wikipedia (English) and Project Gutenberg (English). The same mixture was used in both training phases.
Optimization. Two-dimensional weight matrices were optimized with Muon; embeddings and normalization parameters were optimized with AdamW. The second phase used a lower peak learning rate than the first.
Hardware. A single AMD Instinct MI300X GPU, ROCm and PyTorch, bfloat16 precision.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"IvmeLabs/Ivme-Conversate-XL-v1.1-Base", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("IvmeLabs/Ivme-Conversate-XL-v1.1-Base")
inputs = tokenizer("Hello, my name is", return_tensors="pt")
outputs = model.generate(inputs["input_ids"], max_new_tokens=50)
print(tokenizer.decode(outputs[0]))trust_remote_code=True, since it uses a custom architecture defined in modeling_ivme.py within this repository rather than a built-in transformers model class. Review that file before loading the model, as is good practice for any model requiring trust_remote_code=True.Checkpoints
The final checkpoint is provided at the repository root. Intermediate checkpoints from the second training phase are archived under checkpoints/.
Limitations
The exact cause of the normalization issue described above has not been fully confirmed. Other models trained with the same kernel and configuration may exhibit the same issue and may warrant re-evaluation.
As with any small-scale language model, this model has limited factual reliability, limited multi-step reasoning ability, and may produce plausible-sounding but incorrect output. It is not instruction-tuned and is intended as a base model for further fine-tuning or research use.