README.md

---
license: apache-2.0
language:
- en
library_name: transformers
tags:
- ivmelabs
- causal-lm
- from-scratch
pipeline_tag: text-generation
---

THIS MODEL HAS BEEN DEEMED BROKEN.

STOP USING CONVERSATE-XL-v1 AND CONVERSATE-XL-v1.1

THANK YOU










<style>
@import url('https://fonts.googleapis.com/css2?family=Newsreader:ital,opsz,wght@0,6..72,380;0,6..72,500;0,6..72,600;0,6..72,700;1,6..72,400&family=JetBrains+Mono:wght@400;500;600&display=swap');

*, html, body, div, p, table, td, th {
  box-sizing: border-box;
}
body{
  margin:0;
  background:
    repeating-linear-gradient(180deg, rgba(0,0,0,0.014) 0px, rgba(0,0,0,0.014) 1px, transparent 1px, transparent 3px),
    #f7f4ec !important;
  color:#1c1a16;
  font-family:'Newsreader', Georgia, serif;
}
::selection{background:#ff0000; color:#f7f4ec;}
a{color:#c40000; text-decoration:none; border-bottom:1px solid rgba(255,0,0,0.35);}
a:hover{border-bottom-color:#ff0000;}

.card{
  max-width:760px;
  margin:0 auto;
  padding:52px 40px 70px;
}

.eyebrow{
  font-family:'JetBrains Mono', monospace;
  font-size:10.5px;
  letter-spacing:0.24em;
  text-transform:uppercase;
  color:#ff0000;
  margin-bottom:16px;
}

h1.title{
  font-size:38px;
  font-weight:700;
  letter-spacing:-0.01em;
  margin:0 0 6px;
  line-height:1.08;
}
.codename-line{
  font-family:'JetBrains Mono', monospace;
  font-size:12px;
  color:#8a8474;
  letter-spacing:0.03em;
  margin-bottom:28px;
}
.codename-line b{color:#4a463d; font-weight:500;}

.hero-logo{
  width:100%;
  max-width:340px;
  display:block;
  margin:0 auto 32px;
  border:1px solid #d9d2bf;
  filter:contrast(1.03);
}

p.dek{
  font-size:16.5px;
  line-height:1.7;
  color:#2b2822;
  margin:0 0 40px;
  padding-bottom:36px;
  border-bottom:1px solid #d9d2bf;
}
p.dek a{border-bottom-color:rgba(255,0,0,0.35);}

/* stat strip */
.stat-strip{
  display:grid;
  grid-template-columns: repeat(4, 1fr);
  border-top:1px solid #1c1a16;
  border-bottom:1px solid #1c1a16;
  margin-bottom:44px;
}
.stat{
  padding:14px 12px;
  border-right:1px solid #d9d2bf;
  text-align:center;
}
.stat:last-child{border-right:none;}
.stat-num{
  font-family:'JetBrains Mono', monospace;
  font-size:18px;
  font-weight:600;
  color:#1c1a16;
  display:block;
}
.stat-label{
  font-family:'JetBrains Mono', monospace;
  font-size:9px;
  letter-spacing:0.06em;
  text-transform:uppercase;
  color:#8a8474;
  margin-top:3px;
  display:block;
}

section{margin-bottom:48px;}

.section-head{
  display:flex;
  align-items:baseline;
  gap:14px;
  margin-bottom:18px;
}
.section-index{
  font-family:'JetBrains Mono', monospace;
  font-size:12px;
  color:#ff0000;
}
h2{
  font-size:22px;
  font-weight:600;
  margin:0;
  letter-spacing:-0.005em;
}

p{
  font-size:15px;
  line-height:1.72;
  color:#2b2822;
  margin:0 0 14px;
}

/* incident report block, the signature element */
.incident{
  border:1px solid #1c1a16;
  position:relative;
  padding:26px 26px 22px;
  margin-bottom:8px;
}
.incident::before{
  content:"FIELD NOTE";
  position:absolute;
  top:-9px; left:20px;
  background:#f7f4ec;
  padding:0 8px;
  font-family:'JetBrains Mono', monospace;
  font-size:9.5px;
  letter-spacing:0.12em;
  color:#ff0000;
}
.incident h3{
  font-size:15px;
  font-weight:600;
  margin:0 0 12px;
  font-family:'JetBrains Mono', monospace;
  letter-spacing:0.01em;
}
.incident p{font-size:14.5px; margin-bottom:12px;}
.incident p:last-child{margin-bottom:0;}
.incident .caveat{
  font-size:12.5px;
  color:#8a8474;
  font-style:italic;
  border-left:2px solid #ff0000;
  padding-left:12px;
  margin-top:16px;
}

/* results table */
table.results{
  width:100%;
  border-collapse:collapse;
  font-size:14px;
  margin-top:6px;
}
table.results th{
  text-align:left;
  font-family:'JetBrains Mono', monospace;
  font-size:10px;
  letter-spacing:0.06em;
  text-transform:uppercase;
  color:#8a8474;
  font-weight:500;
  border-bottom:2px solid #1c1a16;
  padding:8px 12px 8px 0;
}
table.results td{
  padding:12px 12px 12px 0;
  border-bottom:1px solid #d9d2bf;
}
table.results tr:last-child td{border-bottom:1px solid #1c1a16;}
table.results td:first-child{font-weight:600;}
td.delta{
  font-family:'JetBrains Mono', monospace;
  font-size:12.5px;
  color:#c40000;
}
td.num{font-family:'JetBrains Mono', monospace; color:#4a463d;}

/* spec list */
.spec-grid{
  display:grid;
  grid-template-columns: 1fr 1fr;
  gap:0 32px;
}
.spec-grid ul{
  margin:0;
  padding:0;
  list-style:none;
}
.spec-grid li{
  font-size:14px;
  line-height:1.9;
  color:#2b2822;
  border-bottom:1px solid #ece7d8;
  padding:7px 0;
}
.spec-grid li b{
  font-family:'JetBrains Mono', monospace;
  font-size:11px;
  color:#8a8474;
  font-weight:500;
  display:block;
  text-transform:uppercase;
  letter-spacing:0.03em;
}

/* code block */
pre{
  background:#1c1a16 !important;
  padding:20px 22px;
  overflow-x:auto;
  border-left:3px solid #ff0000;
  margin:0 0 14px;
  border-radius:0;
}
pre code{
  background:transparent !important;
  background-color:transparent !important;
  border:none !important;
  border-radius:0 !important;
  padding:0 !important;
  font-family:'JetBrains Mono', monospace !important;
  font-size:12.5px !important;
  line-height:1.65 !important;
  color:#f7f4ec !important;
  white-space:pre !important;
  display:block;
}
code{
  font-family:'JetBrains Mono', monospace !important;
  font-size:0.88em !important;
  background-color:#ece5d3 !important;
  color:#a02020 !important;
  padding:2px 6px !important;
  border-radius:2px !important;
  border:1px solid #d9d2bf !important;
  white-space:nowrap;
}
/* explicit token colors via manual spans, no JS dependency */
.tok-kw { color:#7fb8e8 !important; font-weight:600; }
.tok-str { color:#a8d992 !important; }
.tok-cls { color:#e8c66a !important; }
.tok-fn { color:#8fc7ff !important; }
.tok-num { color:#e8b56a !important; }

.note-box{
  font-size:13.5px;
  color:#4a463d;
  background:rgba(255,0,0,0.03);
  border-left:3px solid #ff0000;
  padding:14px 18px;
  line-height:1.6;
}

footer{
  border-top:1px solid #d9d2bf;
  margin-top:56px;
  padding-top:20px;
  font-family:'JetBrains Mono', monospace;
  font-size:10.5px;
  color:#8a8474;
  display:flex;
  justify-content:space-between;
}
footer .fmark{color:#ff0000;}

@media (max-width: 600px){
  .card{padding:36px 20px 50px;}
  .stat-strip{grid-template-columns:repeat(2,1fr);}
  .stat:nth-child(2){border-right:none;}
  .spec-grid{grid-template-columns:1fr;}
  footer{flex-direction:column; gap:6px;}
}
</style>

<div class="card">

<div class="eyebrow">İvmeLabs &middot; Model Card &middot; Conversate Family</div>

<h1 class="title">İvme-Conversate-XL-v1.1-Base</h1>
<div class="codename-line">Codename <b>Large Apple 1.1</b> &middot; dense decoder-only &middot; 125.6M parameters</div>

<img class="hero-logo" src="https://cdn-uploads.huggingface.co/production/uploads/670562d6ac129959c16f84d4/HsQXUoC_SharUpQAPJVIQ.png" alt="Conversate-XL-v1.1 logo">

<p class="dek">A 125.6M parameter dense decoder-only transformer, trained from scratch by İvmeLabs. Part of the Conversate family; see the <a href="https://huggingface.co/IvmeLabs">İvmeLabs organization page</a> for related models.</p>

<div class="stat-strip">
<div class="stat"><span class="stat-num">125.6M</span><span class="stat-label">Parameters</span></div>
<div class="stat"><span class="stat-num">10B</span><span class="stat-label">Tokens Trained</span></div>
<div class="stat"><span class="stat-num">12</span><span class="stat-label">Layers</span></div>
<div class="stat"><span class="stat-num">1024</span><span class="stat-label">Context Length</span></div>
</div>

<section>
<div class="section-head"><span class="section-index">01</span><h2>Summary</h2></div>
<p>This model was trained in two phases totaling 10B tokens. During evaluation of the first-phase checkpoint, a diagnostic comparison against the model's initial weights revealed that every RMSNorm scale parameter in the network, 25 tensors across 12 layers plus the output norm, had not been updated during training and remained at its default initialization value. All other parameters had trained normally.</p>

<div class="incident">
<h3>What happened, and what we did about it</h3>
<p>The cause was traced to the fused normalization kernel used during the first training phase, which does not appear to backpropagate correctly into its scale parameter under the configuration used.</p>
<p>A second training phase was run with a corrected normalization implementation, continuing from the first-phase checkpoint on the original data mixture for an additional 5B tokens at a reduced learning rate. This checkpoint is the result of that second phase.</p>
<div class="caveat">This has not been independently verified against the exact hardware and library combination involved, and should be treated as a probable rather than confirmed cause.</div>
</div>
</section>

<section>
<div class="section-head"><span class="section-index">02</span><h2>Results</h2></div>
<table class="results">
<tr><th>Metric</th><th>Phase 1 only</th><th>This checkpoint</th></tr>
<tr><td>BLiMP</td><td class="num">75.71%</td><td class="num">78.94%</td></tr>
<tr><td>ARC&#8209;Easy (accuracy)</td><td class="num">41.54%</td><td class="num">44.87%</td></tr>
<tr><td>WikiText&#8209;2 (byte&#8209;level perplexity)</td><td class="num">2.1639</td><td class="num">1.9821</td></tr>
</table>
</section>

<section>
<div class="section-head"><span class="section-index">03</span><h2>Architecture</h2></div>
<div class="spec-grid">
<ul>
<li><b>Layers &middot; Hidden Size</b>12 layers, hidden size 768</li>
<li><b>Attention</b>12 heads, head dimension 64</li>
<li><b>Feed-forward</b>SwiGLU, intermediate dimension 3072</li>
<li><b>Positional Encoding</b>Rotary (theta = 10000)</li>
</ul>
<ul>
<li><b>Normalization</b>RMSNorm, pre-normalization</li>
<li><b>Embeddings</b>Tied input/output, no bias terms</li>
<li><b>Vocabulary</b>16,000 (byte-pair encoding)</li>
<li><b>Max Sequence Length</b>1024 tokens</li>
</ul>
</div>
</section>

<section>
<div class="section-head"><span class="section-index">04</span><h2>Training</h2></div>
<p><b>Data.</b> A mixture of DCLM-baseline, FineWeb-Edu, and FineMath as the primary corpus, supplemented with Wikipedia (English) and Project Gutenberg (English). The same mixture was used in both training phases.</p>
<p><b>Optimization.</b> Two-dimensional weight matrices were optimized with Muon; embeddings and normalization parameters were optimized with AdamW. The second phase used a lower peak learning rate than the first.</p>
<p><b>Hardware.</b> A single AMD Instinct MI300X GPU, ROCm and PyTorch, bfloat16 precision.</p>
</section>

<section>
<div class="section-head"><span class="section-index">05</span><h2>Usage</h2></div>
    <pre><code><span class="tok-kw">from</span> transformers <span class="tok-kw">import</span> <span class="tok-cls">AutoModelForCausalLM</span>, <span class="tok-cls">AutoTokenizer</span>

model = <span class="tok-cls">AutoModelForCausalLM</span>.<span class="tok-fn">from_pretrained</span>(
    <span class="tok-str">"IvmeLabs/Ivme-Conversate-XL-v1.1-Base"</span>, trust_remote_code=<span class="tok-kw">True</span>
)
tokenizer = <span class="tok-cls">AutoTokenizer</span>.<span class="tok-fn">from_pretrained</span>(<span class="tok-str">"IvmeLabs/Ivme-Conversate-XL-v1.1-Base"</span>)

inputs = <span class="tok-fn">tokenizer</span>(<span class="tok-str">"Hello, my name is"</span>, return_tensors=<span class="tok-str">"pt"</span>)
outputs = model.<span class="tok-fn">generate</span>(inputs[<span class="tok-str">"input_ids"</span>], max_new_tokens=<span class="tok-num">50</span>)
<span class="tok-fn">print</span>(tokenizer.<span class="tok-fn">decode</span>(outputs[<span class="tok-num">0</span>]))</code></pre>
<div class="note-box">Requires <code>trust_remote_code=True</code>, since it uses a custom architecture defined in <code>modeling_ivme.py</code> within this repository rather than a built-in transformers model class. Review that file before loading the model, as is good practice for any model requiring <code>trust_remote_code=True</code>.</div>
</section>

<section>
<div class="section-head"><span class="section-index">06</span><h2>Checkpoints</h2></div>
<p>The final checkpoint is provided at the repository root. Intermediate checkpoints from the second training phase are archived under <code>checkpoints/</code>.</p>
</section>

<section>
<div class="section-head"><span class="section-index">07</span><h2>Limitations</h2></div>
<p>The exact cause of the normalization issue described above has not been fully confirmed. Other models trained with the same kernel and configuration may exhibit the same issue and may warrant re-evaluation.</p>
<p>As with any small-scale language model, this model has limited factual reliability, limited multi-step reasoning ability, and may produce plausible-sounding but incorrect output. It is not instruction-tuned and is intended as a base model for further fine-tuning or research use.</p>
</section>

<footer>
<span>İvmeLabs <span class="fmark">/</span> Conversate</span>
<span>Hugging Face Model Card</span>
</footer>

</div>