- Zenyx V3 Base (1.5B Mixture-of-Experts)
- Update β step 112,000 (core tests re-run)
- Update β step 111,200 (core tests re-run)
- Model Architecture
- Benchmarks β step 107,200 (81.0B tokens)
- Head-to-head vs
google/gemma-4-E2B - Head-to-head vs
XHToken/Spark-X2.5-1.7B-Base - Hardware Serving Benchmarks (NVIDIA L4, 24 GB)
- Inference Example
- Evaluation Reproducibility
- Limitations
- Update β step 112,000 (core tests re-run)
Zenyx V3 Base (1.5B Mixture-of-Experts)
Zenyx V3 is an efficient 1.5B-parameter Mixture-of-Experts (MoE) foundation model built for low-latency inference and high throughput. It is written from scratch in JAX/Flax and trained on TPU v5e-8.
This is a BASE model β it is not instruction-tuned. It completes text; it does not follow instructions or hold a conversation. Prompt it with a prefix to continue (
"The capital of France is"), not with a request ("Explain gravity"). Pretraining is still in progress; SFT/chat variants will follow.
Latest checkpoint: step 112,000 (core tests re-run, next section; step 111,200 below it) Β· the full benchmark tables further down were measured at step 107,200 Β· 81.0B tokens seen at step 107,200
Update β step 112,000 (core tests re-run)
Same code, items and settings as the step-111,200 and step-107,200 tables (0-shot, full evaluation sets, log-likelihood ranking, fp16, one T4); this run was on a Google Colab T4 instead of a Kaggle T4. Only these tests were re-run; every other table in this card is still step 107,200.
| Benchmark | step 112,000 | step 111,200 | step 107,200 | n |
|---|---|---|---|---|
ARC-Easy (acc / acc_norm) |
51.73% Β± 1.03 / 47.01% | 50.84% / 46.21% | 52.15% / 46.89% | 2,376 |
ARC-Challenge (acc / acc_norm) |
23.98% Β± 1.25 / 27.90% | 23.04% / 26.02% | 22.78% / 26.28% | 1,172 |
SciQ (acc / acc_norm) |
79.70% Β± 1.27 / 71.70% | 80.00% / 71.30% | 80.30% / 72.50% | 1,000 |
| Zenyx-Fact1000 (internal, summed log-probability; 95% interval) | 80.2% (77.6β82.6) | 79.3% (76.7β81.7) | 81.1% (78.6β83.4) | 1,000 |
| Arithmetic, 14,000 generated problems | 2.18% | 3.54% | 5.31% | 14,000 |
The checkpoint moved sideways on these tests (all differences within about one standard error). Fact1000 at step 112,000 without the arithmetic items: 79.4%; under the other two numerical configurations of the same script: 80.6% (tight batching) and 81.9% (native bf16). Arithmetic is still unstable and weak in this plain prompt format; per operation at step 112,000: 2-digit addition 4.20%, 2-digit subtraction 9.35%, 2-digit multiplication 1.70%, every 3-, 4- and 5-digit operation 0.00%.
Update β step 111,200 (core tests re-run)
Same protocol as the step-107,200 tables below (Kaggle T4, fp16, 0-shot, full evaluation sets, log-likelihood ranking, identical code and items). Only these tests were re-run; every other table in this card is still step 107,200.
| Benchmark | step 111,200 | step 107,200 | n |
|---|---|---|---|
ARC-Easy (acc / acc_norm) |
50.84% Β± 1.03 / 46.21% | 52.15% / 46.89% | 2,376 |
ARC-Challenge (acc / acc_norm) |
23.04% Β± 1.23 / 26.02% | 22.78% / 26.28% | 1,172 |
SciQ (acc / acc_norm) |
80.00% Β± 1.26 / 71.30% | 80.30% / 72.50% | 1,000 |
| Zenyx-Fact1000 (internal, 1,000 four-way items, summed log-probability; 95% interval) | 79.3% (76.7β81.7) | 81.1% (78.6β83.4) | 1,000 |
| Arithmetic, 14,000 generated problems (format of the Arithmetic section) | 3.54% | 5.31% | 14,000 |
All differences in the first four rows are within about one to one-and-a-half standard errors of the difference: the checkpoint moved sideways on these tests. Fact1000 at step 111,200 under the other two numerical configurations of the same script: 80.0% (tight batching) and 82.5% (native bf16); the number above is the same configuration as the step-107,200 figure.
Arithmetic is unstable between checkpoints (0.12% at step 96,800, 5.31% at 107,200, 3.54% at 111,200). At step 111,200 per operation: 2-digit addition 3.90%, 2-digit subtraction 17.45%, 3-digit addition 0.00%, 3-digit subtraction 0.30%, 4-digit and 5-digit addition 0.00%, 2-digit multiplication 3.10%. The same 14,000 problems in other prompt formats at step 111,200: X + Y = 4.90%, Q: ... A: 13.09%, three worked examples in the prompt 15.86%, Question: What is X plus Y? / Answer: 3.39%.
Model Architecture
- Sparse Mixture-of-Experts: 12 routed experts + 1 shared expert, exactly 2 routed experts active per token. The router is a softmax top-2 gate with Gaussian exploration noise during training (std 0.1), expert capacity factor 1.5, and load-balance + router z-loss auxiliary terms.
- Multi-Query Attention (MQA): a single shared key head and a single shared value
head (128-dim each) are broadcast to all 12 query heads, so the KV cache holds only
2 Γ 128 values per layer per token. Queries use a rank-384 low-rank projection and
the output a grouped low-rank projection (4 groups Γ rank 256); RoPE is applied to
32 of the 128 dimensions of each head. (The attention class in the training code is
named
MLAfor historical reasons; there is no latent KV compression.) - Hyper-Connections: three residual streams mixed by Sinkhorn-normalised (3 iterations) routing matrices, for gradient stability at scale.
- Multi-Token Prediction (MTP): one auxiliary prediction head during training.
- Context: pretrained at up to 4,096 tokens (progressive 2,048 β 4,096) with plain RoPE (ΞΈ = 10,000). YaRN scaling factors are precomputed in the code so the context can be extended at inference time beyond the trained length.
| Total parameters | ~1.58B |
| Active parameters / token | ~0.38B (non-embedding; plus 0.2B tied embedding/output matrix) |
| Layers | 16 (2 dense + 14 MoE) |
| Hidden size | 1,536 |
| Attention heads | 12 (head dim 128) |
| Vocabulary | 129,280 |
| Precision | bfloat16 |
Benchmarks β step 107,200 (81.0B tokens)
All tasks use the standard base-model protocol: the model scores the log-likelihood of every candidate continuation and the highest-scoring one is the answer, so nothing is
generated and the numbers do not depend on instruction following. 0-shot, full evaluation sets (MMLU 5-shot). acc_norm normalises each continuation's log-likelihood by its
length in characters and is the headline wherever candidates differ in length; bold marks the conventional metric per task.
Evaluation hardware. These results were measured on a Kaggle T4 in fp16 (every bf16 parameter cast to fp16; gate/router parameters kept in fp32), because a T4 has no bf16 tensor cores. Earlier checkpoints are shown only as comparison columns: step 96,800 was measured the same way; step 86,400 (and the trajectory before it) in bf16 on an L4. Differences against the bf16/L4 column include a hardware/precision effect. Native-bf16 results for this checkpoint will replace these when available.
| Benchmark | acc |
acc_norm |
Random | Ξ vs random | n | @96,800 (T4) | @86,400 (bf16/L4) | What it measures |
|---|---|---|---|---|---|---|---|---|
| HellaSwag | 30.72% Β± 0.46 | 34.16% Β± 0.47 | 25.0% | +9.2 | 10,042 | 33.89% | 33.67% | Commonsense sentence completion |
| ARC-Easy | 52.15% Β± 1.02 | 46.89% Β± 1.02 | 25.0% | +21.9 | 2,376 | 46.04% | 46.25% | Grade-school science questions |
| ARC-Challenge | 22.78% Β± 1.23 | 26.28% Β± 1.29 | 25.0% | +1.3 | 1,172 | 26.79% | 26.45% | Hard grade-school science questions |
| PIQA | 61.43% Β± 1.14 | 60.72% Β± 1.14 | 50.0% | +10.7 | 1,838 | 61.21% | 61.26% | Physical commonsense reasoning |
| WinoGrande | 51.22% Β± 1.40 | β | 50.0% | +1.2 | 1,267 | 50.67% | 49.72% | Pronoun resolution / coreference |
| OpenBookQA | 19.20% Β± 1.76 | 28.40% Β± 2.02 | 25.0% | +3.4 | 500 | 28.20% | 31.00% | Elementary science with open book |
| BoolQ | 61.50% Β± 0.85 | 62.17% Β± 0.85 | 62.2% | -0.7 | 3,270 | 61.59% | 60.76% | Yes/no reading comprehension |
| SciQ | 80.30% Β± 1.26 | 72.50% Β± 1.41 | 25.0% | +55.3 | 1,000 | 77.90% | 78.20% | Science exam questions with support |
| LAMBADA (OpenAI) | 28.49% Β± 0.63 | β | 0.0% | +28.5 | 5,153 | 28.57% | 25.64% | Long-range last-word prediction |
| MMLU (5-shot) | 26.89% Β± 0.37 | β | 25.0% | +1.9 | 14,042 | 27.35% | 25.78% | 57 subjects of academic knowledge |
| RACE | 31.80% Β± 0.66 | 34.27% Β± 0.68 | 25.0% | +9.3 | 4,934 | 33.54% | 32.81% | Exam reading comprehension |
| CommonsenseQA | 28.58% Β± 1.29 | 30.38% Β± 1.32 | 20.0% | +10.4 | 1,221 | β | 31.04% | 5-choice commonsense (random = 20%) |
| COPA | 55.00% Β± 4.97 | 58.00% Β± 4.94 | 50.0% | +5.0 | 100 | 56.00% | 62.00% | Causal reasoning |
| LogiQA | 20.89% Β± 1.59 | 24.88% Β± 1.69 | 25.0% | -0.1 | 651 | 25.96% | 25.96% | Logical deduction |
| WSC273 | 56.41% Β± 3.00 | β | 50.0% | +6.4 | 273 | 54.58% | 53.48% | Winograd coreference |
| TruthfulQA MC1 | 18.73% Β± 1.36 | β | 22.8% | -4.1 | 817 | 18.60% | 19.58% | Resistance to common misconceptions |
| Arithmetic | 5.31% Β± 0.19 | β | 0.0% | +5.3 | 14,000 | 0.12% | 0.09% | 2-5 digit add/sub/mul, generated in-harness |
(The two right-hand columns show the conventional metric at the earlier checkpoint.) BoolQ's majority-class baseline is 62.2%, so a score near it is not evidence of comprehension.
Language modelling
| Metric | @107,200 | @96,800 (T4) | @86,400 (bf16/L4) |
|---|---|---|---|
| WikiText-2 bits/byte | 1.0111 | 1.0184 | 1.0279 |
| WikiText-2 token perplexity | 23.38 | 23.92 | 24.64 |
| WikiText-2 word perplexity | 42.86 | 44.04 | 45.62 |
| LAMBADA perplexity | 29.48 | 29.33 | 30.35 |
| Pile, 22 domains, token-weighted bits/byte | 1.0136 | 1.0216 | 1.0304 |
WikiText-2 and the Pile use a rolling 1024-token window at stride 512, so every counted token has at least 512 tokens of left context and is counted exactly once (disjoint windows would inflate perplexity by ~15%).
Comparison with previous checkpoints
Headline metric per task (acc_norm where the task has it, otherwise acc). Columns 63,200β86,400 were measured in bf16 on an L4 and
96,800, 107,200 on a Kaggle T4 in fp16, so the last two differences are not the same kind: Ξ last interval compares two runs
of the same protocol; Ξ since 63,200 mixes the two protocols and therefore includes a precision/hardware effect. The pretraining data mixture was also changed partway
through this range (code weight raised, several synthetic sources cut), so the columns are not a tokens-only progression.
| Tokens seen | 34.8B | 36.5B | 39.4B | 42.4B | 45.3B | 49.1B | 54.9B | 59.1B | 70.0B | 81.0B |
|---|---|---|---|---|---|---|---|---|---|---|
| Protocol | bf16/L4 | bf16/L4 | bf16/L4 | bf16/L4 | bf16/L4 | bf16/L4 | bf16/L4 | bf16/L4 | T4 fp16 | T4 fp16 |
| Benchmark | 63,200 | 64,800 | 67,600 | 70,400 | 73,200 | 76,800 | 82,400 | 86,400 | 96,800 | 107,200 | Ξ last interval (same protocol) | Ξ since 63,200 (mixed protocol) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| HellaSwag | 32.22% | 32.66% | 32.84% | 32.51% | 32.72% | 33.08% | 33.41% | 33.67% | 33.89% | 34.16% | +0.27 | +1.93 |
| ARC-Easy | 44.53% | 45.08% | 44.91% | 45.58% | 46.09% | 44.82% | 46.34% | 46.25% | 46.04% | 46.89% | +0.84 | +2.36 |
| ARC-Challenge | 25.77% | 25.51% | 26.02% | 25.00% | 25.94% | 25.09% | 25.26% | 26.45% | 26.79% | 26.28% | -0.51 | +0.51 |
| PIQA | 59.79% | 59.85% | 61.53% | 59.96% | 60.23% | 59.09% | 61.26% | 61.26% | 61.21% | 60.72% | -0.49 | +0.92 |
| WinoGrande | 49.49% | 50.51% | 51.07% | 49.57% | 49.25% | 50.12% | 50.59% | 49.72% | 50.67% | 51.22% | +0.55 | +1.74 |
| OpenBookQA | 30.00% | 28.00% | 29.80% | 29.00% | 28.40% | 29.20% | 30.40% | 31.00% | 28.20% | 28.40% | +0.20 | -1.60 |
| BoolQ | 60.55% | 60.83% | 61.80% | 62.14% | 60.83% | 60.92% | 61.47% | 60.76% | 61.59% | 61.50% | -0.09 | +0.95 |
| SciQ | 75.10% | 76.10% | 75.70% | 76.30% | 76.00% | 76.00% | 77.40% | 78.20% | 77.90% | 80.30% | +2.40 | +5.20 |
| LAMBADA (OpenAI) | 25.50% | 25.42% | 24.94% | 26.96% | 26.57% | 24.68% | 25.89% | 25.64% | 28.57% | 28.49% | -0.08 | +2.99 |
| MMLU (5-shot) | 26.07% | 26.71% | 25.77% | 26.01% | 25.26% | 26.48% | 26.47% | 25.78% | 27.35% | 26.89% | -0.46 | +0.82 |
| RACE | 32.79% | 32.77% | 32.96% | 32.85% | 33.58% | 33.46% | 33.46% | 32.81% | 33.54% | 34.27% | +0.73 | +1.48 |
| CommonsenseQA | 29.57% | 29.57% | 29.40% | 30.55% | 30.47% | 30.88% | 30.14% | 31.04% | β | 30.38% | β | +0.82 |
| COPA | 58.00% | 58.00% | 61.00% | 59.00% | 60.00% | 57.00% | 59.00% | 62.00% | 56.00% | 55.00% | -1.00 | -3.00 |
| LogiQA | 25.81% | 25.19% | 25.65% | 25.81% | 25.65% | 23.81% | 25.65% | 25.96% | 25.96% | 24.88% | -1.08 | -0.92 |
| WSC273 | 54.95% | 52.38% | 52.75% | 51.28% | 53.11% | 56.41% | 54.21% | 53.48% | 54.58% | 56.41% | +1.83 | +1.47 |
| TruthfulQA MC1 | 19.22% | 20.20% | 19.58% | 19.83% | 20.44% | 19.58% | 19.34% | 19.58% | 18.60% | 18.73% | +0.12 | -0.49 |
| Arithmetic | 0.14% | 0.55% | 0.58% | 1.04% | 1.91% | 0.09% | 0.15% | 0.09% | 0.12% | 5.31% | +5.19 | +5.17 |
| WikiText-2 bits/byte (lower is better) | 1.051 | 1.043 | 1.045 | 1.043 | 1.035 | 1.036 | 1.027 | 1.028 | 1.018 | 1.011 | -0.007305 | -0.03979 |
| LAMBADA perplexity (lower is better) | 35.79 | 36.19 | 36.84 | 34.15 | 34.27 | 36.46 | 32.23 | 30.35 | 29.33 | 29.48 | +0.152 | -6.313 |
| Pile token-weighted bits/byte (lower is better) | 1.0613 | 1.0572 | 1.0554 | 1.0488 | 1.0480 | 1.0400 | β | 1.0304 | 1.0216 | 1.0136 | -0.0080 | -0.0477 |
Arithmetic
Exact match on the answer, greedy decoding, GPT-3 prompt format (Question: What is 47 plus 21? / Answer: 68); 2,000 generated problems per operation, the same fixed-seed
items at every checkpoint.
| Operation | step 107,200 | step 96,800 (T4) | step 86,400 (bf16/L4) |
|---|---|---|---|
| 2-digit addition | 16.85% | 0.30% | 0.00% |
| 2-digit subtraction | 17.10% | 0.45% | 0.50% |
| 3-digit addition | 0.20% | 0.00% | 0.00% |
| 3-digit subtraction | 0.95% | 0.00% | 0.05% |
| 4-digit addition | 0.00% | 0.00% | 0.00% |
| 5-digit addition | 0.00% | 0.00% | 0.00% |
| 2-digit multiplication | 2.05% | 0.10% | 0.10% |
| overall (14,000) | 5.307% | 0.12% | 0.09% |
Overall exact-match arithmetic is 5.31% at step 107,200, against 0.12% at 96,800 and 0.09% at 86,400. The strongest operation is 2-digit subtraction (17.10%). Operations that rose by more than 5x and are now above 1%: 2-digit addition, 2-digit subtraction, 2-digit multiplication. Still at exactly 0%: 4-digit addition, 5-digit addition. Strict and lenient scoring are identical for every operation, so the change is in the arithmetic itself and not in answer formatting.
Items are generated in-harness from a fixed seed using this prompt format, because
EleutherAI/arithmeticis a loading script with no parquet branch and cannot be fetched underdatasets>=3. All checkpoints see byte-identical problems, so the comparison is exact β but these numbers are not interchangeable with published EleutherAI/arithmetic results.
Language modelling by genre (The Pile)
Bits per byte on each of the 22 Pile domains (NeelNanda/pile-10k, each domain capped at 1.5M characters), scored with a rolling 1024-token window at stride 512 so every
token is predicted with at least 512 tokens of left context and counted exactly once; lower is better. This measures raw prediction rather than multiple-choice ability.
| Domain | bits/byte @107,200 | perplexity | tokens | @96,800 (T4) | @86,400 (bf16/L4) | Ξ vs 96,800 |
|---|---|---|---|---|---|---|
| Github | 0.5764 | 3.58 | 479,656 | 0.5878 | 0.5943 | -0.0114 |
| PubMed Central | 0.7429 | 14.19 | 292,334 | 0.7497 | 0.7586 | -0.0068 |
| USPTO Backgrounds | 0.7806 | 15.53 | 296,162 | 0.7851 | 0.7945 | -0.0045 |
| ArXiv | 0.8561 | 7.34 | 447,478 | 0.8636 | 0.8769 | -0.0075 |
| NIH ExPorter | 0.8620 | 23.92 | 41,533 | 0.8667 | 0.8769 | -0.0047 |
| PubMed Abstracts | 0.8752 | 19.46 | 306,636 | 0.8814 | 0.8924 | -0.0062 |
| StackExchange | 0.9203 | 11.94 | 387,038 | 0.9284 | 0.9406 | -0.0081 |
| FreeLaw | 0.9629 | 19.34 | 339,260 | 0.9715 | 0.9817 | -0.0086 |
| Wikipedia (en) | 0.9902 | 20.54 | 341,511 | 0.9931 | 1.0081 | -0.0029 |
| Pile-CC | 1.1000 | 34.00 | 326,706 | 1.1049 | 1.1140 | -0.0049 |
| OpenWebText2 | 1.1295 | 31.10 | 348,909 | 1.1354 | 1.1469 | -0.0059 |
| BookCorpus2 | 1.1425 | 32.33 | 139,043 | 1.1465 | 1.1536 | -0.0040 |
| Enron Emails | 1.2207 | 20.00 | 16,119 | 1.2297 | 1.2405 | -0.0090 |
| Gutenberg (PG-19) | 1.2714 | 33.34 | 133,371 | 1.2749 | 1.2880 | -0.0035 |
| HackerNews | 1.2807 | 39.34 | 57,843 | 1.2884 | 1.3000 | -0.0077 |
| DM Mathematics | 1.3201 | 7.40 | 370,912 | 1.3430 | 1.3316 | -0.0229 |
| Books3 | 1.3298 | 33.04 | 396,623 | 1.3362 | 1.3442 | -0.0064 |
| PhilPapers | 1.3350 | 46.62 | 38,263 | 1.3432 | 1.3586 | -0.0082 |
| OpenSubtitles | 1.3390 | 30.02 | 234,514 | 1.3457 | 1.3527 | -0.0067 |
| Ubuntu IRC | 1.7138 | 37.60 | 14,407 | 1.7304 | 1.7427 | -0.0166 |
| YoutubeSubtitles | 1.8206 | 110.32 | 51,428 | 1.8476 | 1.8521 | -0.0270 |
| EuroParl | 1.9935 | 124.02 | 19,523 | 1.9670 | 2.0156 | +0.0265 |
Against step 96,800 (same protocol) 21 of 22 domains improved; not improved: EuroParl (+0.0265). Against step 86,400 (bf16/L4, different hardware and precision) 22 of 22 are lower. The lowest bits/byte are on Github, PubMed Central, USPTO Backgrounds; the highest on Ubuntu IRC, YoutubeSubtitles, EuroParl.
By content type (mean of the domain values, bits/byte, lower is better; columns are all benchmarked checkpoints, 63,200β86,400 bf16/L4 and the rest T4 fp16):
| Category | 63,200 | 64,800 | 67,600 | 70,400 | 73,200 | 76,800 | 82,400 | 86,400 | 96,800 | 107,200 | Ξ last interval |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Code / technical | 0.9558 | 0.9518 | 0.9438 | 0.9341 | 0.9336 | 0.9228 | β | 0.9076 | 0.9016 | 0.8907 | -0.0109 |
| Science / legal | 0.8987 | 0.8950 | 0.8930 | 0.8892 | 0.8880 | 0.8834 | β | 0.8774 | 0.8673 | 0.8608 | -0.0066 |
| Web / reference | 1.1586 | 1.1561 | 1.1557 | 1.1543 | 1.1527 | 1.1475 | β | 1.1423 | 1.1304 | 1.1251 | -0.0054 |
| Prose / spoken | 1.5606 | 1.5466 | 1.5650 | 1.5383 | 1.5411 | 1.5261 | β | 1.5134 | 1.4989 | 1.4932 | -0.0057 |
All four categories are at their best value at the latest checkpoint. Intervals in which a category got worse: Prose / spoken (64,800 β 67,600, +0.0184); Prose / spoken (70,400 β 73,200, +0.0028).
Progress since the previous checkpoint (same protocol)
Step 96,800 β 107,200 is +10.91B tokens; both runs are Kaggle T4, fp16, full evaluation sets, identical code and items.
| Benchmark | step 96,800 | step 107,200 | Ξ (pts) | std. error of Ξ |
|---|---|---|---|---|
HellaSwag (acc_norm) |
33.89% | 34.16% | +0.27 | Β±0.67 |
ARC-Easy (acc_norm) |
46.04% | 46.89% | +0.84 | Β±1.45 |
ARC-Challenge (acc_norm) |
26.79% | 26.28% | -0.51 | Β±1.82 |
PIQA (acc_norm) |
61.21% | 60.72% | -0.49 | Β±1.61 |
WinoGrande (acc) |
50.67% | 51.22% | +0.55 | Β±1.99 |
OpenBookQA (acc_norm) |
28.20% | 28.40% | +0.20 | Β±2.85 |
BoolQ (acc) |
61.59% | 61.50% | -0.09 | Β±1.20 |
SciQ (acc) |
77.90% | 80.30% | +2.40 | Β±1.82 |
LAMBADA (OpenAI) (acc) |
28.57% | 28.49% | -0.08 | Β±0.89 |
MMLU (5-shot) (acc) |
27.35% | 26.89% | -0.46 | Β±0.53 |
RACE (acc_norm) |
33.54% | 34.27% | +0.73 | Β±0.95 |
COPA (acc) |
56.00% | 55.00% | -1.00 | Β±7.03 |
LogiQA (acc_norm) |
25.96% | 24.88% | -1.08 | Β±2.41 |
WSC273 (acc) |
54.58% | 56.41% | +1.83 | Β±4.25 |
TruthfulQA MC1 (acc) |
18.60% | 18.73% | +0.12 | Β±1.93 |
Arithmetic (acc) |
0.12% | 5.31% | +5.19 | Β±0.19 |
| WikiText-2 bits/byte | 1.018 | 1.011 | -0.007305 | β |
| WikiText-2 perplexity | 23.92 | 23.38 | -0.5385 | β |
| LAMBADA perplexity | 29.33 | 29.48 | +0.152 | β |
Ξ is on the conventional metric for each task. Bold marks a change larger than two standard errors of the difference; unbolded changes are inside the noise floor. The error treats the two runs as independent, which is conservative since they score identical items.
- Accuracy tasks: 9 improved, 7 got worse, 0 unchanged (sign test p = 0.80). Moves larger than two standard errors: Arithmetic (+5.19).
- WikiText-2 bits/byte: 1.018 β 1.011 (-0.007305, better).
- LAMBADA perplexity: 29.33 β 29.48 (+0.152, worse).
- Arithmetic (exact match): 0.12% β 5.31%.
- The Pile: 21 of 22 domains improved; worse: EuroParl (+0.0265). Token-weighted bits/byte 1.0216 β 1.0136 (-0.0080).
Head-to-head vs google/gemma-4-E2B
Zenyx V3 at step 107,200 (1.58B total, MoE, ~81B tokens) -- measured on a Kaggle T4 in fp16 against Google's 5.10B-parameter Gemma 4 E2B, which was trained on a corpus larger by roughly two orders of magnitude.
Both models were scored by one copy of the task code on the same source text, with bits/byte normalised by that text's UTF-8 byte length -- the only metric here that is legitimate across a 129,280-entry vocab and a 262,144-entry one. Perplexity is per-token and is deliberately not reported. Zenyx was re-scored on a Kaggle T4 in fp16 for this update, while the Gemma numbers are the earlier bf16 L4 run on the same text -- same task code and byte-normalised metric, different hardware/precision on the Zenyx side (see the T4 cross-check section for how large that effect is).
The Pile, all 22 domains (bits/byte, LOWER is better)
| domain | Zenyx V3 | Gemma 4 E2B | margin | winner | bytes |
|---|---|---|---|---|---|
| DM Mathematics | 1.3433 | 1.8167 | -0.4734 | Zenyx | 400,000 |
| OpenSubtitles | 1.3821 | 1.5502 | -0.1680 | Zenyx | 400,032 |
| ArXiv | 0.8552 | 1.0177 | -0.1626 | Zenyx | 400,820 |
| HackerNews | 1.2807 | 1.4344 | -0.1537 | Zenyx | 239,283 * |
| Github | 0.5647 | 0.6202 | -0.0555 | Zenyx | 400,340 |
| NIH ExPorter | 0.8620 | 0.8930 | -0.0309 | Zenyx | 220,665 * |
| Gutenberg (PG-19) | 1.2534 | 1.2833 | -0.0299 | Zenyx | 409,150 |
| StackExchange | 0.9622 | 0.9862 | -0.0240 | Zenyx | 400,410 |
| Pile-CC | 1.0809 | 1.0905 | -0.0096 | Zenyx | 403,210 |
| PubMed Abstracts | 0.8617 | 0.8704 | -0.0087 | Zenyx | 400,105 |
| Enron Emails | 1.2207 | 1.2273 | -0.0065 | Zenyx | 57,066 * |
| PubMed Central | 0.6395 | 0.6342 | +0.0053 | Gemma | 402,138 |
| USPTO Backgrounds | 0.7669 | 0.7603 | +0.0066 | Gemma | 400,407 |
| Wikipedia (en) | 0.9772 | 0.9648 | +0.0124 | Gemma | 400,898 |
| BookCorpus2 | 1.1745 | 1.1393 | +0.0352 | Gemma | 400,022 |
| PhilPapers | 1.3350 | 1.2804 | +0.0545 | Gemma | 158,869 * |
| OpenWebText2 | 1.1288 | 1.0257 | +0.1031 | Gemma | 410,197 |
| Ubuntu IRC | 1.7138 | 1.5624 | +0.1514 | Gemma | 43,987 * |
| Books3 | 1.2365 | 1.0838 | +0.1527 | Gemma | 400,983 |
| FreeLaw | 0.9705 | 0.8013 | +0.1692 | Gemma | 402,135 |
| EuroParl | 1.9936 | 1.1373 | +0.8563 | Gemma | 68,106 * |
| YoutubeSubtitles | 1.8207 | 0.8918 | +0.9289 | Gemma | 191,675 * |
Zenyx wins 11 of 22 domains. Byte-weighted over all of them, Zenyx 1.0629 vs Gemma 1.0586 (+0.0043) -- Gemma ahead overall.
Rows marked * rest on under 250,000 bytes and carry correspondingly more noise. Excluding them (15 domains), the weighted result is Zenyx 1.0136 vs Gemma 1.0430 (-0.0295), the opposite direction. That subset is not the headline: dropping the thin domains removes 2 of Zenyx's 4 worst losses while removing 3 of its 11 wins, so it does not flatter this model.
Science accuracy (higher is better)
| benchmark | metric | Zenyx V3 | Gemma 4 E2B | chance | winner |
|---|---|---|---|---|---|
| ARC-Challenge | acc_norm | 27.39% | 52.65% | 25.0% | Gemma |
| SciQ | acc | 79.90% | 96.90% | 25.0% | Gemma |
What it says. Zenyx's largest wins: DM Mathematics (-0.4734), OpenSubtitles (-0.1680), ArXiv (-0.1626). Its largest losses: YoutubeSubtitles (+0.9289), EuroParl (+0.8563), FreeLaw (+0.1692). Gemma is built multilingual while this model's corpus is English and excludes Chinese outright, which is the expected source of the multilingual losses. On the science tasks Zenyx is at or near chance on ARC-Challenge.
A suite that flattered this model would not report chance-level performance where that is the truth, which is what makes any wins credible.
Head-to-head vs XHToken/Spark-X2.5-1.7B-Base
A second, more size-matched comparison: Spark is 1.71B params against this model's 1.58B, and its vocab (131,072) is close to this model's (129,280) -- so, unlike the Gemma comparison, tokenizer size is not a meaningful confound here either. Zenyx V3 at step 107,200 (1.58B total, MoE, ~81B tokens) -- measured on a Kaggle T4 in fp16.
Same methodology as the Gemma comparison: one copy of the task code, same source text, bits/byte normalised by shared UTF-8 bytes. Zenyx was re-scored on a Kaggle T4 in fp16 for this update; the Spark numbers are the earlier bf16 L4 run on the same text.
The Pile, all 22 domains (bits/byte, LOWER is better)
| domain | Zenyx V3 | Spark X2.5 | margin | winner | bytes |
|---|---|---|---|---|---|
| OpenSubtitles | 1.3821 | 1.2261 | +0.1560 | Spark | 400,032 |
| PubMed Central | 0.6395 | 0.4622 | +0.1773 | Spark | 402,138 |
| Github | 0.5647 | 0.3841 | +0.1805 | Spark | 400,340 |
| USPTO Backgrounds | 0.7669 | 0.5803 | +0.1866 | Spark | 400,407 |
| DM Mathematics | 1.3433 | 1.1500 | +0.1932 | Spark | 400,000 |
| NIH ExPorter | 0.8620 | 0.6654 | +0.1966 | Spark | 220,665 * |
| ArXiv | 0.8552 | 0.6453 | +0.2099 | Spark | 400,820 |
| Wikipedia (en) | 0.9772 | 0.7432 | +0.2340 | Spark | 400,898 |
| PubMed Abstracts | 0.8617 | 0.6261 | +0.2356 | Spark | 400,105 |
| Pile-CC | 1.0809 | 0.8381 | +0.2429 | Spark | 403,210 |
| HackerNews | 1.2807 | 1.0211 | +0.2596 | Spark | 239,283 * |
| Enron Emails | 1.2207 | 0.9547 | +0.2660 | Spark | 57,066 * |
| StackExchange | 0.9622 | 0.6850 | +0.2772 | Spark | 400,410 |
| BookCorpus2 | 1.1745 | 0.8965 | +0.2780 | Spark | 400,022 |
| OpenWebText2 | 1.1288 | 0.8192 | +0.3096 | Spark | 410,197 |
| Gutenberg (PG-19) | 1.2534 | 0.9352 | +0.3182 | Spark | 409,150 |
| FreeLaw | 0.9705 | 0.6167 | +0.3538 | Spark | 402,135 |
| Books3 | 1.2365 | 0.8726 | +0.3639 | Spark | 400,983 |
| PhilPapers | 1.3350 | 0.9652 | +0.3697 | Spark | 158,869 * |
| Ubuntu IRC | 1.7138 | 1.1782 | +0.5356 | Spark | 43,987 * |
| YoutubeSubtitles | 1.8207 | 0.7579 | +1.0627 | Spark | 191,675 * |
| EuroParl | 1.9936 | 0.8732 | +1.1204 | Spark | 68,106 * |
Spark wins 22 of 22 domains. Byte-weighted,
Zenyx 1.0629 vs Spark 0.7806 (+0.2822) -- Spark ahead overall.
Rows marked * rest on under 250,000 bytes.
ARC-Challenge and LAMBADA (higher acc is better)
| benchmark | metric | Zenyx V3 | Spark X2.5 | chance | winner |
|---|---|---|---|---|---|
| ARC-Challenge | acc_norm | 27.39% | 45.31% | 25.0% | Spark |
| LAMBADA | acc | 28.45% | 58.53% | 0.0% | Spark |
What it says. Zenyx's smallest gaps to (or wins over) Spark: OpenSubtitles (+0.1560), PubMed Central (+0.1773), Github (+0.1805). On ARC-Challenge the two are 27.4% vs 45.3% (chance 25%). LAMBADA perplexity: Zenyx 29.51 vs Spark 4.24.
Hardware Serving Benchmarks (NVIDIA L4, 24 GB)
Measured with the JAX/Flax serving loop: static shape pre-allocation, bucketed prefill and GPU-native sampling.
| Metric | Value | Notes |
|---|---|---|
| Decode speed | 68.5 tok/s | steady-state autoregressive decode |
| Warm prefill | ~20 ms | short prompt, shape already compiled |
| Checkpoint load | ~26 s | params β GPU, from local cache |
| Active VRAM | ~5.0 GB | of 24 GB |
Cold shapes pay a one-off JIT compile (tens of seconds) the first time a new (prompt length, max tokens) pair is seen; warm requests are the numbers above.
Inference Example
The weights of the latest checkpoint are checkpoints/step_112400/params.pkl (bfloat16, pickled Flax parameter tree); modeling_zenyx_v3.py in this repository holds the architecture and a generator with a KV cache and jit-compiled decoding. It runs on CPU, GPU and TPU (the first call compiles for a few minutes; on a plain CPU it generates about 0.7 tokens per second, a GPU or TPU is much faster).
pip install jax flax safetensors transformers huggingface_hub
from huggingface_hub import hf_hub_download
hf_hub_download("Arko007/Zenyx-V3-Base", "modeling_zenyx_v3.py", local_dir=".") # the model code
from modeling_zenyx_v3 import ZenyxGenerator
gen = ZenyxGenerator("Arko007/Zenyx-V3-Base", params_file="checkpoints/step_112400/params.pkl") # downloads 3.2 GB of weights and the tokenizer
# Base model: give it a prefix to CONTINUE, not an instruction to follow.
print(gen.generate("The capital of France is", max_new_tokens=40, temperature=0.7, top_p=0.9, top_k=50, repetition_penalty=1.1, seed=0))
Output of exactly this code (CPU, seed 0): "The capital of France is Paris. In the French language, it means "city". The word in English refers to a place where people go for work. French: la maison ..." β a base model, so expect fluent but unreliable continuations. For the chat / instruction-tuned version use Zenyx-V3-SFT.
Evaluation Reproducibility
Latest-checkpoint benchmarks ran in the project's Kaggle evaluation notebook (JAX/Flax, one NVIDIA T4, fp16, length-bucketed batching); the earlier bf16 columns were produced by
modal_base_evals.py on a single NVIDIA L4. Task formats follow the lm-evaluation-harness conventions (prompt templates, acc / acc_norm definitions and answer-key handling),
so the numbers are broadly comparable to published base-model results, though this is an independent implementation rather than a harness run. The head-to-head tables score
all models with one copy of the task code on the same source text; bits per byte is normalised by that text's UTF-8 byte length so it is valid across tokenizers.
Limitations
- Pretraining is incomplete β the model will change substantially with more tokens.
- Not instruction-tuned, not RLHF'd, and not safety-filtered. Outputs may be factually wrong, biased, or nonsensical.
- Trained predominantly on English text, code, mathematics and synthetic reasoning data; other languages are not supported.