Zenyx V3 Base (1.5B Mixture-of-Experts)

Zenyx V3 is an efficient 1.5B-parameter Mixture-of-Experts (MoE) foundation model built for low-latency inference and high throughput. It is written from scratch in JAX/Flax and trained on TPU v5e-8.

This is a BASE model β€” it is not instruction-tuned. It completes text; it does not follow instructions or hold a conversation. Prompt it with a prefix to continue ("The capital of France is"), not with a request ("Explain gravity"). Pretraining is still in progress; SFT/chat variants will follow.

Latest checkpoint: step 112,000 (core tests re-run, next section; step 111,200 below it) Β· the full benchmark tables further down were measured at step 107,200 Β· 81.0B tokens seen at step 107,200

Update β€” step 112,000 (core tests re-run)

Same code, items and settings as the step-111,200 and step-107,200 tables (0-shot, full evaluation sets, log-likelihood ranking, fp16, one T4); this run was on a Google Colab T4 instead of a Kaggle T4. Only these tests were re-run; every other table in this card is still step 107,200.

Benchmark step 112,000 step 111,200 step 107,200 n
ARC-Easy (acc / acc_norm) 51.73% Β± 1.03 / 47.01% 50.84% / 46.21% 52.15% / 46.89% 2,376
ARC-Challenge (acc / acc_norm) 23.98% Β± 1.25 / 27.90% 23.04% / 26.02% 22.78% / 26.28% 1,172
SciQ (acc / acc_norm) 79.70% Β± 1.27 / 71.70% 80.00% / 71.30% 80.30% / 72.50% 1,000
Zenyx-Fact1000 (internal, summed log-probability; 95% interval) 80.2% (77.6–82.6) 79.3% (76.7–81.7) 81.1% (78.6–83.4) 1,000
Arithmetic, 14,000 generated problems 2.18% 3.54% 5.31% 14,000

The checkpoint moved sideways on these tests (all differences within about one standard error). Fact1000 at step 112,000 without the arithmetic items: 79.4%; under the other two numerical configurations of the same script: 80.6% (tight batching) and 81.9% (native bf16). Arithmetic is still unstable and weak in this plain prompt format; per operation at step 112,000: 2-digit addition 4.20%, 2-digit subtraction 9.35%, 2-digit multiplication 1.70%, every 3-, 4- and 5-digit operation 0.00%.

Update β€” step 111,200 (core tests re-run)

Same protocol as the step-107,200 tables below (Kaggle T4, fp16, 0-shot, full evaluation sets, log-likelihood ranking, identical code and items). Only these tests were re-run; every other table in this card is still step 107,200.

Benchmark step 111,200 step 107,200 n
ARC-Easy (acc / acc_norm) 50.84% Β± 1.03 / 46.21% 52.15% / 46.89% 2,376
ARC-Challenge (acc / acc_norm) 23.04% Β± 1.23 / 26.02% 22.78% / 26.28% 1,172
SciQ (acc / acc_norm) 80.00% Β± 1.26 / 71.30% 80.30% / 72.50% 1,000
Zenyx-Fact1000 (internal, 1,000 four-way items, summed log-probability; 95% interval) 79.3% (76.7–81.7) 81.1% (78.6–83.4) 1,000
Arithmetic, 14,000 generated problems (format of the Arithmetic section) 3.54% 5.31% 14,000

All differences in the first four rows are within about one to one-and-a-half standard errors of the difference: the checkpoint moved sideways on these tests. Fact1000 at step 111,200 under the other two numerical configurations of the same script: 80.0% (tight batching) and 82.5% (native bf16); the number above is the same configuration as the step-107,200 figure.

Arithmetic is unstable between checkpoints (0.12% at step 96,800, 5.31% at 107,200, 3.54% at 111,200). At step 111,200 per operation: 2-digit addition 3.90%, 2-digit subtraction 17.45%, 3-digit addition 0.00%, 3-digit subtraction 0.30%, 4-digit and 5-digit addition 0.00%, 2-digit multiplication 3.10%. The same 14,000 problems in other prompt formats at step 111,200: X + Y = 4.90%, Q: ... A: 13.09%, three worked examples in the prompt 15.86%, Question: What is X plus Y? / Answer: 3.39%.

Model Architecture

  • Sparse Mixture-of-Experts: 12 routed experts + 1 shared expert, exactly 2 routed experts active per token. The router is a softmax top-2 gate with Gaussian exploration noise during training (std 0.1), expert capacity factor 1.5, and load-balance + router z-loss auxiliary terms.
  • Multi-Query Attention (MQA): a single shared key head and a single shared value head (128-dim each) are broadcast to all 12 query heads, so the KV cache holds only 2 Γ— 128 values per layer per token. Queries use a rank-384 low-rank projection and the output a grouped low-rank projection (4 groups Γ— rank 256); RoPE is applied to 32 of the 128 dimensions of each head. (The attention class in the training code is named MLA for historical reasons; there is no latent KV compression.)
  • Hyper-Connections: three residual streams mixed by Sinkhorn-normalised (3 iterations) routing matrices, for gradient stability at scale.
  • Multi-Token Prediction (MTP): one auxiliary prediction head during training.
  • Context: pretrained at up to 4,096 tokens (progressive 2,048 β†’ 4,096) with plain RoPE (ΞΈ = 10,000). YaRN scaling factors are precomputed in the code so the context can be extended at inference time beyond the trained length.
Total parameters ~1.58B
Active parameters / token ~0.38B (non-embedding; plus 0.2B tied embedding/output matrix)
Layers 16 (2 dense + 14 MoE)
Hidden size 1,536
Attention heads 12 (head dim 128)
Vocabulary 129,280
Precision bfloat16

Benchmarks β€” step 107,200 (81.0B tokens)

All tasks use the standard base-model protocol: the model scores the log-likelihood of every candidate continuation and the highest-scoring one is the answer, so nothing is generated and the numbers do not depend on instruction following. 0-shot, full evaluation sets (MMLU 5-shot). acc_norm normalises each continuation's log-likelihood by its length in characters and is the headline wherever candidates differ in length; bold marks the conventional metric per task.

Evaluation hardware. These results were measured on a Kaggle T4 in fp16 (every bf16 parameter cast to fp16; gate/router parameters kept in fp32), because a T4 has no bf16 tensor cores. Earlier checkpoints are shown only as comparison columns: step 96,800 was measured the same way; step 86,400 (and the trajectory before it) in bf16 on an L4. Differences against the bf16/L4 column include a hardware/precision effect. Native-bf16 results for this checkpoint will replace these when available.

Benchmark acc acc_norm Random Ξ” vs random n @96,800 (T4) @86,400 (bf16/L4) What it measures
HellaSwag 30.72% Β± 0.46 34.16% Β± 0.47 25.0% +9.2 10,042 33.89% 33.67% Commonsense sentence completion
ARC-Easy 52.15% Β± 1.02 46.89% Β± 1.02 25.0% +21.9 2,376 46.04% 46.25% Grade-school science questions
ARC-Challenge 22.78% Β± 1.23 26.28% Β± 1.29 25.0% +1.3 1,172 26.79% 26.45% Hard grade-school science questions
PIQA 61.43% Β± 1.14 60.72% Β± 1.14 50.0% +10.7 1,838 61.21% 61.26% Physical commonsense reasoning
WinoGrande 51.22% Β± 1.40 β€” 50.0% +1.2 1,267 50.67% 49.72% Pronoun resolution / coreference
OpenBookQA 19.20% Β± 1.76 28.40% Β± 2.02 25.0% +3.4 500 28.20% 31.00% Elementary science with open book
BoolQ 61.50% Β± 0.85 62.17% Β± 0.85 62.2% -0.7 3,270 61.59% 60.76% Yes/no reading comprehension
SciQ 80.30% Β± 1.26 72.50% Β± 1.41 25.0% +55.3 1,000 77.90% 78.20% Science exam questions with support
LAMBADA (OpenAI) 28.49% Β± 0.63 β€” 0.0% +28.5 5,153 28.57% 25.64% Long-range last-word prediction
MMLU (5-shot) 26.89% Β± 0.37 β€” 25.0% +1.9 14,042 27.35% 25.78% 57 subjects of academic knowledge
RACE 31.80% Β± 0.66 34.27% Β± 0.68 25.0% +9.3 4,934 33.54% 32.81% Exam reading comprehension
CommonsenseQA 28.58% Β± 1.29 30.38% Β± 1.32 20.0% +10.4 1,221 β€” 31.04% 5-choice commonsense (random = 20%)
COPA 55.00% Β± 4.97 58.00% Β± 4.94 50.0% +5.0 100 56.00% 62.00% Causal reasoning
LogiQA 20.89% Β± 1.59 24.88% Β± 1.69 25.0% -0.1 651 25.96% 25.96% Logical deduction
WSC273 56.41% Β± 3.00 β€” 50.0% +6.4 273 54.58% 53.48% Winograd coreference
TruthfulQA MC1 18.73% Β± 1.36 β€” 22.8% -4.1 817 18.60% 19.58% Resistance to common misconceptions
Arithmetic 5.31% Β± 0.19 β€” 0.0% +5.3 14,000 0.12% 0.09% 2-5 digit add/sub/mul, generated in-harness

(The two right-hand columns show the conventional metric at the earlier checkpoint.) BoolQ's majority-class baseline is 62.2%, so a score near it is not evidence of comprehension.

Language modelling

Metric @107,200 @96,800 (T4) @86,400 (bf16/L4)
WikiText-2 bits/byte 1.0111 1.0184 1.0279
WikiText-2 token perplexity 23.38 23.92 24.64
WikiText-2 word perplexity 42.86 44.04 45.62
LAMBADA perplexity 29.48 29.33 30.35
Pile, 22 domains, token-weighted bits/byte 1.0136 1.0216 1.0304

WikiText-2 and the Pile use a rolling 1024-token window at stride 512, so every counted token has at least 512 tokens of left context and is counted exactly once (disjoint windows would inflate perplexity by ~15%).

Comparison with previous checkpoints

Headline metric per task (acc_norm where the task has it, otherwise acc). Columns 63,200–86,400 were measured in bf16 on an L4 and 96,800, 107,200 on a Kaggle T4 in fp16, so the last two differences are not the same kind: Ξ” last interval compares two runs of the same protocol; Ξ” since 63,200 mixes the two protocols and therefore includes a precision/hardware effect. The pretraining data mixture was also changed partway through this range (code weight raised, several synthetic sources cut), so the columns are not a tokens-only progression.

Tokens seen 34.8B 36.5B 39.4B 42.4B 45.3B 49.1B 54.9B 59.1B 70.0B 81.0B
Protocol bf16/L4 bf16/L4 bf16/L4 bf16/L4 bf16/L4 bf16/L4 bf16/L4 bf16/L4 T4 fp16 T4 fp16
Benchmark 63,200 64,800 67,600 70,400 73,200 76,800 82,400 86,400 96,800 107,200 Ξ” last interval (same protocol) Ξ” since 63,200 (mixed protocol)
HellaSwag 32.22% 32.66% 32.84% 32.51% 32.72% 33.08% 33.41% 33.67% 33.89% 34.16% +0.27 +1.93
ARC-Easy 44.53% 45.08% 44.91% 45.58% 46.09% 44.82% 46.34% 46.25% 46.04% 46.89% +0.84 +2.36
ARC-Challenge 25.77% 25.51% 26.02% 25.00% 25.94% 25.09% 25.26% 26.45% 26.79% 26.28% -0.51 +0.51
PIQA 59.79% 59.85% 61.53% 59.96% 60.23% 59.09% 61.26% 61.26% 61.21% 60.72% -0.49 +0.92
WinoGrande 49.49% 50.51% 51.07% 49.57% 49.25% 50.12% 50.59% 49.72% 50.67% 51.22% +0.55 +1.74
OpenBookQA 30.00% 28.00% 29.80% 29.00% 28.40% 29.20% 30.40% 31.00% 28.20% 28.40% +0.20 -1.60
BoolQ 60.55% 60.83% 61.80% 62.14% 60.83% 60.92% 61.47% 60.76% 61.59% 61.50% -0.09 +0.95
SciQ 75.10% 76.10% 75.70% 76.30% 76.00% 76.00% 77.40% 78.20% 77.90% 80.30% +2.40 +5.20
LAMBADA (OpenAI) 25.50% 25.42% 24.94% 26.96% 26.57% 24.68% 25.89% 25.64% 28.57% 28.49% -0.08 +2.99
MMLU (5-shot) 26.07% 26.71% 25.77% 26.01% 25.26% 26.48% 26.47% 25.78% 27.35% 26.89% -0.46 +0.82
RACE 32.79% 32.77% 32.96% 32.85% 33.58% 33.46% 33.46% 32.81% 33.54% 34.27% +0.73 +1.48
CommonsenseQA 29.57% 29.57% 29.40% 30.55% 30.47% 30.88% 30.14% 31.04% β€” 30.38% β€” +0.82
COPA 58.00% 58.00% 61.00% 59.00% 60.00% 57.00% 59.00% 62.00% 56.00% 55.00% -1.00 -3.00
LogiQA 25.81% 25.19% 25.65% 25.81% 25.65% 23.81% 25.65% 25.96% 25.96% 24.88% -1.08 -0.92
WSC273 54.95% 52.38% 52.75% 51.28% 53.11% 56.41% 54.21% 53.48% 54.58% 56.41% +1.83 +1.47
TruthfulQA MC1 19.22% 20.20% 19.58% 19.83% 20.44% 19.58% 19.34% 19.58% 18.60% 18.73% +0.12 -0.49
Arithmetic 0.14% 0.55% 0.58% 1.04% 1.91% 0.09% 0.15% 0.09% 0.12% 5.31% +5.19 +5.17
WikiText-2 bits/byte (lower is better) 1.051 1.043 1.045 1.043 1.035 1.036 1.027 1.028 1.018 1.011 -0.007305 -0.03979
LAMBADA perplexity (lower is better) 35.79 36.19 36.84 34.15 34.27 36.46 32.23 30.35 29.33 29.48 +0.152 -6.313
Pile token-weighted bits/byte (lower is better) 1.0613 1.0572 1.0554 1.0488 1.0480 1.0400 β€” 1.0304 1.0216 1.0136 -0.0080 -0.0477

Arithmetic

Exact match on the answer, greedy decoding, GPT-3 prompt format (Question: What is 47 plus 21? / Answer: 68); 2,000 generated problems per operation, the same fixed-seed items at every checkpoint.

Operation step 107,200 step 96,800 (T4) step 86,400 (bf16/L4)
2-digit addition 16.85% 0.30% 0.00%
2-digit subtraction 17.10% 0.45% 0.50%
3-digit addition 0.20% 0.00% 0.00%
3-digit subtraction 0.95% 0.00% 0.05%
4-digit addition 0.00% 0.00% 0.00%
5-digit addition 0.00% 0.00% 0.00%
2-digit multiplication 2.05% 0.10% 0.10%
overall (14,000) 5.307% 0.12% 0.09%

Overall exact-match arithmetic is 5.31% at step 107,200, against 0.12% at 96,800 and 0.09% at 86,400. The strongest operation is 2-digit subtraction (17.10%). Operations that rose by more than 5x and are now above 1%: 2-digit addition, 2-digit subtraction, 2-digit multiplication. Still at exactly 0%: 4-digit addition, 5-digit addition. Strict and lenient scoring are identical for every operation, so the change is in the arithmetic itself and not in answer formatting.

Items are generated in-harness from a fixed seed using this prompt format, because EleutherAI/arithmetic is a loading script with no parquet branch and cannot be fetched under datasets>=3. All checkpoints see byte-identical problems, so the comparison is exact β€” but these numbers are not interchangeable with published EleutherAI/arithmetic results.

Language modelling by genre (The Pile)

Bits per byte on each of the 22 Pile domains (NeelNanda/pile-10k, each domain capped at 1.5M characters), scored with a rolling 1024-token window at stride 512 so every token is predicted with at least 512 tokens of left context and counted exactly once; lower is better. This measures raw prediction rather than multiple-choice ability.

Domain bits/byte @107,200 perplexity tokens @96,800 (T4) @86,400 (bf16/L4) Ξ” vs 96,800
Github 0.5764 3.58 479,656 0.5878 0.5943 -0.0114
PubMed Central 0.7429 14.19 292,334 0.7497 0.7586 -0.0068
USPTO Backgrounds 0.7806 15.53 296,162 0.7851 0.7945 -0.0045
ArXiv 0.8561 7.34 447,478 0.8636 0.8769 -0.0075
NIH ExPorter 0.8620 23.92 41,533 0.8667 0.8769 -0.0047
PubMed Abstracts 0.8752 19.46 306,636 0.8814 0.8924 -0.0062
StackExchange 0.9203 11.94 387,038 0.9284 0.9406 -0.0081
FreeLaw 0.9629 19.34 339,260 0.9715 0.9817 -0.0086
Wikipedia (en) 0.9902 20.54 341,511 0.9931 1.0081 -0.0029
Pile-CC 1.1000 34.00 326,706 1.1049 1.1140 -0.0049
OpenWebText2 1.1295 31.10 348,909 1.1354 1.1469 -0.0059
BookCorpus2 1.1425 32.33 139,043 1.1465 1.1536 -0.0040
Enron Emails 1.2207 20.00 16,119 1.2297 1.2405 -0.0090
Gutenberg (PG-19) 1.2714 33.34 133,371 1.2749 1.2880 -0.0035
HackerNews 1.2807 39.34 57,843 1.2884 1.3000 -0.0077
DM Mathematics 1.3201 7.40 370,912 1.3430 1.3316 -0.0229
Books3 1.3298 33.04 396,623 1.3362 1.3442 -0.0064
PhilPapers 1.3350 46.62 38,263 1.3432 1.3586 -0.0082
OpenSubtitles 1.3390 30.02 234,514 1.3457 1.3527 -0.0067
Ubuntu IRC 1.7138 37.60 14,407 1.7304 1.7427 -0.0166
YoutubeSubtitles 1.8206 110.32 51,428 1.8476 1.8521 -0.0270
EuroParl 1.9935 124.02 19,523 1.9670 2.0156 +0.0265

Against step 96,800 (same protocol) 21 of 22 domains improved; not improved: EuroParl (+0.0265). Against step 86,400 (bf16/L4, different hardware and precision) 22 of 22 are lower. The lowest bits/byte are on Github, PubMed Central, USPTO Backgrounds; the highest on Ubuntu IRC, YoutubeSubtitles, EuroParl.

By content type (mean of the domain values, bits/byte, lower is better; columns are all benchmarked checkpoints, 63,200–86,400 bf16/L4 and the rest T4 fp16):

Category 63,200 64,800 67,600 70,400 73,200 76,800 82,400 86,400 96,800 107,200 Ξ” last interval
Code / technical 0.9558 0.9518 0.9438 0.9341 0.9336 0.9228 β€” 0.9076 0.9016 0.8907 -0.0109
Science / legal 0.8987 0.8950 0.8930 0.8892 0.8880 0.8834 β€” 0.8774 0.8673 0.8608 -0.0066
Web / reference 1.1586 1.1561 1.1557 1.1543 1.1527 1.1475 β€” 1.1423 1.1304 1.1251 -0.0054
Prose / spoken 1.5606 1.5466 1.5650 1.5383 1.5411 1.5261 β€” 1.5134 1.4989 1.4932 -0.0057

All four categories are at their best value at the latest checkpoint. Intervals in which a category got worse: Prose / spoken (64,800 β†’ 67,600, +0.0184); Prose / spoken (70,400 β†’ 73,200, +0.0028).

Progress since the previous checkpoint (same protocol)

Step 96,800 β†’ 107,200 is +10.91B tokens; both runs are Kaggle T4, fp16, full evaluation sets, identical code and items.

Benchmark step 96,800 step 107,200 Ξ” (pts) std. error of Ξ”
HellaSwag (acc_norm) 33.89% 34.16% +0.27 Β±0.67
ARC-Easy (acc_norm) 46.04% 46.89% +0.84 Β±1.45
ARC-Challenge (acc_norm) 26.79% 26.28% -0.51 Β±1.82
PIQA (acc_norm) 61.21% 60.72% -0.49 Β±1.61
WinoGrande (acc) 50.67% 51.22% +0.55 Β±1.99
OpenBookQA (acc_norm) 28.20% 28.40% +0.20 Β±2.85
BoolQ (acc) 61.59% 61.50% -0.09 Β±1.20
SciQ (acc) 77.90% 80.30% +2.40 Β±1.82
LAMBADA (OpenAI) (acc) 28.57% 28.49% -0.08 Β±0.89
MMLU (5-shot) (acc) 27.35% 26.89% -0.46 Β±0.53
RACE (acc_norm) 33.54% 34.27% +0.73 Β±0.95
COPA (acc) 56.00% 55.00% -1.00 Β±7.03
LogiQA (acc_norm) 25.96% 24.88% -1.08 Β±2.41
WSC273 (acc) 54.58% 56.41% +1.83 Β±4.25
TruthfulQA MC1 (acc) 18.60% 18.73% +0.12 Β±1.93
Arithmetic (acc) 0.12% 5.31% +5.19 Β±0.19
WikiText-2 bits/byte 1.018 1.011 -0.007305 β€”
WikiText-2 perplexity 23.92 23.38 -0.5385 β€”
LAMBADA perplexity 29.33 29.48 +0.152 β€”

Ξ” is on the conventional metric for each task. Bold marks a change larger than two standard errors of the difference; unbolded changes are inside the noise floor. The error treats the two runs as independent, which is conservative since they score identical items.

  • Accuracy tasks: 9 improved, 7 got worse, 0 unchanged (sign test p = 0.80). Moves larger than two standard errors: Arithmetic (+5.19).
  • WikiText-2 bits/byte: 1.018 β†’ 1.011 (-0.007305, better).
  • LAMBADA perplexity: 29.33 β†’ 29.48 (+0.152, worse).
  • Arithmetic (exact match): 0.12% β†’ 5.31%.
  • The Pile: 21 of 22 domains improved; worse: EuroParl (+0.0265). Token-weighted bits/byte 1.0216 β†’ 1.0136 (-0.0080).

Head-to-head vs google/gemma-4-E2B

Zenyx V3 at step 107,200 (1.58B total, MoE, ~81B tokens) -- measured on a Kaggle T4 in fp16 against Google's 5.10B-parameter Gemma 4 E2B, which was trained on a corpus larger by roughly two orders of magnitude.

Both models were scored by one copy of the task code on the same source text, with bits/byte normalised by that text's UTF-8 byte length -- the only metric here that is legitimate across a 129,280-entry vocab and a 262,144-entry one. Perplexity is per-token and is deliberately not reported. Zenyx was re-scored on a Kaggle T4 in fp16 for this update, while the Gemma numbers are the earlier bf16 L4 run on the same text -- same task code and byte-normalised metric, different hardware/precision on the Zenyx side (see the T4 cross-check section for how large that effect is).

The Pile, all 22 domains (bits/byte, LOWER is better)

domain Zenyx V3 Gemma 4 E2B margin winner bytes
DM Mathematics 1.3433 1.8167 -0.4734 Zenyx 400,000
OpenSubtitles 1.3821 1.5502 -0.1680 Zenyx 400,032
ArXiv 0.8552 1.0177 -0.1626 Zenyx 400,820
HackerNews 1.2807 1.4344 -0.1537 Zenyx 239,283 *
Github 0.5647 0.6202 -0.0555 Zenyx 400,340
NIH ExPorter 0.8620 0.8930 -0.0309 Zenyx 220,665 *
Gutenberg (PG-19) 1.2534 1.2833 -0.0299 Zenyx 409,150
StackExchange 0.9622 0.9862 -0.0240 Zenyx 400,410
Pile-CC 1.0809 1.0905 -0.0096 Zenyx 403,210
PubMed Abstracts 0.8617 0.8704 -0.0087 Zenyx 400,105
Enron Emails 1.2207 1.2273 -0.0065 Zenyx 57,066 *
PubMed Central 0.6395 0.6342 +0.0053 Gemma 402,138
USPTO Backgrounds 0.7669 0.7603 +0.0066 Gemma 400,407
Wikipedia (en) 0.9772 0.9648 +0.0124 Gemma 400,898
BookCorpus2 1.1745 1.1393 +0.0352 Gemma 400,022
PhilPapers 1.3350 1.2804 +0.0545 Gemma 158,869 *
OpenWebText2 1.1288 1.0257 +0.1031 Gemma 410,197
Ubuntu IRC 1.7138 1.5624 +0.1514 Gemma 43,987 *
Books3 1.2365 1.0838 +0.1527 Gemma 400,983
FreeLaw 0.9705 0.8013 +0.1692 Gemma 402,135
EuroParl 1.9936 1.1373 +0.8563 Gemma 68,106 *
YoutubeSubtitles 1.8207 0.8918 +0.9289 Gemma 191,675 *

Zenyx wins 11 of 22 domains. Byte-weighted over all of them, Zenyx 1.0629 vs Gemma 1.0586 (+0.0043) -- Gemma ahead overall.

Rows marked * rest on under 250,000 bytes and carry correspondingly more noise. Excluding them (15 domains), the weighted result is Zenyx 1.0136 vs Gemma 1.0430 (-0.0295), the opposite direction. That subset is not the headline: dropping the thin domains removes 2 of Zenyx's 4 worst losses while removing 3 of its 11 wins, so it does not flatter this model.

Science accuracy (higher is better)

benchmark metric Zenyx V3 Gemma 4 E2B chance winner
ARC-Challenge acc_norm 27.39% 52.65% 25.0% Gemma
SciQ acc 79.90% 96.90% 25.0% Gemma

What it says. Zenyx's largest wins: DM Mathematics (-0.4734), OpenSubtitles (-0.1680), ArXiv (-0.1626). Its largest losses: YoutubeSubtitles (+0.9289), EuroParl (+0.8563), FreeLaw (+0.1692). Gemma is built multilingual while this model's corpus is English and excludes Chinese outright, which is the expected source of the multilingual losses. On the science tasks Zenyx is at or near chance on ARC-Challenge.

A suite that flattered this model would not report chance-level performance where that is the truth, which is what makes any wins credible.

Head-to-head vs XHToken/Spark-X2.5-1.7B-Base

A second, more size-matched comparison: Spark is 1.71B params against this model's 1.58B, and its vocab (131,072) is close to this model's (129,280) -- so, unlike the Gemma comparison, tokenizer size is not a meaningful confound here either. Zenyx V3 at step 107,200 (1.58B total, MoE, ~81B tokens) -- measured on a Kaggle T4 in fp16.

Same methodology as the Gemma comparison: one copy of the task code, same source text, bits/byte normalised by shared UTF-8 bytes. Zenyx was re-scored on a Kaggle T4 in fp16 for this update; the Spark numbers are the earlier bf16 L4 run on the same text.

The Pile, all 22 domains (bits/byte, LOWER is better)

domain Zenyx V3 Spark X2.5 margin winner bytes
OpenSubtitles 1.3821 1.2261 +0.1560 Spark 400,032
PubMed Central 0.6395 0.4622 +0.1773 Spark 402,138
Github 0.5647 0.3841 +0.1805 Spark 400,340
USPTO Backgrounds 0.7669 0.5803 +0.1866 Spark 400,407
DM Mathematics 1.3433 1.1500 +0.1932 Spark 400,000
NIH ExPorter 0.8620 0.6654 +0.1966 Spark 220,665 *
ArXiv 0.8552 0.6453 +0.2099 Spark 400,820
Wikipedia (en) 0.9772 0.7432 +0.2340 Spark 400,898
PubMed Abstracts 0.8617 0.6261 +0.2356 Spark 400,105
Pile-CC 1.0809 0.8381 +0.2429 Spark 403,210
HackerNews 1.2807 1.0211 +0.2596 Spark 239,283 *
Enron Emails 1.2207 0.9547 +0.2660 Spark 57,066 *
StackExchange 0.9622 0.6850 +0.2772 Spark 400,410
BookCorpus2 1.1745 0.8965 +0.2780 Spark 400,022
OpenWebText2 1.1288 0.8192 +0.3096 Spark 410,197
Gutenberg (PG-19) 1.2534 0.9352 +0.3182 Spark 409,150
FreeLaw 0.9705 0.6167 +0.3538 Spark 402,135
Books3 1.2365 0.8726 +0.3639 Spark 400,983
PhilPapers 1.3350 0.9652 +0.3697 Spark 158,869 *
Ubuntu IRC 1.7138 1.1782 +0.5356 Spark 43,987 *
YoutubeSubtitles 1.8207 0.7579 +1.0627 Spark 191,675 *
EuroParl 1.9936 0.8732 +1.1204 Spark 68,106 *

Spark wins 22 of 22 domains. Byte-weighted, Zenyx 1.0629 vs Spark 0.7806 (+0.2822) -- Spark ahead overall. Rows marked * rest on under 250,000 bytes.

ARC-Challenge and LAMBADA (higher acc is better)

benchmark metric Zenyx V3 Spark X2.5 chance winner
ARC-Challenge acc_norm 27.39% 45.31% 25.0% Spark
LAMBADA acc 28.45% 58.53% 0.0% Spark

What it says. Zenyx's smallest gaps to (or wins over) Spark: OpenSubtitles (+0.1560), PubMed Central (+0.1773), Github (+0.1805). On ARC-Challenge the two are 27.4% vs 45.3% (chance 25%). LAMBADA perplexity: Zenyx 29.51 vs Spark 4.24.


Hardware Serving Benchmarks (NVIDIA L4, 24 GB)

Measured with the JAX/Flax serving loop: static shape pre-allocation, bucketed prefill and GPU-native sampling.

Metric Value Notes
Decode speed 68.5 tok/s steady-state autoregressive decode
Warm prefill ~20 ms short prompt, shape already compiled
Checkpoint load ~26 s params β†’ GPU, from local cache
Active VRAM ~5.0 GB of 24 GB

Cold shapes pay a one-off JIT compile (tens of seconds) the first time a new (prompt length, max tokens) pair is seen; warm requests are the numbers above.


Inference Example

The weights of the latest checkpoint are checkpoints/step_112400/params.pkl (bfloat16, pickled Flax parameter tree); modeling_zenyx_v3.py in this repository holds the architecture and a generator with a KV cache and jit-compiled decoding. It runs on CPU, GPU and TPU (the first call compiles for a few minutes; on a plain CPU it generates about 0.7 tokens per second, a GPU or TPU is much faster).

pip install jax flax safetensors transformers huggingface_hub
from huggingface_hub import hf_hub_download
hf_hub_download("Arko007/Zenyx-V3-Base", "modeling_zenyx_v3.py", local_dir=".")      # the model code

from modeling_zenyx_v3 import ZenyxGenerator
gen = ZenyxGenerator("Arko007/Zenyx-V3-Base", params_file="checkpoints/step_112400/params.pkl")   # downloads 3.2 GB of weights and the tokenizer

# Base model: give it a prefix to CONTINUE, not an instruction to follow.
print(gen.generate("The capital of France is", max_new_tokens=40, temperature=0.7, top_p=0.9, top_k=50, repetition_penalty=1.1, seed=0))

Output of exactly this code (CPU, seed 0): "The capital of France is Paris. In the French language, it means "city". The word in English refers to a place where people go for work. French: la maison ..." β€” a base model, so expect fluent but unreliable continuations. For the chat / instruction-tuned version use Zenyx-V3-SFT.


Evaluation Reproducibility

Latest-checkpoint benchmarks ran in the project's Kaggle evaluation notebook (JAX/Flax, one NVIDIA T4, fp16, length-bucketed batching); the earlier bf16 columns were produced by modal_base_evals.py on a single NVIDIA L4. Task formats follow the lm-evaluation-harness conventions (prompt templates, acc / acc_norm definitions and answer-key handling), so the numbers are broadly comparable to published base-model results, though this is an independent implementation rather than a harness run. The head-to-head tables score all models with one copy of the task code on the same source text; bits per byte is normalised by that text's UTF-8 byte length so it is valid across tokenizers.

Limitations

  • Pretraining is incomplete β€” the model will change substantially with more tokens.
  • Not instruction-tuned, not RLHF'd, and not safety-filtered. Outputs may be factually wrong, biased, or nonsensical.
  • Trained predominantly on English text, code, mathematics and synthetic reasoning data; other languages are not supported.
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Arko007/Zenyx-V3-Base

Finetunes
1 model