Model Submission Request: MultivexAI Plyx-15M & Supra-1.6-50M
#14
by MultivexAI - opened
Benchmark Scores Summary
Side-by-side zero-shot comparison between both MultivexAI models (with exact standard errors extracted from raw evaluation logs).
| Task | Metric | Plyx-15M (~16.1M) | Supra-1.6-50M (~51.8M) |
|---|---|---|---|
| MMLU | acc | 22.95% (±0.35%) | 22.94% (±0.35%) |
| HellaSwag | acc_norm | 25.76% (±0.44%) | 29.66% (±0.46%) |
| ARC-Easy | acc_norm | 30.85% (±0.95%) | 43.18% (±1.02%) |
| ARC-Challenge | acc_norm | 21.33% (±1.20%) | 25.60% (±1.28%) |
| PIQA | acc | 55.33% (±1.16%) | 60.61% (±1.14%) |
| SciQ | acc_norm | 43.80% (±1.57%) | 66.00% (±1.50%) |
| WinoGrande | acc | 51.22% (±1.40%) | 52.41% (±1.40%) |
| OpenBookQA | acc_norm | 25.80% (±1.96%) | 27.20% (±1.99%) |
| ArithMark-2.0 | acc | 25.32% (±0.87%) | 30.52% (±0.92%) |
Model Breakdown: MultivexAI/Plyx-15M
- Model Link: https://huggingface.co/MultivexAI/Plyx-15M
- Arch: Llama architecture, 16,060,672 parameters (Loads via
AutoModelForCausalLM) - Specs: 8 Layers
- Type: Base model (pretrained on ~600M tokens from fineweb-pro, fineweb-edu, and finepdfs)
- Multiple-choice (zero-shot, full test/validation splits)
| Task | acc | acc_stderr | acc_norm | acc_norm_stderr |
|---|---|---|---|---|
| mmlu | 0.2295 | 0.0035 | — | — |
| hellaswag | 0.2570 | 0.0044 | 0.2576 | 0.0044 |
| arc_easy | 0.3085 | 0.0095 | 0.3085 | 0.0095 |
| arc_challenge | 0.1920 | 0.0115 | 0.2133 | 0.0120 |
| piqa | 0.5533 | 0.0116 | 0.5294 | 0.0116 |
| sciq | 0.4850 | 0.0158 | 0.4380 | 0.0157 |
| winogrande | 0.5122 | 0.0140 | — | — |
| openbookqa | 0.1320 | 0.0152 | 0.2580 | 0.0196 |
ArithMark-2.0 (AxiomicLabs, n=2500, chance=0.25)
| Metric | Value |
|---|---|
| acc | 0.2532 ± 0.0087 |
| Breakdown | ops=1 (25.68%) | ops=2 (26.40%) | ops=3 (22.80%) |
Model Breakdown: MultivexAI/Supra-1.6-50M-Instruct-Ultra-exp
- Model Link: https://huggingface.co/MultivexAI/Supra-1.6-50M-Instruct-Ultra-exp
- Arch:
LlamaForCausalLM, 51,786,240 parameters (Loads viaAutoModelForCausalLM) - Specs: Hidden size: 512, Layers: 12, Attention Heads: 8, KV Heads: 4, Context Length: 5,120 tokens
- Type: Experimental instruction-tuned model (built on top of
SupraLabs/Supra-1.5-50M-Base-exp) - Multiple-choice (zero-shot, full test/validation splits)
| Task | acc | acc_stderr | acc_norm | acc_norm_stderr |
|---|---|---|---|---|
| mmlu | 0.2294 | 0.0035 | — | — |
| hellaswag | 0.2811 | 0.0045 | 0.2966 | 0.0046 |
| arc_easy | 0.4676 | 0.0102 | 0.4318 | 0.0102 |
| arc_challenge | 0.2235 | 0.0122 | 0.2560 | 0.0128 |
| piqa | 0.6061 | 0.0114 | 0.5941 | 0.0115 |
| sciq | 0.7270 | 0.0141 | 0.6600 | 0.0150 |
| winogrande | 0.5241 | 0.0140 | — | — |
| openbookqa | 0.1740 | 0.0170 | 0.2720 | 0.0199 |
ArithMark-2.0 (AxiomicLabs, n=2500, chance=0.25)
| Metric | Value |
|---|---|
| acc | 0.3052 ± 0.0092 |
| Breakdown | ops=1 (29.36%) | ops=2 (34.53%) | ops=3 (27.40%) |
System & Environment Info
OS: Ubuntu 22.04.5 LTS (x86_64)
GPU: 1x Tesla T4 (CUDA 12.8 / Driver 580.82.07)
Python: 3.12.13
PyTorch: 2.11.0+cu128
Transformers: 5.10.2
LM-Eval: 0.4.13.dev0
Seeds: random_seed=0, numpy_seed=1234, torch_seed=1234, fewshot_seed=1234
They're up!
Datdanboi25 changed discussion status to closed