Research: Pretrained Models
Collection
Pretrained Models with Average PPL Rating • 2 items • Updated • 1
Byrne-86M-Base
The base model of the Byrne family (distilled step-4000 checkpoint) — a strong general base for continued pretraining / fine-tuning. A ~86M-parameter, from-scratch SpikeWhaleLM decoder (Multi-head Latent Attention,
n-gram engram memory, hash-lookup layers, hyper-connections, HRM refinement, MTP) with a
custom ChatML-aware tokenizer. Trained with Modal credits during the Small Models,
Big Adventures Hackathon.
Related: main model → Byrne-86M
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Quazim0t0/Byrne-86M-Base", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Quazim0t0/Byrne-86M-Base", trust_remote_code=True)
Evaluation
Zero-shot, lm-eval-harness-style scoring over full splits (acc = raw continuation
log-likelihood, acc_norm = byte-length-normalized).
| Task | acc | acc_norm |
|---|---|---|
| arc_easy | 0.4205 | 0.3931 |
| arc_challenge | 0.1877 | 0.2389 |
| hellaswag | 0.2792 | 0.2927 |
| winogrande | 0.5193 | — |
| piqa | 0.5941 | 0.5860 |
| openbookqa | 0.1420 | 0.2820 |
| boolq | 0.6171 | — |
ArithMark-2.0 (AxiomicLabs)
— official metric is raw acc: 0.2732.
Language modeling: WikiText-2 byte_ppl (↓) 2.3753 · BLiMP (↑) 0.7356.
- Downloads last month
- -