Open-weight Qwen3.8-Flash-Next hits a RoboDojo Score of 31.39 with no depth or task-verifier feedback, above GPT-6 Astra's published 28.97 and every listed VLA/WAM baseline.
The lever is the harness. The same Qwen scores 1.90 with RoboProbe, 7.75 with RPent, 21.01 with GAP (Hybrid) and 31.39 with uni-robot-agent. Under that harness GPT-6 Astra reaches 51.90, so a gap to closed models remains.
📊 The numbers • Qwen costs $1.09 per episode; Astra costs $21.29 • Best VLA/WAM baseline, Xiaomi-Robotics-1: 20.07 Score • Real robot, 19 tabletop tasks: Qwen 49.50, fine-tuned pi0.5 47.75, zero-shot Dream-1 35.50 • No depth keeps 87.5% of Qwen's score; RGB only without camera parameters drops it to 9.13
🛠️ Try it • pip install uni-robot-agent, then uni-robot-agent setup --integration robodojo (MIT, Python 3.11, Linux GPU host) • Reusable ideas: immutable observation snapshots, duplicate-action rejection, pipelined execution (84.5% to 93.8% of motion overlaps model calls)
⚠️ Caveats • Scores are low in absolute terms; Qwen needs 95.5 min to reach a score of 25, Astra 42.6 • The package has no physical-robot support yet; the real-world benchmark is in-house on one dual-arm platform
Motherlode on MTEB and Hugging Face Inference RiverRider/motherlode-code-small-en-v0.1 is a 33.4 M-parameter English embedding model for code and text. It is a drop-in for bge-small-en-v1.5: the mean of two fine-tunes of that base, one trained to match a GitHub issue to the file that needs changing and one trained to reproduce the geometry of gte-modernbert-base (a 4.5× larger model).
Where it differs on ranking metrics On the MTEB(Code, v1) MultiSWEbenchRR task it leads the models under 100 M parameters at recall@10 (52.48 versus 52.18 for granite-embedding-small-english-r2). On the clean subset of SWE-bench Verified it improves recall@1 over the base model (178 of 411 versus 145), a gap that holds across the engine settings used in Sunstone. On the CoIR code-search probe it raises nDCG@10 from 0.467 to 0.576.
The gains are concentrated on localization and code retrieval rather than uniform improvement across every MTEB task. The model remains competitive on the broader English MTEB(eng, v2) average (63.47, 4th of 15 under 100 M) while carrying a clear advantage on the code-file tasks it was built for. The weights are compatible with Hugging Face Inference Providers for feature extraction and sentence similarity. The repository also ships matching ONNX, GGUF, and sentence-transformers exports.
Self-distillation methods usually rely on an Exponential Moving Average (EMA) teachers to get synced and stable critics. However, this doesn't scale well for large models since holding both teacher and student (trained) on GPU increases latency, zero3 communication, etc.
We can mitigate this with LoRA finetuning: the backbone is shared & frozen, and we need to maintain two light adapters: one for the student (trained) and one for the teacher (EMA of the student adapter).
Getting this to work cleanly under TRL and ZeRO-3 took some fixes, so I packaged it as a plug-and-play repo. It supports:
-> robust SDFT implementation that runs under ZeRO-3
-> ZeRO-3 + LoRA integration with adapter switching for teacher/student passes
-> Saving and resuming EMA teacher checkpoint
-> working deepspeed configs that fix memory spikes when GPU memory is tight
Introducing AfricanSpeech FR-CMR — Cameroon French Speech Dataset
How well do ASR systems recognize spontaneous French spoken in Cameroon?
We've released our dataset page featuring:
- 1,172 speech clips | 162 speakers | 4.89 hours - Human-validated ASR references and demographic metadata - Audio demos and a benchmark comparing five ASR systems
Available for commercial licensing and evaluation.
Carbon-X brings DNA and language together: it reads sequences and answers questions about them 🧬
To accompany the release article, I put together Carbon-X Playground: five experiments exploring sequence completion, single-letter changes, regulatory roles, DNA origins and motif finding.
Try curated samples or your own sequences, ask questions and compare answers with references when available.
Italian Open Time Series: 46 public Italian series for zero-shot forecasting, and a first live test of Timestral
Italian data is almost absent from the corpora that time-series foundation models are trained and benchmarked on. So I collected 46 open series (CC0/CC BY): smog in Milan, bikes in Bologna, tourism, energy, earthquakes, deaths, Italian Wikipedia. Gaps stay NaN, and a tasks.yaml makes every window reproducible with fev.
Seven models, zero-shot, 19 tasks (Timestral 5M to 209M, Chronos-2, TimesFM 3.0, Toto 2.0). Skill vs seasonal naive, fev-bench style: • Backtest, windows ending 14 Oct, Timestral's release day: Toto 2.0 28.4, Timestral-209M 28.2 • Live, data published after that day (6 tasks, ~1 week): Timestral-209M 32.2, Timestral-123M 30.9, TimesFM 3.0 30.3, Toto 2.0 27.7 • Monthly consumer prices of electricity and gas: plain AutoETS beats every foundation model The live round is small; I'll re-run it every week.
Two surprises. Weather beats the calendar. On a Bologna bike counter, giving Timestral the observed weather of the forecast days (a perfect forecast) cut the error by 17%. Holidays and bridge days alone: 4%. Daylight saving time. The model works in UTC; cyclists don't. Across the 25 Oct clock change my first version put Monday's rush hour at 5 am, until I moved the series to Italian wall-clock time.
For Bologna there's a small page: pick a street and a day in the next week for an hourly bike forecast, computed live from the city's counts and Open-Meteo, plus a map of counters and air-quality stations.
Not included: GME wholesale power prices and Terna load data. Their terms forbid redistribution, and GME bans automated extraction for any purpose, AI training included.
Jebadiah 27B v2.1 with the T1e inference recipe scores Full 64.4 on Jev Decision Index 0.4.1: rank 25 overall and tie rank 6, above the Jev 1.14 reference at 60.24. Board data: October 22, 2026.
The smaller 9B v2.1 with T1e scores Full 50.9, rank 80 overall and tie rank 18, with a board median latency of 125.1 ms versus 303.2 ms for the 27B. These are board reproduction timings on one RTX PRO 6000, not a promise for another setup.
The 27B is a LoRA fine-tune of Qwen3.8-27B. These are benchmark scores on the board's scale, not accuracy percentages or a guarantee for every runtime or quantized build.
The public index is 60.46. Private new-domain tasks map to 72.52 on the public scale. The full result combines private same-skill tests and private new-domain tasks at 60/40. The 0.9-point tie band defines groups; the overall position is 25.
The board describes these as the same published v2.1 weights with a new inference recipe. Its lab check reproduced the submitted decisions with 100.00% agreement for the 9B and 99.90% for the 27B, each on 831 rows. That is reproduction agreement, not accuracy.
We use North Mini Code 1.0 locally (Ollama, q4_K_M) with OpenCode as our primary coding model. It's been the strongest small agentic coder we've tested. The one gap in daily work is that we can't paste screenshots of errors, terminal output or UI bugs into a session, since the model is text-only. Switching to a separate vision model mid-task breaks the workflow.
Is native image input planned for a future North Mini Code release, for example by building in North Micro Vision's encoder? We'd test an early build.
Introducing Live Subtitles, a Windows tray app that captions anything your PC is playing and isn't butt ugly. live and fully local. It runs NVIDIA's Parakeet-unified 0.6B on your GPU, puts words on screen about 0.36 s after they're spoken, and uses less than one CPU core while it does it.
I built a benchmark for it out of real TV, podcasts, earnings calls and a short film. Parakeet-unified scores 12.1% WER there, close to Nemotron streaming's 11.4%, but with much better punctuation and lower worst-case delay. Once a caption is on screen it never gets rewritten or jumps lines, and the overlay disappears when nobody's talking.
No NVIDIA GPU? It falls back to Nemotron on the CPU, which also covers 30 other languages. It can also write SRT/ASS/VTT files for videos on disk.
Windows only. The GPU engine needs an NVIDIA card with ~3.2 GB of free VRAM. MIT.
Edit: the first version of this post said 13.8% vs 17.0%. The replies caught a scoring bug (numbers weren't normalized), so the benchmark now uses Whisper's English normalizer and the numbers above are re-scored.
MixtureKit 1.1 is out 🚀 Turn the fine-tuned models you already have on the Hub into one Mixture-of-Experts model.
pip install mixturekit
Give it a base model and its fine-tunes (math, code, a language...). MixtureKit merges them, trains the routers that pick an expert for every token, and shows you what they learned:
🧩 3 composition methods behind one YAML: Branch-Train-MiX, Mixtral-style MoE, Branch-Train-Stitch 💻 Merges stream the experts tensor by tensor: they run on a laptop 🎯 Router training with a load balancing loss, optional LoRA 🔍 A routing visualizer: in the image, two Nile 4B experts split a mixed Egyptian sentence, Arabic script to one expert, Arabizi to the other 📦 Export to Mixtral / Qwen3-MoE, then GGUF for llama.cpp 🤗 mixturekit push uploads the result with a generated model card ✅ Llama 3, Qwen 2.5/3/3.5, Gemma 3/4 experts; tested on transformers 4.52 to 5.19
In the paper, a BTX model built with MixtureKit outperforms the dense baselines on multiple benchmarks.
We open-sourced Game-Creation-Agents — an agentic system that turns a one-line game idea into a playable browser game. 🎮
Why it matters: - It lowers the barrier: you don’t need a full studio or deep engine skills to start. - It compresses prototyping from weeks to hours. - It lets creators test more ideas, faster, and focus on what’s actually fun. - It brings game creation closer to a conversation — and makes it accessible to more people.
This isn’t just about generating code. It’s about giving more people the power to bring their ideas to life.
I retracted my own launch claim in public commits, then shipped what survived.
MK-SH-1 is a 140M decision model I built on an open encoder. It returns calibrated probabilities, a choice, and an explicit abstention below a frozen gate (tau = 0.79, written down before the curve existed).
ECE 0.016 · 90% coverage in-sample; held-out numbers in the card · 106 ms on CPU.
Now that the Beatrix recall problem will soon be resolved, I can move forward to teaching her useful information related to pragmatism.
Beatrix V3 has a lot of trouble with communication, but the model herself is entirely filled with information. With minimal processing and curation, this information will be extractable. It's not the most robust information, but we can say for certain that the information is most definitely represented usefully in a way that can be resurfaced, purposed, utilized, enhanced, expanded, and reduced as necessary.
With the AMOE directly hosting and supporting the softmax attention behavioral arms, it brought me a new form of understanding and respect for token recall at depth. I knew Beatrix would be able to represent the data, but what I got was very unexpected.
This model when those numbers don't line up, she simply speaks gibberish. Absolute gibberish as shown in the 2s softmax control variant. That model turned into a spinning top that flew off the rails, so if you query that model you will only see gibberish.
With those numbers lined up, she speaks... I don't know how to describe it. It's not gibberish, but it's definitely not, expected. Like an understanding of confusion.
There are multiple behaviors that her arms are meant to train which are currently devoid, and in some aspects the arms work perfectly like the caption arm.
If you use the 3s model with ONLY the caption arm, she will complete a simple caption for you. It may not be very good, and she's not the best model for it, but she most definitely can do it.
Don't need all 9 arms, but it doesn't hurt to have them on. Just takes too long.
SOFTMAX attention however, inferenced at bf16 this should be fine. Lightning quick you could say. TRAINING I have to train the softmax attention at fp32 forward with some rounding black magic for backward. Otherwise the v3 model simply implodes.
There have been a huge series of problems related to her in the coming writeup.
Decision Index 0.3 public suite, my own runs, submitted to the board: - 27B: 57.03, up from 55.11 - 9B: 47.20, up from 44.09
Knowledge, Language and Retrieval went up on both. Tools went down: 2.10 on the 27B (When2Call -6.38, home appliance -3.41) and 0.84 on the 9B. If you route tool calls with it, read the card before you switch.
GGUF (Q8_0, Q5_K_M, Q4_K_M) and MLX (8-bit, 4-bit) are up for both, with agreement counts against the full weights on each card.
We’re releasing Lunara Art Eval, an art evaluation dataset for studying aesthetic quality, emotional resonance and content integrity in image generation.
It contains 8,000 generated images across 1,000 shared prompts, covering diverse artistic styles. The eight models are Qwen-Image, AuraFlow, GPT-Image-1 Mini, Stable Diffusion 3.5 Large Turbo, HiDream-I1 Fast, FLUX.2-Klein-4B, Z-Image-Turbo and Moonworks Lunara.
Each image includes scores on the three dimensions, allowing comparisons across models using the same prompts and evaluator.
The paper also introduces Lunara’s Diffusion Mixture Transformer architecture, with fewer than 10B active parameters, and CAT, an active-learning-inspired training algorithm that iteratively updates the training distribution.
We’re a new lab focused on model architectures and training algorithms, working closely with artists and creators. Our earlier public datasets are:
🐻 Kodiak-v0.3-1B: seven new kinds of decision, measured on real data.
Kodiak is an open 1B encoder that answers typed questions with calibrated confidence, or says "can't tell". New in v0.3: • pairwise judge (which answer is better?) • long-answer hallucination checks • stance and sarcasm • policy violation and refund-eligibility checks against your written rules • agent step safety (run it, ask first, or never)
We stopped trusting our own synthetic tests and judged v0.3 on real labelled data it never trained on (RAGBench, MT-Bench human judgments, SemEval stance): 0.21 → 0.36, averaged over 3 training runs. Nothing else got worse, and when it says "can't tell" it's right 93% of the time (up from 88%).
Privacy is moving ad decisions onto the device. The auction runs locally, next to the context it scores.
The budget still lives on a server.
So every device bids against the balance it saw at its last sync. Between syncs, campaigns spend money they no longer have.
I simulated how much that costs: 50 devices, 36 campaigns, about 300,000 auctions per run, 30 seeds per cell, budgets frozen before evaluation. Even pacing, second-score payment:
- Sync every tick: spend lands 18% over budget (32% with iPinYou-calibrated values). - Sync every 10 ticks: 5.8 times the budget. - Sync every 50 ticks: 17.7 times the budget (10.1 times with iPinYou-calibrated values).
Zero-lag sync stays within about 1%. The lag alone does the damage.
The incentive side runs the other way. With second-score payment and zero lag, 98% of auctions are manipulable (92% with iPinYou values). Switch the payment rule to critical-bid and that drops to 0 at every sync interval, in both setups.
Sync fixes the budget. The payment rule, not the sync interval, is what removes manipulability.
All 12 result sets are on the Hub, plus the 50,808 context labels every run samples from.
Eval · EXP-046 A LoRA Specialist Beat Zero-Shot on Every Group. Merging 3 of Them Gave Most of the Gain Back. Three Qwen2.5-7B LoRA specialists, one per risk group (vulnerability, deletion, sensitive_publication), trained to predict how likely a causal chain actually completes to its harmful outcome. Each one genuinely beat its own zero-shot baseline: * vulnerability: MAE 0.098 → 0.085 * deletion: MAE 0.144 → 0.113 * sensitive_publication: MAE 0.134 → 0.100 This wasn't a task already saturated zero-shot (unlike a same-day decomposition-classifier tune, EXP-045, where the base model was already at 100% before any training). Real signal, real improvement, on a task with actual headroom. Then the equal-weight merge of all three specialists into one adapter — same convention that held up cleanly on a binary refusal task back in EXP-031 (6 specialists merged, -1pp swing, noise) — landed within 0.001–0.004 MAE of the unspecialized base model on every group. Not "close to the best specialist." Close to zero fine-tuning at all. Likely mechanism: merging LoRAs that each shift a continuous number in group-specific directions cancels out under linear combination, in a way merging LoRAs that enforce a shared binary behavior doesn't. Not investigated yet: whether a routed combination (pick the right specialist per group at inference, not blend weights) holds the gain a flat merge loses. One bug caught before writing this up, not after: the eval script's output filename only encoded before/after, not which adapter — the merged-eval run silently overwrote each specialist's own result file. Caught by checking the downloaded file's own recorded adapter path against what was expected, not by trusting the script's own success message. Fixed, specialists re-run cleanly under distinct filenames — numbers matched within sampling noise. Adapters, raw eval data (before / each specialist / merged, 9 files), and the full writeup are up.
I audited one of my own evaluations. The ranking did not hold up the way I expected.
Eight open models, one task: infer the structure of a prompt. Then ask again with the identical call. Caching off.
- Agreement between repeated identical calls (mean Jaccard) ranged from 0.39 to 0.96 across models. - Only 35 of 127 prompt-model cells were perfectly reproducible on every run. - I bootstrapped the reproducibility ranking over prompts. The two least reproducible models kept their rank in 99% and 86% of resamples. The middle four kept theirs in 27% to 48%.
So the table reliably finds the worst model. It does not reliably find the best.
Reproducible is also not the same as correct. F1 against gold annotations ran from 0.56 to 0.99.
By the audit date, 4 of the 8 model variants had been retired (HTTP 410). The study as specified can no longer be re-run. The saved outputs are what survives, so I published all of them: every inferred structure, every run, the prompts and the annotations.
Surogate Speech is out 🐇 small open speech models for agents, the speech side of Surogate.
First language shipped is Romanian, and the numbers came out better than I expected: 5.69% WER on FLEURS with a 116M model. Canary 1B gets 5.95% on the same clips, Whisper large-v3 8.42%. Leaderboard runner, one RTX 5090.
What's in it:
🎧 jackrabbit-110m-ro: speech recognition, 2,500× real time on one GPU ⚡ jackrabbit-110m-ro-streaming: live, final text about 0.7 s after you stop talking 🗣️ amami-357m-ro: TTS with three voices, runs on a CPU
Serving is one line with our engine: surogate serve --stt surogate/jackrabbit-110m-ro
Transcripts for every clip are public if you want to rescore it.
