tiny-random/qwen2.5-omni
Any-to-Any • 5.39M • Updated • 55 • 2
Syafiq Kamarul Azman
syaffers
AI & ML interests
Inference and model servers
Recent Activity
liked a model 4 days ago
tiny-random/qwen2.5-omni liked a model 4 days ago
hmellor/tiny-random-LlamaForCausalLM reacted to Banaxi-Tech's post with 🔥 5 days ago
Today we wanted to release BananaMind 2 Pico, our smallest model yet at ~0.9M parameters. Instead, we accidentally ran a very expensive experiment on what happens when you push a tiny model way past its useful token budget.
Short version: we trained on 200B tokens (~222K:1 tokens-per-parameter). The model peaked at 20B tokens with an INT Index of 4.55, then degraded monotonically over the next 160B to 3.31 — a 27% regression. Three of four Open SLM benchmarks were worse at the end of training than they were at 10% through.
The useful compute-optimal range for Pico-tier models looks like ~22K–30K tokens per parameter. Ratios like 7K:1, 15K:1, and 22K:1 all work fine — TinyStories and most sub-3M community models sit in this range. Push much further and benchmarks start rotting.
Follow us for more:
https://huggingface.co/BananaMind
@vovaRL
@Banaxi-Tech
Full writeup with all checkpoints, the Chinchilla-ratio control run, and the schedule-vs-overtraining analysis: https://huggingface.co/blog/Banaxi-Tech/ovdadadadd
And if anyone, i dont know the reason why you would, wants the 20B token checkpoint reply and ill upload it as BananaMind 2.1 Pico EXPOrganizations
None yet
liked 2
models 4 days ago
reacted to Banaxi-Tech's post with 🔥 5 days ago
Post
2732
Today we wanted to release BananaMind 2 Pico, our smallest model yet at ~0.9M parameters. Instead, we accidentally ran a very expensive experiment on what happens when you push a tiny model way past its useful token budget.
Short version: we trained on 200B tokens (~222K:1 tokens-per-parameter). The model peaked at 20B tokens with an INT Index of 4.55, then degraded monotonically over the next 160B to 3.31 — a 27% regression. Three of four Open SLM benchmarks were worse at the end of training than they were at 10% through.
The useful compute-optimal range for Pico-tier models looks like ~22K–30K tokens per parameter. Ratios like 7K:1, 15K:1, and 22K:1 all work fine — TinyStories and most sub-3M community models sit in this range. Push much further and benchmarks start rotting.
Follow us for more:
BananaMind
@vovaRL
@Banaxi-Tech
Full writeup with all checkpoints, the Chinchilla-ratio control run, and the schedule-vs-overtraining analysis: https://huggingface.co/blog/Banaxi-Tech/ovdadadadd
And if anyone, i dont know the reason why you would, wants the 20B token checkpoint reply and ill upload it as BananaMind 2.1 Pico EXP
Short version: we trained on 200B tokens (~222K:1 tokens-per-parameter). The model peaked at 20B tokens with an INT Index of 4.55, then degraded monotonically over the next 160B to 3.31 — a 27% regression. Three of four Open SLM benchmarks were worse at the end of training than they were at 10% through.
The useful compute-optimal range for Pico-tier models looks like ~22K–30K tokens per parameter. Ratios like 7K:1, 15K:1, and 22K:1 all work fine — TinyStories and most sub-3M community models sit in this range. Push much further and benchmarks start rotting.
Follow us for more:
@vovaRL
@Banaxi-Tech
Full writeup with all checkpoints, the Chinchilla-ratio control run, and the schedule-vs-overtraining analysis: https://huggingface.co/blog/Banaxi-Tech/ovdadadadd
And if anyone, i dont know the reason why you would, wants the 20B token checkpoint reply and ill upload it as BananaMind 2.1 Pico EXP
- 3 replies
liked a
model 6 days ago
reacted to appvoid's post with 🔥 12 days ago
Post
1353
Giving free early access to the gguf for some of you today! Tell me what you think.
CEAMFA/palmer-007-preview
CEAMFA/palmer-007-preview
updated a
model 13 days ago
upvoted a collection 13 days ago
liked a
model 14 days ago
published a
model 15 days ago
updated a
model 23 days ago
published a
model 23 days ago
reacted to ManniX-ITA's post with 🚀 29 days ago
Post
2051
🚀 New release: Qwen3.6-27B-A3B-Coder
A code-specialized MoE carved out of Qwen3.6-35B-A3B by pure expert pruning — no fine-tuning, no distillation. I profiled all 256 experts on balanced corpora plus targeted code benchmarks (LiveCodeBench +
MultiPL-E), built a competence map with the code classes up-weighted 1.5×, and dropped the 72 weakest experts per layer (256→184, ~35B→27B). Router, attention, norms, the MTP head and the vision tower are all
preserved; active params stay at A3B and routing is baked to top-10 (revert to top-8 anytime).
📊 Benchmarks (Q6_K, temp 0.6):
• MultiPL-E 0.840
• HumanEval 0.970
• LiveCodeBench 0.688
• GSM8K 0.970 · ARC-C 0.944 · AIME 0.733
• GPQA-Diamond 0.773 · MATH-500 0.620 · IFEval 0.730
• Average 0.808
27B footprint, A3B speed, coding that punches well above its size — and the preserved MTP head gives you speculative decoding out of the box (text + vision).
🔗 Model: ManniX-ITA/Qwen3.6-27B-A3B-Coder
📦 GGUF (+MTP): ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF
🦙 Ollama: https://ollama.com/mannix/qwen3.6-27b-a3b-coder
A code-specialized MoE carved out of Qwen3.6-35B-A3B by pure expert pruning — no fine-tuning, no distillation. I profiled all 256 experts on balanced corpora plus targeted code benchmarks (LiveCodeBench +
MultiPL-E), built a competence map with the code classes up-weighted 1.5×, and dropped the 72 weakest experts per layer (256→184, ~35B→27B). Router, attention, norms, the MTP head and the vision tower are all
preserved; active params stay at A3B and routing is baked to top-10 (revert to top-8 anytime).
📊 Benchmarks (Q6_K, temp 0.6):
• MultiPL-E 0.840
• HumanEval 0.970
• LiveCodeBench 0.688
• GSM8K 0.970 · ARC-C 0.944 · AIME 0.733
• GPQA-Diamond 0.773 · MATH-500 0.620 · IFEval 0.730
• Average 0.808
27B footprint, A3B speed, coding that punches well above its size — and the preserved MTP head gives you speculative decoding out of the box (text + vision).
🔗 Model: ManniX-ITA/Qwen3.6-27B-A3B-Coder
📦 GGUF (+MTP): ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF
🦙 Ollama: https://ollama.com/mannix/qwen3.6-27b-a3b-coder
- 13 replies
updated a
model 2 months ago
published a
model 2 months ago
updated a
model 3 months ago
published a
model 3 months ago
liked a
model 7 months ago
updated a
model 8 months ago
published a
model 8 months ago
reacted to cgeorgiaw's post with ❤️ 12 months ago
Post
6169
🚀🚀🚀 The largest ever dataset of co-folded 3D protein-ligand structures just dropped on HF!!
Meet SAIR (Structurally Augmented IC₅₀ Repository): 5M+ AI-generated complexes with experimentally measured drug potency data from SandboxAQ. 🚀🚀🚀
Check it out and explore here: SandboxAQ/SAIR
Meet SAIR (Structurally Augmented IC₅₀ Repository): 5M+ AI-generated complexes with experimentally measured drug potency data from SandboxAQ. 🚀🚀🚀
Check it out and explore here: SandboxAQ/SAIR
- 3 replies