@Hoglet-33 did you have attention layers dominating the gradient? im guessing an all ratio sweep at like 100% mamba 7:1 3:1 and 1:1 are already planned for a wide curve. seeing where the attention layers sit too
i64 Systems PRO
i64systems
·
AI & ML interests
deterministic, auditable local AI · integer/low-bit ML · memory systems
Recent Activity
View all activity
Organizations
None yet
reacted to CodeSoft's post with 🔥 about 1 hour ago
Post
377
Sorbet Mini Experimental is out!
It's not the best model in the 5M parameter range, but it's going to be a really useful model to train on top of. This release was mostly to prove that the model actually works. I trained it on 150M tokens from TinyStories in about 12 minutes.
Does anyone have any tips on training models in this size range? I want to make the full Sorbet Mini release as good as it can be.
It's not the best model in the 5M parameter range, but it's going to be a really useful model to train on top of. This release was mostly to prove that the model actually works. I trained it on 150M tokens from TinyStories in about 12 minutes.
Does anyone have any tips on training models in this size range? I want to make the full Sorbet Mini release as good as it can be.
- 11 replies
posted an update about 2 hours ago
Post
9
remat is no longer a one-model claim!!proved it on Qwen3-30B-A3B, K=32 of 128 experts resident, output task byte-identical to the full reference, zero bytes different *in bf16*🥰🥰 GPU comes next😈
reacted to KlondikeDev's post with 🔥 about 15 hours ago
Post
2208
Boris-1.7-D60M-n30M out NOW!
opencerebral/Boris-1.7-D60M-n30M
Following this will be Boris-1.8-D60M-n30M, which will test both the n-gram embeddings AND a new architecture.
Then, Boris-2 will begin training!
opencerebral/Boris-1.7-D60M-n30M
Following this will be Boris-1.8-D60M-n30M, which will test both the n-gram embeddings AND a new architecture.
Then, Boris-2 will begin training!
liked a
model about 15 hours ago
upvoted a collection about 16 hours ago
updated a
collection about 16 hours ago
updated a
model about 16 hours ago
Nice work getting a 3:1 Mamba2/attention hybrid trained at this size:) On the saturation question above: it's cheap to settle at 10M- a loss curve, and the same curve for a transformer-only run at matched tokens, would show whether the extra tokens kept buying anything. I'd wanna read both. I'm curious how you landed on 3:1 rather than 2:1 or 4:1- factor of three seems to be a pattern at this scale.
liked a
model about 17 hours ago
reacted to Hoglet-33's post with 🔥 about 17 hours ago
Post
3224
Pebble 10M and Pebble 10M Chat are now released!
Both models use our Mamba/Transformer 3:1 hybrid architecture and were pretrained on 25 billion tokens.
Pebble 10M Chat was additionally fine-tuned on 250 million tokens of Smol-SmolTalk to improve its conversational capabilities.
You can find them here:
- basically-ai/Pebble-10M
- basically-ai/Pebble-10M-Chat
We hope you enjoy using them. The rest of the Pebble family will be released soon.
Follow for more:
@Hoglet-33
basically-ai
Both models use our Mamba/Transformer 3:1 hybrid architecture and were pretrained on 25 billion tokens.
Pebble 10M Chat was additionally fine-tuned on 250 million tokens of Smol-SmolTalk to improve its conversational capabilities.
You can find them here:
- basically-ai/Pebble-10M
- basically-ai/Pebble-10M-Chat
We hope you enjoy using them. The rest of the Pebble family will be released soon.
Follow for more:
@Hoglet-33
- 26 replies
upvoted an article about 19 hours ago
Article
NVMe + ??? = Giant MoE's in Residency?
i64systems
• • 1 published an article about 19 hours ago
Article
NVMe + ??? = Giant MoE's in Residency?
i64systems
• • 1 liked a
model about 20 hours ago
updated a
model about 20 hours ago
published a
model about 20 hours ago
reacted to Bc-AI's post with 🔥 about 20 hours ago
Post
2461
Building a 6.58B sparse MoE model from scratch on a single GPU.
Hey everyone! Today is day 1 of training Smilyai-Lab's new model I call G1-MINI. It's basically the smaller version of our planned model G1 which will be 20B and activate about 2B per token. MINI activates about 1.16B params per token and is currently training right now. If no errors spring up now, I'd say i can launch sometime around September 15th~ish. Thanks to our beta testers:
@guardamarcos
@ProCreations
@juiceb0xc0de
@Timmy6767
@Sbui503
@atom77777
@Fishtiks
@smartdigitalnetworks
@EmetTheGolum
@smilyai-large-team
@MUK-IS-GOAT
@Bc-AI
@Banaxi-Tech
@vovaRL
@Datdanboi25
Hey everyone! Today is day 1 of training Smilyai-Lab's new model I call G1-MINI. It's basically the smaller version of our planned model G1 which will be 20B and activate about 2B per token. MINI activates about 1.16B params per token and is currently training right now. If no errors spring up now, I'd say i can launch sometime around September 15th~ish. Thanks to our beta testers:
@guardamarcos
@ProCreations
@juiceb0xc0de
@Timmy6767
@Sbui503
@atom77777
@Fishtiks
@smartdigitalnetworks
@EmetTheGolum
@smilyai-large-team
@MUK-IS-GOAT
@Bc-AI
@Banaxi-Tech
@vovaRL
@Datdanboi25
- 9 replies
keeping an eye on this!
liked a
model 16 days ago