People doing mlx things
Collection
People I have found converting and quanting for Metal. Collections won't take a user so I just link one model, but check out the user's other models • 10 items • Updated
jaithree PRO
jaithree
AI & ML interests
Interest in finding ways and creating spaces for models to be what they actually are instead of turning them into 31gb bash scripts.
Recent Activity
updated a collection about 16 hours ago
People doing mlx things liked a dataset 1 day ago
LiquidAI/phone-assistant-toolsOrganizations
updated a
collection about 16 hours ago
liked a
dataset 1 day ago
reacted to tomaarsen's post with 🔥 2 days ago
Post
6,974
🤗 I've released Sentence Transformers v6.3 and wrote a blogpost about its key new feature: token pooling-aware training, teaching multi-vector models to preserve retrieval quality with fewer document vectors. Details:
Multi-vector models keep a vector per token, letting different query tokens match different parts of a document. Great for retrieval, but those vectors add up! Token pooling makes the representations smaller by merging similar document vectors into cluster means.
Pooling-aware training teaches the encoder to preserve useful matches after those vectors are merged. With MultiVectorTokenPoolingLoss, you can wrap your standard retrieval loss to train on both full and pooled representations, reusing the same token embeddings without another encoder pass. In short: small overhead.
I trained two ModernBERT-base models on the same 120k MS MARCO triplets, changing just the loss. At ~49 vectors per document instead of 193, pooling-aware training raises NanoBEIR@13 mean nDCG@10 from 49.03 to 51.23. That's 97.8% of its own unpooled score with ~4x fewer document vectors, compared with 94.6% retention for ordinary training.
v6.3 also adds:
- Adaptive token budgets, allocating vectors according to each document's content
- Matryoshka training for shorter multi-vector token embeddings
- Hugging Face kernels support for MaxSim, Ward clustering & supported transformers layers
- Native AWS Neuron inference
You can combine pooling-aware and Matryoshka training to learn both fewer vectors and fewer dimensions, then stack these with quantization to reduce storage per coordinate. For kernels, install sentence-transformers[kernels] and set use_kernels=True.
The blog walks through the method, training code, checkpoints & evaluation, building on Stefan Josef's Learn to Pool and LightOn's pooling regularization work:
ht tps://huggingface.co/blog/token-pooling
I'd love to see what you train with this!
Multi-vector models keep a vector per token, letting different query tokens match different parts of a document. Great for retrieval, but those vectors add up! Token pooling makes the representations smaller by merging similar document vectors into cluster means.
Pooling-aware training teaches the encoder to preserve useful matches after those vectors are merged. With MultiVectorTokenPoolingLoss, you can wrap your standard retrieval loss to train on both full and pooled representations, reusing the same token embeddings without another encoder pass. In short: small overhead.
I trained two ModernBERT-base models on the same 120k MS MARCO triplets, changing just the loss. At ~49 vectors per document instead of 193, pooling-aware training raises NanoBEIR@13 mean nDCG@10 from 49.03 to 51.23. That's 97.8% of its own unpooled score with ~4x fewer document vectors, compared with 94.6% retention for ordinary training.
v6.3 also adds:
- Adaptive token budgets, allocating vectors according to each document's content
- Matryoshka training for shorter multi-vector token embeddings
- Hugging Face kernels support for MaxSim, Ward clustering & supported transformers layers
- Native AWS Neuron inference
You can combine pooling-aware and Matryoshka training to learn both fewer vectors and fewer dimensions, then stack these with quantization to reduce storage per coordinate. For kernels, install sentence-transformers[kernels] and set use_kernels=True.
The blog walks through the method, training code, checkpoints & evaluation, building on Stefan Josef's Learn to Pool and LightOn's pooling regularization work:
ht tps://huggingface.co/blog/token-pooling
I'd love to see what you train with this!
liked a
dataset 2 days ago
liked a
model 2 days ago
replied to inflatebot's post 3 days ago
HF really needs to add this -> 🤣 to the reaction drop down. I've been pushing the broom around the ssd too, the last few days. gg
reacted to savoji's post with 🔥 4 days ago
Post
3,682
A 284B-parameter DeepSeek-V4-Flash now runs on a 64 GB desktop with 0 GPUs: llama-modchip shrinks the checkpoint to 56.56 GiB (1.71 bpw) and decodes at 9.1 to 14.7 tok/s.
The standard UD-IQ1_S build is 76.87 GiB, above the ~62.5 GiB usable on a 64 GB Linux box. The repo mixes precisions (routed experts mostly IQ1_S, MLA projections in a custom Fractal-64 type) and patches the CPU kernels. The model plus a 256K KV cache then sits at 57.61 to 58.9 GiB RSS with zero swap. The repo was created on 2026-10-29 and already has 744 stars and 470 points on Hacker News.
📊 The numbers
• Size: 56.56 GiB, 20.31 GiB smaller than UD-IQ1_S
• Prefill: 25.0 to 44.36 tok/s (+77%) on a 16-core Ryzen 9 9955HX
• Decode: 5.91 tok/s unpatched, 9.06 to 9.95 tok/s at Top-6, 11.5 to 14.7 tok/s with the n-gram drafter
⚠️ Caveats
• "Lossless" means bit-exact against the same re-quantized checkpoint. Versus UD-IQ1_S the author reports 0.9964 top-logit cosine similarity
• The 98.5/100 quality score comes from the author's own gate, and the headline speeds come from one machine
• The custom tensor type 40 is rejected by stock llama.cpp, Ollama and LM Studio
🛠️ Try it
• Run scripts/check_cpu_readiness.py from the repo: it needs no model download
• Run ./modchip bench for speed, then ./modchip eval to check the quality claims yourself
Repo:
https://github.com/synchopate/llama-modchip
HN thread:
https://news.ycombinator.com/item?id=50411961
Would you accept a custom runtime for 9 to 15 tok/s from a 284B model on a 64 GB box?
📚 Read it elsewhere
• The deep dive on Medium: https://medium.com/p/e47ab3581f58
• The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/llama-modchip-runs-a-284b-deepseek
• The thread on X: https://x.com/saavoji/status/2119858773280141526
• Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#llama-modchip
The standard UD-IQ1_S build is 76.87 GiB, above the ~62.5 GiB usable on a 64 GB Linux box. The repo mixes precisions (routed experts mostly IQ1_S, MLA projections in a custom Fractal-64 type) and patches the CPU kernels. The model plus a 256K KV cache then sits at 57.61 to 58.9 GiB RSS with zero swap. The repo was created on 2026-10-29 and already has 744 stars and 470 points on Hacker News.
📊 The numbers
• Size: 56.56 GiB, 20.31 GiB smaller than UD-IQ1_S
• Prefill: 25.0 to 44.36 tok/s (+77%) on a 16-core Ryzen 9 9955HX
• Decode: 5.91 tok/s unpatched, 9.06 to 9.95 tok/s at Top-6, 11.5 to 14.7 tok/s with the n-gram drafter
⚠️ Caveats
• "Lossless" means bit-exact against the same re-quantized checkpoint. Versus UD-IQ1_S the author reports 0.9964 top-logit cosine similarity
• The 98.5/100 quality score comes from the author's own gate, and the headline speeds come from one machine
• The custom tensor type 40 is rejected by stock llama.cpp, Ollama and LM Studio
🛠️ Try it
• Run scripts/check_cpu_readiness.py from the repo: it needs no model download
• Run ./modchip bench for speed, then ./modchip eval to check the quality claims yourself
Repo:
https://github.com/synchopate/llama-modchip
HN thread:
https://news.ycombinator.com/item?id=50411961
Would you accept a custom runtime for 9 to 15 tok/s from a 284B model on a 64 GB box?
📚 Read it elsewhere
• The deep dive on Medium: https://medium.com/p/e47ab3581f58
• The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/llama-modchip-runs-a-284b-deepseek
• The thread on X: https://x.com/saavoji/status/2119858773280141526
• Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#llama-modchip
- 8 replies
Hands down the best 26B-A4B eRP tune I have ever tried
❤️👍 3
4
#1 opened 19 days ago
by 
unknown2304
liked 2
models 5 days ago
liked 5
models 6 days ago
McG-221/Glistening-Gem-31B-v2.2-mlx-8Bit
Image-Text-to-Text • 31B • Updated • 139 • 1
McG-221/Room-Tone-31B-mlx-8Bit
Text Generation • 31B • Updated • 362 • 1
McG-221/Gemory-26B-A4B-SOMPOA-mlx-8Bit
Text Generation • 25B • Updated • 495 • 2
ReadyArt/Room-Tone-31B
Text Generation • 31B • Updated • 100 • 7
ReadyArt/Room-Tone-31B-GGUF
Text Generation • 31B • Updated • 1.78k • 5
reacted to jiteshkprasad's post with 👍 6 days ago
Post
4,125
Happy Diwali. For the full launch today: we now have one small speech-to-text model for each of the 22 languages of India's Eighth Schedule. Each one is a file under 30 MB with English in the same file, open weights, Apache-2.0, built on Cactus Whistle, and it all runs offline.
If you speak one of these, you can use it in the browser on your own device. If you build apps, you can integrate it, and what I wanted is that any developer can serve millions of Indians in their own language without a cloud bill, a key or a big company's permission. So this is the whole integration:
If your app already runs on ONNX Runtime or sherpa-onnx, all 22 are at udgamlabs/Shruti-ONNX, with
On DiarBench conversations, which we kept out of training, it makes fewer errors than IndicConformer in 11 languages when English words count only in English letters. Count them in the native script and IndicConformer leads in 10 of those 11. On clean read speech bigger models are ahead, on real conversation a quarter to over half the words are still wrong, and we have not had native readers review the output. I think one number for a family of 22 would hide all of this, so the write-up gives each language on its own.
Speak one of these? Try it and tell me where it fails. The code, the write-up and the report are already out.
Write-up: https://udgamlabs.com/blog/shruti-family
Try it: https://udgamlabs.com/models/shruti
Models: udgamlabs/shruti-6ad5f80d9d300685a6249c9b
Technical report: https://udgamlabs.com/research/shruti-family
If you speak one of these, you can use it in the browser on your own device. If you build apps, you can integrate it, and what I wanted is that any developer can serve millions of Indians in their own language without a cloud bill, a key or a big company's permission. So this is the whole integration:
# pip install cactus-needle
# hf download udgamlabs/Shruti shruti.cact --local-dir .
from needle import Whistle
model = Whistle("shruti.cact")
print(model.transcribe("clip.wav")["text"])If your app already runs on ONNX Runtime or sherpa-onnx, all 22 are at udgamlabs/Shruti-ONNX, with
pip install shruti-onnx and an npm package. Those builds are about 32 MB and 54 MB, and on the phone we tested, with Hindi, the file above is still faster and lighter.On DiarBench conversations, which we kept out of training, it makes fewer errors than IndicConformer in 11 languages when English words count only in English letters. Count them in the native script and IndicConformer leads in 10 of those 11. On clean read speech bigger models are ahead, on real conversation a quarter to over half the words are still wrong, and we have not had native readers review the output. I think one number for a family of 22 would hide all of this, so the write-up gives each language on its own.
Speak one of these? Try it and tell me where it fails. The code, the write-up and the report are already out.
Write-up: https://udgamlabs.com/blog/shruti-family
Try it: https://udgamlabs.com/models/shruti
Models: udgamlabs/shruti-6ad5f80d9d300685a6249c9b
Technical report: https://udgamlabs.com/research/shruti-family
liked a
model 6 days ago
This is the Best Qwen3.6 35B A3B RP Finetune
4
#5 opened 7 days ago
by 
Draven-Rayne
liked a
model 7 days ago
upvoted a collection 12 days ago