Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
danielhanchen 
posted an update 2 days ago
Hellomaniamcoollol

locally my ass

·
danielhanchen

Our guide shows how llama.cpp allows disk, RAM and VRAM offloading, so it optimally allocates it. For eg Mac unified memory systems are well suited.
marksverdhei
•
edited about 8 hours ago

No Daniel, I cannot run Kimi K2.5 locally. Do i look like i'm rich? 😭

·
danielhanchen

Sorry! We try our best to make it smaller for folks to run
YellowjacketGames

I'll give this a try on our office GPU rig. VRAM limited to 96gb but DRAM is 1024gb attached to a 64 core threadripper.

I doubt I'll be able to pull 40 TPS on this rig, but hey, local is local!

Thx to all the Unsloth guys as usual!

In this post