Spaces:
Running
Suggestions for releases
Hey guys, put all your interesting suggestions here
May I ask what we will be working on—the model itself, or the runtime for the model?
hey congrats ur the first commenter, u can suggest anything really, the runtime for the model I think youre talking about Anvil, the development tbh has been pretty slow on it the past few weeks but here it is : https://github.com/Solstice-Labs/anvil
if you wanna work with models feel free to suggest and open pull requests or create repos within this org and your suggestions are welcome
Thanks! I'd love to see Anvil improve support for large MoE models and explore more advanced inference optimizations.
1. My current hardware and benchmark results
I'm currently testing two inference setups with different hardware configurations.
| Component | ds4 | Strata |
|---|---|---|
| Hardware | Apple Silicon Mac M5 Pro | NVIDIA RTX 3060 |
| Unified/System RAM | 64 GB | 48 GB DDR4 |
| GPU VRAM | Unified memory | 12 GB |
| Inference runtime | ds4 | Strata |
| Current model | Qwen3.8-Flash-Next-IQ2_0 | Qwen3.8-Flash-Next-IQ2_0 |
| Observed generation speed | ~60 tokens/s | ~60 tokens/s |
| Model under evaluation | Ornith-1.5-122B | Ornith-1.5-122B |
| Ornith benchmark | Testing in progress | Testing in progress |
Current baseline: Both setups achieve approximately 60 tokens/s with Qwen3.8-Flash-Next-IQ2_0. I'm now evaluating Ornith-1.5-122B to see how well these runtimes handle a significantly larger MoE model.
Note: These are my observed results, not controlled head-to-head benchmarks. Context length, prompt size, quantization, and generation settings can affect performance.
2. What I'd like to see in Anvil
| Priority | Feature / Improvement | Why it matters |
|---|---|---|
| High | Large MoE model support | Run models with hundreds of billions of parameters on consumer hardware. |
| High | Expert caching and efficient offloading | Reduce the performance penalty when model experts cannot all fit in memory. |
| High | Apple Silicon / Metal optimization | Make better use of unified memory and memory bandwidth on Apple Silicon. |
| High | NVIDIA GPU optimization | Improve inference when VRAM is limited and system RAM must be used. |
| Medium | Speculative decoding | Increase generation throughput when compatible draft models or native MTP support are available. |
| Medium | Long-context KV cache optimization | Reduce memory consumption while maintaining practical generation speed. |
| Medium | Transparent benchmarking | Report prompt-processing speed, generation tokens/s, RAM/VRAM usage, context length, and quantization. |
3. Benchmark results I'd like to compare
As I continue testing Ornith-1.5-122B, these are the metrics I'd like to collect across both systems.
| Benchmark | ds4 — M5 Pro 64 GB | Strata — RTX 3060 12 GB + 48 GB RAM |
|---|---|---|
| Qwen3.8-Flash-Next-IQ2_0 generation | ~60 tokens/s | ~60 tokens/s |
| Ornith-1.5-122B generation | TBD | TBD |
| Prompt processing speed | TBD | TBD |
| Peak memory usage | TBD | TBD |
| Performance at long context | TBD | TBD |
| Expert offloading overhead | TBD | TBD |
My main goal is to run large, quantized MoE models at practical speeds on consumer hardware—not just get them to load successfully.
I'd be happy to share additional benchmark results as testing progresses. I'd also be interested in testing Anvil against my existing runtimes, particularly for large MoE models, expert offloading, and long-context inference.
Keep up the good work!
Oh wow, nice reply dude (thanks a lot for noticing) ill surely consider that and get to work, and YOU keep up the good work, and enjoy
This is the beginning of something big dude, and thank you for choosing to notice and being a part of it early
thanks, keep up the good work
btw let me guess - mac mini right, i was planning to buy THE same model but now im buying an m5 ultra 96 gig so ill check on that and we can have prototyping with releases w/ u
btw how did you manage such large models on such cards?
Thanks! Here is my current setup and some real-world results that might be useful for Anvil.
- My hardware and inference runtimes
I’m currently testing local LLM inference on two different systems:
Component System 1 System 2
Hardware Apple Silicon Mac M5 Pro Intel Core i7-14700K
GPU Apple GPU / Metal NVIDIA RTX 3060 12 GB
Memory 64 GB unified memory 48 GB DDR4 system RAM
Dedicated GPU VRAM Unified memory 12 GB
Runtime ds4 (ds4-metal) Strata
- Models, quantization, and observed performance
Model Quantization Runtime / Hardware Generation speed
Qwen3.8-Flash-Next IQ2_0 ds4 on M5 Pro 64 GB ~60 tokens/s
Qwen3.8-Flash-Next IQ2_0 Strata on RTX 3060 12 GB + 48 GB RAM ~60 tokens/s
Ornith-1.5-122B IQ2_XS Strata ~9 tokens/s
Ornith-1.5-122B Still evaluating ds4 on M5 Pro 64 GB Testing in progress
The Qwen results are my current baseline. The Ornith result on Strata illustrates the performance challenge of running a much larger model on consumer hardware. I’m still testing Ornith on the M5 Pro, so I don’t have a confirmed speed to report for that configuration yet.
- One hardware recommendation
After experimenting with these models, my advice to anyone buying a Mac specifically for local LLM inference would be: don’t spend money on a 96 GB Mac just to find yourself running out of memory again. If your budget allows it, go straight for an M5 Max with 128 GB of unified memory.
More memory gives you much more room to experiment with larger models, higher quantizations, and longer contexts. Of course, memory capacity alone doesn’t guarantee higher tokens/s; memory bandwidth, model architecture, quantization, and inference runtime still matter.
I’d be interested in testing Anvil against ds4 and Strata with the models listed above.
Keep up the great work!
hey dude discord server is out
also thx so much for results
and i would be HONOURED for you to benchmark anvil or contribute in it
Ok bro, i will test anvil at tomorrow, my macbook is still testing Ornith. I will join discord and give you result later.
ok bye just join it ok dont forget!
cuz i dont see ya yet take your time just join
hey i think we should make a minimax 3.1 flash uncensored version