Benchmark validity?
I have compared Ornith 1.0 35B A3B in APEX I-Quality quantization with GRM-3.2-Sky in Q4_K_M quantization in Benchlocal with 9x runs. The same sampling parameters were used for both:
jinja = true
reasoning-preserve = true
temp = 0.6
top-p = 0.95
top-k = 20
repeat-penalty = 1.05
b = 16384
ub = 2048
HermesAgent-20:
Ornith 1.0 35B A3B: 88
GRM-3.2-Sky: 83
BugFind-15:
Ornith 1.0 35B A3B: 92
GRM-3.2-Sky: 85
Note that this is an unfair comparison since Ornith is using a superior quaint, but I don't have the hardware to quantize 35B models currently. Will update if someone releases APEX quantizations for this model. Still, the fact that GRM-3.2-Sky, which is supposedly superior to Ornith-1.0-35B according to benchmarks, is worse than it even at Q4_K_M is suspicious and brings the validity of GRM-3.2-Sky's benchmarks into question.
im making some of my own quants- its based on apex- but i modified it slightly
ill comment the link here when theyre done cooking!
Just curious, can you quantize a 35B model (~70gb) if you only gave 32GB ram?
Just curious, can you quantize a 35B model (~70gb) if you only gave 32GB ram?
you can- I did but on 12gb
just takes a painstaking amount of python orchestration- around like 2000 lines
I might open source- but im a perfectionist so idk when ill release it
I do quantize models I like with it- so just @ me and ill consider giving it a quant!
I'll post my results in a few hours. HermesAgent-20 and BugFind-15 are the main ones I care about, but if you want I can run others. I have a R9700 AI PRO, so I will run it in Quality mode. (though I use Q8 KV if possible so I might need to reduce ctx size to like 200k)
I'll post my results in a few hours. HermesAgent-20 and BugFind-15 are the main ones I care about, but if you want I can run others. I have a R9700 AI PRO, so I will run it in Quality mode. (though I use Q8 KV if possible so I might need to reduce ctx size to like 200k)
Is there the update?
el4's quaint ended up being broken, so I'm waiting until it is fixed.
working on the fix right now
my refactor broke the mapping- a fix should be either ready today or tmrw
Sampling parameters:
reasoning-preserve = true
jinja = true
temp = 0.6
top-p = 0.95
top-k = 20
repeat-penalty = 1.05
HermesAgent-20: 79 (Ornith 1.0 35B got 88)
BugFind-15: 84 (Ornith 1.0 35B got 92)
Thanks to el4 for releasing a good quality APEX quaint for this model! These are the 2 benchmarks I value the most, but I am open to running any others included in BenchLocal.
Wait, decrease?
Yup, same results as with Q4_K_M. If you want, I'm making a github repo of a bunch of models to see how they actually stack up against each other in BenchLocal. Planning to add this one too.
https://github.com/BarerRocket2678/AI-Model-Benchmarks/tree/main
How if it in BF16 format? Hopefully you can help ya, thankyou so much for the insights :D<
Yup, same results as with Q4_K_M. If you want, I'm making a github repo of a bunch of models to see how they actually stack up against each other in BenchLocal. Planning to add this one too.
https://github.com/BarerRocket2678/AI-Model-Benchmarks/tree/main
Thats expected- the goal of an apex quant is to get the same or better with lower vram usage
Thanks for running the benchs!
Unfortunately, I can't run these types of models in BF16, since that would be 70GB of VRAM. Plus, I-Quality APEX and UD-Q4_K_M have almost zero performance decrease from BF16 and at scale it is preferable to use INT4 or FP4 for increased performance. (Deepseek V4 even uses a mix of FP8 and FP4 by default)
I used Q8_0, and initially, writing code seemed quick, getting 80% of it working.
The remaining 20% involved detailed work, bug fixes, logic adjustments, and feature additions. It led me astray; fixing one thing caused another to break, haha.
I had to go back to 27B 🚀, where it handled everything in a snap.
I used Q8_0, and initially, writing code seemed quick, getting 80% of it working.
The remaining 20% involved detailed work, bug fixes, logic adjustments, and feature additions. It led me astray; fixing one thing caused another to break, haha.
I had to go back to 27B 🚀, where it handled everything in a snap.
The characteristic of this architecture is unique. I've seen, all the qwen variants are great to make the macro architecture. But, if there is the bug, we can't use the same model. Better use another moe modeltype, such gemma or OSS. They seems better at instruction following and not too much shit talking.
Check out my new quants! they use a dynamic map instead of a static apex map- so its mathematically superior to apex