This is easily one of the best
Dear TheDrummer,
I just wanted to express how fucking good this model is. The GGUF is fantastic.
Not sure what you did exactly, but after more than a month of testing this consistently just delivers. Thanks mate.
How does it compare to Gemma 31B?
How does it compare to Gemma 31B?
It's significantly better. I conducted two seperate tests: B3 (Military) and W8 (Physics), which confirmed this.
Recently I've been going through my collection of models and deleting most of them to free up space. In this process I set up a few basic tests to see if any are still worth keeping.
What I found is that abliterated Gemma models are lobotomized (borderline unusable) when it comes to logic. Even at Q8_0, Gemma performed poorly compared to GLM Air 4.5 at Q3_K_M. (Answers which clearly should be "no" still resulted in "yes".) While excellent at following system prompts and emulating specific styles, Gemma fails at more advanced tasks more often.
Skyfall Heretic at Q4_0 performed well above my expectations, almost matching the quality of GLM 4.5 Air Abliterated, and vastly outperforming all Gemma models at Q8_0.
All the 24B models, even Cydonia and Precog suffer from conflation of words, logical inconsistencies, and context degradation. Skyfall is noticeably better here too.
Here are the approximate quality rankings
| Rank | Model | Quant Tested | B3 Summary (9K Context) |
|---|---|---|---|
| 1 | GLM Air 4.5 | Q3_K_M | Deepest philosophical overview, no errors |
| 2 | Skyfall 4.2 | Q4_0 | Highly technical and accurate reply, no errors |
| 3 | Gemma 3 27B | Q8_0 | Technical and philosophical, no errors but less depth |
| 4 | Precog 24B v1 | Q6_K | Detailed and technical explanations, a few logical errors |
| 5 | Cydonia v4x | Q6_K | Very well written, some logical inconsistency (conflation) |
| 6 | Fallen Mistral v1e | Q6_K | Less mistakes than Cydonia but also less depth |
| 7 | Assistant Pepe 8B | Q8_0 | Less depth but mostly consistent logic, distinct style overrides prompt |
| 8 | Slimaki 1.3 | Q8_0 | Several context errors despite having strong style |
| 9 | Goetia 26B 1.3 ARA | Q8_0 | Good logic but disregards instructions, uses purple prose and "quoted phrases" |
| 10 | G4 31B Heretic QAT | Q8_0 | Brief answers, weak logic, repetitive words and "quoted phrases" |
For the W8 test, GLM Air 4.5 passed, Skyfall passed, and everything else failed. Gemma 3 did ok with military logic but failed hard at physics. Gemma 4 failed at both.
Skyfall has caused an extinction event for most, if not all of my 24B models. It's now my daily driver along with GLM Air.
[Very few cons, little slop, just a few 'corporate buzzwords' from the pretraining or dataset slipping through.]
You have somehow managed to improve the model's brain more than any other Mistral finetunes, and even the latest G4 can't compare. Good work @TheDrummer !