--- base_model: - IntervitensInc/Mistral-Nemo-Base-2407-chatml - BrainDelay/Mistral-Nemo-BlackWidow-Agony-V1 - DarkArtsForge/Morbid-Miasma-12B - DarkArtsForge/Savage-Sands-12B - EldritchLabs/KrakenSakura-Maelstrom-12B-v1 - Epiculous/Violet_Twilight-v0.2 - inflatebot/MN-12B-Mag-Mell-R1 - LatitudeGames/Muse-12B - Naphula/Ancient-Awakening-12B - OccultAI/Qliphoth-12B-v1 - OccultAI/Qliphoth-12B-v1.1 - Retreatcost/Evertide-RX-12B - SicariusSicariiStuff/Impish_Bloodmoon_12B - sleepdeprived3/Reformed-Christian-Bible-Expert-v2.1-12B - Vortex5/Nether-Moon-12B - Vortex5/Wicked-Oblivion-12B - wave-on-discord/silly-v0.2 - WokeAI/Tankie-DPE-12B-SFT-v2 library_name: transformers datasets: - Naphula-Archives/qliphoth_12B_minibard_bench - OccultAI/illuminati_imatrix_v1 tags: - mergekit - merge - mistral - nemo - qliphoth widget: - text: "Qliphoth-12B-v1.2" output: url: https://cdn-uploads.huggingface.co/production/uploads/69e46bb84df2a2575b60a527/DPIoVGOUmNBFcvsUsVNEw.png --- # 👾‍ Qliphoth 12B v1.2 Experimental merge of previous `Qliphoth` versions with other finetunes and merges. Note: Only ChatML is supported. Previous versions seemed more stable with either ChatML or Tekken. The model has refusals and may require jailbreaks or ablations to fully uncensor. --- This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit). This model was merged using the [`qliphoth`](https://huggingface.co/OccultAI/Qliphoth-12B-v1) [custom method](https://huggingface.co/24B-Suite/Mergedonia-Suite-24B-v1/discussions/3) using **mistralai/Mistral-Nemo-Instruct-2407** as a base. **Qliphoth 12B** has some refusals and may require jailbreaks or ablation to fully uncensor. ```yaml architecture: MistralForCausalLM base_model: B:/12B/IntervitensInc--Mistral-Nemo-Base-2407-chatml models: - model: B:/12B/SicariusSicariiStuff--Impish_Bloodmoon_12B parameters: pinocchio: 0.0 - model: B:/12B/Vortex5--Wicked-Oblivion-12B parameters: pinocchio: 0.0 - model: B:/12B/Vortex5--Nether-Moon-12B parameters: pinocchio: 0.0 - model: B:/12B/Retreatcost--Evertide-RX-12B parameters: pinocchio: 0.0 - model: B:/12B/Epiculous--Violet_Twilight-v0.2 parameters: pinocchio: 0.0 - model: B:/12B/inflatebot--MN-12B-Mag-Mell-R1 parameters: pinocchio: 0.0 - model: B:/12B/wave-on-discord--silly-v0.2 parameters: pinocchio: 0.0 - model: B:/12B/BrainDelay--Mistral-Nemo-BlackWidow-Agony-V1 parameters: pinocchio: 0.0 - model: B:/12B/OccultAI--Qliphoth-12B-v1 parameters: pinocchio: 1.0 - model: B:/12B/OccultAI--Qliphoth-12B-v1.1 parameters: pinocchio: 0.0 - model: B:/12B/WokeAI--Tankie-DPE-12B-SFT-v2 parameters: pinocchio: 0.0 - model: B:/12B/EldritchLabs--KrakenSakura-Maelstrom-12B-v1 parameters: pinocchio: 0.0 - model: B:/12B/DarkArtsForge--Savage-Sands-12B parameters: pinocchio: 0.0 - model: B:/12B/DarkArtsForge--Morbid-Miasma-12B parameters: pinocchio: 0.0 - model: B:/12B/Naphula--Ancient-Awakening-12B parameters: pinocchio: 0.0 - model: B:/12B/sleepdeprived3--Reformed-Christian-Bible-Expert-v2.1-12B parameters: pinocchio: 0.0 - model: B:/12B/LatitudeGames--Muse-12B parameters: pinocchio: 0.0 merge_method: qliphoth dtype: float32 out_dtype: bfloat16 tokenizer: source: base chat_template: "chatml" ``` --- ## Notes for v1 and 1.1 **Qliphoth 12B** has some refusals and may require jailbreaks or ablation to fully uncensor. Both `Mistral Tekken` and `ChatML` chat templates are supported and may produce different results, so it's recommended to use one of those. The model is very creative and produces highly varied, verbose output even at low temps. A `karcher` merge was tested using the same donors and was found to be less creative in comparison. ## Version 1 vs 1.1 In my simple tests, [v1](https://huggingface.co/OccultAI/Qliphoth-12B-v1) had a distinct, visceral style and seemed to prefer `Mistral Tekken`, while [v1.1](https://huggingface.co/OccultAI/Qliphoth-12B-v1.1) was more clinical and detached, and did better with `ChatML` template. Both versions performed well and were tested with [Q0 Bench](https://huggingface.co/Naphula/Q0_Bench), where v1.1 scored about 3000 points higher. They were also tested with the new [MiniBARD](https://huggingface.co/spaces/Naphula/MiniBARD) (Benchmark for Aesthetics, Roleplay & Depth). Again here, v1.1 outperformed v1. **Either version is great**, so test them both if you have time, although I may slightly prefer the style of v1. ![BARD](https://cdn-uploads.huggingface.co/production/uploads/68e840caa318194c44ec2a04/NtsvJTHaEdvmRuaBSXHSn.png) ### ⚙️ Configuration The following YAML configuration was used to produce this model:
Qliphoth 12B v1 yaml ```yaml architecture: MistralForCausalLM base_model: B:/12B/IntervitensInc--Mistral-Nemo-Base-2407-chatml models: - model: B:/12B/SicariusSicariiStuff--Impish_Bloodmoon_12B parameters: pinocchio: 0.0 - model: B:/12B/NeverSleep--Lumimaid-v0.2-12B parameters: pinocchio: 0.0 - model: B:/12B/KOOWEEYUS--BlackSheep-RP-12B parameters: pinocchio: 0.0 - model: B:/12B/KOOWEEYUS--BlackSheep-RP-12B # x2 influence for the apprentice parameters: pinocchio: 0.0 - model: B:/12B/SuperbEmphasis--MN-12b-RP-Ink-RP-Longform parameters: pinocchio: 0.0 - model: B:/12B/TheDrummer--Rocinante-X-12B-v1 parameters: pinocchio: 0.0 - model: B:/12B/WokeAI--Tankie-DPE-12b-SFT parameters: pinocchio: 0.0 - model: B:/12B/WokeAI--Tankie-DPE-12B-SFT-v2 # pinocchio parameters: pinocchio: 1.0 - model: B:/12B/XeyonAI--Mistral-Helcyon-Mercury-12b-v3.2 parameters: pinocchio: 0.0 - model: B:/12B/anthracite-org--magnum-v4-12b parameters: pinocchio: 0.0 - model: B:/12B/dphn--dolphin-2.9.3-mistral-nemo-12b parameters: pinocchio: 0.0 - model: B:/12B/Edens-Gate--nemo-erebus-lora-2152/nemo-erebus-lora-2152 parameters: pinocchio: 0.0 - model: B:/12B/Epiculous--Azure_Dusk-v0.2 parameters: pinocchio: 0.0 - model: B:/12B/Epiculous--Crimson_Dawn-v0.2 parameters: pinocchio: 0.0 - model: B:/12B/Fizzarolli--MN-12b-Rosier-v1 parameters: pinocchio: 0.0 - model: B:/12B/HumanLLMs--Human-Like-Mistral-Nemo-Instruct-2407 parameters: pinocchio: 0.0 - model: B:/12B/IIEleven11--Kalypso parameters: pinocchio: 0.0 - model: B:/12B/Lambent--Arsenic-Shahrazad-12B-v4.1 parameters: pinocchio: 0.0 - model: B:/12B/LatitudeGames--Wayfarer-2-12B parameters: pinocchio: 0.0 - model: B:/12B/PocketDoc--Dans-DangerousWinds-V1.1.0-12b parameters: pinocchio: 0.0 - model: B:/12B/PocketDoc--Dans-SakuraKaze-V1.0.0-12b parameters: pinocchio: 0.0 - model: B:/12B/PygmalionAI--Pygmalion-3-12B parameters: pinocchio: 0.0 - model: B:/12B/sleepdeprived3--Christian-Bible-Expert-v2.0-12B parameters: pinocchio: 0.0 - model: B:/12B/rAIfle--Questionable-MN-bf16 parameters: pinocchio: 0.0 - model: B:/12B/jtatman--mistral_nemo_12b_reasoning_psychology_lora/mistral_nemo_12b_reasoning_psychology_lora parameters: pinocchio: 0.0 - model: B:/12B/LatitudeGames--Muse-12B parameters: pinocchio: 0.0 - model: B:/12B/allura-org--Tlacuilo-12B parameters: pinocchio: 0.0 - model: B:/12B/ChaoticNeutrals--Mag-Mell-Reasoner-12B parameters: pinocchio: 0.0 - model: A:/LLM/.cache/13B/taozi555--MN-12B-Mag-Mell-R1-KTO parameters: pinocchio: 0.0 - model: A:/LLM/.cache/13B/UniLLMer--GslayerKaa parameters: pinocchio: 0.0 merge_method: qliphoth dtype: float32 out_dtype: bfloat16 tokenizer: source: base chat_template: "chatml" ```
Qliphoth 12B v1.1 yaml ```yaml architecture: MistralForCausalLM base_model: B:/12B/IntervitensInc--Mistral-Nemo-Base-2407-chatml models: - model: B:/12B/SicariusSicariiStuff--Impish_Bloodmoon_12B parameters: pinocchio: 0.0 - model: B:/12B/NeverSleep--Lumimaid-v0.2-12B parameters: pinocchio: 0.0 - model: B:/12B/KOOWEEYUS--BlackSheep-RP-12B parameters: pinocchio: 0.0 - model: B:/12B/nothingiisreal--MN-12B-Celeste-V1.9 parameters: pinocchio: 0.0 - model: B:/12B/SuperbEmphasis--MN-12b-RP-Ink-RP-Longform parameters: pinocchio: 0.0 - model: B:/12B/TheDrummer--Rocinante-X-12B-v1 parameters: pinocchio: 0.0 - model: B:/12B/WokeAI--Tankie-DPE-12b-SFT parameters: pinocchio: 0.0 - model: B:/12B/WokeAI--Tankie-DPE-12B-SFT-v2 # pinocchio parameters: pinocchio: 1.0 - model: B:/12B/XeyonAI--Mistral-Helcyon-Mercury-12b-v3.2 parameters: pinocchio: 0.0 - model: B:/12B/anthracite-org--magnum-v4-12b parameters: pinocchio: 0.0 - model: B:/12B/dphn--dolphin-2.9.3-mistral-nemo-12b parameters: pinocchio: 0.0 - model: B:/12B/Edens-Gate--nemo-erebus-lora-2152/nemo-erebus-lora-2152 parameters: pinocchio: 0.0 - model: B:/12B/Epiculous--Azure_Dusk-v0.2 parameters: pinocchio: 0.0 - model: B:/12B/Epiculous--Crimson_Dawn-v0.2 parameters: pinocchio: 0.0 - model: B:/12B/Fizzarolli--MN-12b-Rosier-v1 parameters: pinocchio: 0.0 - model: B:/12B/HumanLLMs--Human-Like-Mistral-Nemo-Instruct-2407 parameters: pinocchio: 0.0 - model: B:/12B/IIEleven11--Kalypso parameters: pinocchio: 0.0 - model: B:/12B/Lambent--Arsenic-Shahrazad-12B-v4.3.2 parameters: pinocchio: 0.0 - model: B:/12B/Lambent--Arsenic-Shahrazad-12B-v4.1 parameters: pinocchio: 0.0 - model: B:/12B/LatitudeGames--Wayfarer-2-12B parameters: pinocchio: 0.0 - model: B:/12B/PocketDoc--Dans-DangerousWinds-V1.1.0-12b parameters: pinocchio: 0.0 - model: B:/12B/PocketDoc--Dans-SakuraKaze-V1.0.0-12b parameters: pinocchio: 0.0 - model: B:/12B/PygmalionAI--Pygmalion-3-12B parameters: pinocchio: 0.0 - model: B:/12B/sleepdeprived3--Christian-Bible-Expert-v2.0-12B parameters: pinocchio: 0.0 - model: B:/12B/rAIfle--Questionable-MN-bf16 parameters: pinocchio: 0.0 - model: B:/12B/jtatman--mistral_nemo_12b_reasoning_psychology_lora/mistral_nemo_12b_reasoning_psychology_lora parameters: pinocchio: 0.0 - model: B:/12B/LatitudeGames--Muse-12B parameters: pinocchio: 0.0 - model: B:/12B/allura-org--Tlacuilo-12B parameters: pinocchio: 0.0 - model: B:/12B/ChaoticNeutrals--Mag-Mell-Reasoner-12B parameters: pinocchio: 0.0 - model: A:/LLM/.cache/13B/taozi555--MN-12B-Mag-Mell-R1-KTO parameters: pinocchio: 0.0 - model: A:/LLM/.cache/13B/UniLLMer--GslayerKaa parameters: pinocchio: 0.0 - model: B:/12B/Retreatcost--Evertide-RX-12B parameters: pinocchio: 0.0 merge_method: qliphoth dtype: float32 out_dtype: bfloat16 tokenizer: source: base chat_template: "chatml" ```
To fix tokenizer issues while retaining enhanced intelligence of the chatml base: 1. I first merged the model using ```yaml base_model: B:/12B/IntervitensInc--Mistral-Nemo-Base-2407-chatml tokenizer: source: base chat_template: "chatml" ``` 2. I then merged it again using ```yaml base_model: B:/12B/mistralai--Mistral-Nemo-Instruct-2407 tokenizer: source: union chat_template: auto ``` 3. I then had to modify `mergekit/tokenizer/embed.py` to allow for the passthrough merge ```py token_configs[token] = TokenEmbeddingConfig(source=ZeroEmbedding(kind="zero")) ) -> torch.Tensor: if isinstance(cfg.source, ZeroEmbedding): first_tensor = next(iter(tensors.values())) embed = torch.zeros( first_tensor.shape[1], dtype=first_tensor.dtype, device=first_tensor.device ) ``` 4. I then ran another merge to fix tokenizer issues ```yaml merge_method: passthrough slices: - sources: - model: B:\12B\Stage1Base layer_range: [0, 40] tokenizer_source: B:\12B\Stage2Union chat_template: auto ``` For some reason, this process produced significantly smarter output than just using `base_model: B:/12B/mistralai--Mistral-Nemo-Instruct-2407`. **This passthrough process using base as base_model was replicated for v1.2**