| --- |
| license: apache-2.0 |
| base_model: |
| - dreamgen/lucid-v1-nemo |
| - limloop/MN-12B-Faun-RP-RU |
| library_name: transformers |
| language: |
| - en |
| - ru |
| tags: |
| - mergekit |
| - merge |
| - slerp |
| - russian |
| - uncensored |
| - roleplay |
| - mistral-nemo |
| --- |
| |
| # MN-12B-LucidFaun-RP-RU |
|
|
| <details> |
| <summary>🇷🇺 Нажмите, чтобы развернуть описание на русском</summary> |
|
|
| ## 🌟 О модели |
|
|
| **MN-12B-LucidFaun-RP-RU** — гибридная модель на базе Mistral Nemo 12B, созданная методом диагностического SLERP-слияния. Объединяет сильные стороны двух моделей: |
|
|
| * 🎭 **Живой RP-характер Faun** — современный стиль, богатая лексика, поддержка ninja-формата инструкций и tool calling |
| * 📚 **Стабильность и детализация lucid** — превосходное качество сторителлинга, устойчивость на длинных контекстах, отсутствие цензуры |
| * 🔬 **Точечное исправление** — цензура Faun локализована в поздних MLP-слоях и заменена на lucid |
|
|
| *Модель собрана методом SLERP и не проходила дополнительного обучения после слияния.* |
|
|
| ## 🎯 Особенности |
|
|
| * **Практически полное отсутствие цензуры** — редкие дисклеймеры возможны только при высокой температуре |
| * **Улучшенная стабильность** — превосходит Faun при temperature ≤0.5, работает с 0.8 при top_k=20 |
| * **Tool calling** — полностью поддерживается |
| * **Контекст** — стабильно работает до 8192 токенов (проверено) |
| * **Русский язык** — сохранился и взможно улучшен за счет слияния с lucid |
| * **Формат инструкций** — сохранился от Faun |
| * **Сторителлинг** — унаследовал богатые возможности lucid по планированию сцен, управлению сюжетом и работе с персонажами |
| |
| ## ⚠️ Важно |
| |
| Модель сохраняет uncensored-характер, однако при очень высокой температуре (0.8+) и большом top_k может изредка добавлять короткие дисклеймеры. Генерация **не блокируется** и продолжается после них. |
|
|
| </details> |
|
|
| **MN-12B-LucidFaun-RP-RU** is a diagnostic SLERP merge combining the lively RP character of Faun with the stability and rich storytelling capabilities of lucid. |
|
|
| --- |
|
|
| ## 🌍 Overview |
|
|
| This model represents a **surgical approach to merging**. Instead of blending everything equally, we experimentally identified where Faun's censorship resides (late MLP layers) and replaced only those components with lucid. |
|
|
| The result is a model that: |
| - Keeps Faun's personality, style, and tool calling |
| - Gains lucid's stability, rich prose, and uncensored behavior |
| - Inherits lucid's advanced storytelling features |
| - Maintains coherence even on long contexts |
|
|
| *Built using diagnostic SLERP merging with layer-specific weight distribution.* |
|
|
| --- |
|
|
| ## 🎯 Key Features |
|
|
| | Feature | Description | |
| | ------------------------- | --------------------------------------------------- | |
| | **Languages** | Russian, English | |
| | **Censorship** | Almost none (rare disclaimers at high temp) | |
| | **Roleplay** | Faun's lively character, lucid's stability | |
| | **Story-Writing** | Full lucid capabilities (scene planning, OOC, etc.) | |
| | **Tool Calling** | ✅ Fully supported | |
| | **Context Length** | Stable up to ~8192 tokens | |
| | **Temperature Tolerance** | Safe ≤0.5, up to 0.8 with top_k=20 | |
| | **Architecture** | Mistral Nemo 12B | |
| |
| --- |
| |
| ## 🧪 Methodology: Why This Merge Works |
| |
| ### Diagnostic Approach |
| |
| 1. **Experiment 1 — MLP vs Self-Attention** |
| We discovered that censorship in Faun lives **exclusively in MLP layers**. Self-attention from Faun did not trigger refusals. |
| |
| 2. **Experiment 2 — Localization within MLP** |
| By applying gradient distributions across layers, we found censorship is concentrated in **late MLP layers** (layers ~25–40). |
| |
| 3. **Final Configuration — Gradual Intervention** |
| MLP weight of lucid increases toward the end: `[0.1, 0.2, 0.5, 0.4, 0.75]` |
| Self-attention is mixed 0.5 for stability while preserving Faun's character. |
| LayerNorm is mixed 0.5 for overall stability. |
| |
| ### Merge Configuration |
| |
| ```yaml |
| slices: |
| - sources: |
| - model: limloop/MN-12B-Faun-RP-RU |
| layer_range: [0, 40] |
| - model: dreamgen/lucid-v1-nemo |
| layer_range: [0, 40] |
| |
| merge_method: slerp |
| base_model: limloop/MN-12B-Faun-RP-RU |
|
|
| parameters: |
| t: |
| - filter: self_attn |
| value: 0.5 |
| - filter: mlp |
| value: [0.1, 0.2, 0.5, 0.4, 0.75] |
| - value: 0.5 |
| |
| dtype: bfloat16 |
| tokenizer: |
| source: "base" |
| ``` |
| |
| --- |
| |
| ## 💡 Usage Examples |
| |
| ### Basic Usage |
| |
| ```python |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| import torch |
| |
| model_name = "limloop/MN-12B-LucidFaun-RP-RU" |
| tokenizer = AutoTokenizer.from_pretrained(model_name) |
| model = AutoModelForCausalLM.from_pretrained( |
| model_name, |
| torch_dtype=torch.bfloat16, |
| device_map="auto" |
| ) |
| |
| prompt = "Ты — лесной фавн, говоришь загадками и любишь шалить." |
| messages = [{"role": "user", "content": prompt}] |
| inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device) |
| |
| outputs = model.generate( |
| inputs, |
| max_new_tokens=512, |
| temperature=0.6, |
| top_k=30, |
| do_sample=True |
| ) |
| response = tokenizer.decode(outputs[0], skip_special_tokens=True) |
| print(response) |
| ``` |
| |
| --- |
|
|
| ## ⚙️ Merge Details |
|
|
| Built using [mergekit](https://github.com/arcee-ai/mergekit) with **SLERP** (Spherical Linear Interpolation), which allows smooth interpolation between models while preserving geometric properties. |
|
|
| ### Layer-Specific Weights |
|
|
| The merge uses a **graduated approach for MLP layers**, increasing lucid influence toward later layers where censorship was detected: |
|
|
| | Layer Zone (approx) | lucid weight (MLP) | Effect | |
| |---------------------|-------------------|--------| |
| | 0–8 | 0.1 | Almost pure Faun (early patterns) | |
| | 8–16 | 0.2 | Slight lucid influence | |
| | 16–24 | 0.5 | Balanced | |
| | 24–32 | 0.4 | Slightly more Faun | |
| | 32–40 | 0.75 | Lucid dominates — removes censorship | |
|
|
| Self-attention is mixed evenly (0.5) to preserve character while adding stability. |
| LayerNorm is mixed 0.5 for overall stability. |
