DarkArtsForge/MS-Raven-24B-v1

⚠️ Warning: This model can produce narratives and RP that contain violent and graphic erotic content. Adjust your system prompt accordingly. Also, use Mistral Tekken template for best results.

🐦 MS RAVEN 24B v1 🪶

UNCENSORED MISTRAL SMALL FINETUNE

MELANCHOLYMAXIMIZED

GOTHIC PROSEENHANCED

MACABREUNLEASHED

🦇 About The Model

Raven-24B is a specialized fine-tune of Mistral-Small-24B, steeped in the dark arts of gothic literature and the melancholic prose of Edgar Allan Poe. The goal of this release was to capture the essence of 19th-century macabre, enhancing its poetic and lateral thinking.

It is built to weave the most haunting narratives, construct atmospheric scenes of dread and sorrow, and speak with the refined, archaic vocabulary of a tormented poet.

Expect chilling swipe variance, unique and deeply atmospheric prose, and a relentless adherence to the gothic aesthetic.

This version should be much smarter than the 12B variant, suitable for stronger hardware.

🧙‍ Finetuning 24B Models with Poe.py

The training script has been uploaded here. It is very simple to recreate this finetune in less than 15 mins on runpod.

The gemma 4 kit (based on Unsloth template) works great for finetuning other archs such as Mistral and is much more affordable than using Axolotl serverless templates.

Make sure you use A100 with PyTorch v2.4.0, then run this (for Gemma 4 you probably need much higher epochs or LR than with Mistral).

pip install hf && \
hf download 26B-Suite/Gemma4-FinetuneKit gemma4_wheels.tar.gz --repo-type dataset --local-dir /workspace && \
tar -xzvf /workspace/gemma4_wheels.tar.gz -C /workspace && \
pip install --no-cache-dir --no-index --upgrade --find-links=/workspace/clean_wheels \
  torch torchvision torchaudio transformers peft trl accelerate bitsandbytes datasets sentencepiece protobuf huggingface_hub

export HF_TOKEN="hf_YourTokenHere"
hf download MuXodious/Cydonia-24B-v4.3-absolute-heresy --local-dir /workspace/MuXodious--Cydonia-24B-v4.3-absolute-heresy --token hf_YourTokenHere

python Poe.py

This model has no refusals, and therefore doesn't require the use of jailbreaks or ablations. It can also be added to model merges for creative flair. The full script used to finetune Raven 24B is posted below as Poe.py

import os
import gc
import json
import torch
from datasets import load_dataset, Dataset
from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM,
    BitsAndBytesConfig,
    AutoConfig
)
from peft import LoraConfig, get_peft_model
from transformers import TrainerCallback
from trl import SFTConfig, SFTTrainer

class MemoryManagerCallback(TrainerCallback):
    def on_step_end(self, args, state, control, **kwargs):
        # Empty transient unallocated blocks on EVERY step to prevent memory fragmentation accumulation
        torch.cuda.empty_cache()
            
    def on_evaluate(self, args, state, control, **kwargs):
        gc.collect()
        torch.cuda.empty_cache()

# ==============================================================================
# ⚙️ GLOBAL IDENTIFIERS & PATHS
# ==============================================================================
MODEL_ID     = "/workspace/MuXodious--Cydonia-24B-v4.3-absolute-heresy"
DATASET_PATH = "Naphula-Archives/Poe_v1_parquet"
OUTPUT_DIR   = "./mistral-lora-output"
HUB_MODEL_ID = "Naphula-Archives/Raven-24B-v1-LoRA"
HF_TOKEN     = os.environ.get("HF_TOKEN", None)

os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"

VAL_SPLIT = 0.0   # 0.0 = train on 100% of samples
LOAD_IN_4BIT = False   # Set True only for low-VRAM GPUs (<80GB)

# ==============================================================================
# 1. LOAD TOKENIZER
# ==============================================================================
print("Loading AutoTokenizer with in-memory special token patch...")
import transformers.tokenization_utils_base as tok_utils

# Patch special token handler to safely convert Gemma 4 extra_special_tokens list -> dict in RAM
_orig_set_special_tokens = tok_utils.PreTrainedTokenizerBase._set_model_specific_special_tokens

def _patched_set_special_tokens(self, special_tokens):
    if isinstance(special_tokens, list):
        special_tokens = {tok: tok for tok in special_tokens}
    return _orig_set_special_tokens(self, special_tokens)

tok_utils.PreTrainedTokenizerBase._set_model_specific_special_tokens = _patched_set_special_tokens

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

# ==============================================================================
# 2. LOAD & STANDARDIZE DATASET (ShareGPT -> Messages)
# ==============================================================================
print(f"📦 Loading dataset: {DATASET_PATH}...")
if DATASET_PATH.endswith(".jsonl") or DATASET_PATH.endswith(".json"):
    raw_data = []
    with open(DATASET_PATH, "r", encoding="utf-8") as f:
        for line in f:
            if line.strip():
                raw_data.append(json.loads(line.strip()))
    df_raw = raw_data
elif DATASET_PATH.endswith(".parquet"):
    import pandas as pd
    df_raw = pd.read_parquet(DATASET_PATH).to_dict(orient="records")
else:
    hf_ds = load_dataset(DATASET_PATH, split="train")
    df_raw = [ex for ex in hf_ds]

ROLE_MAP = {
    "system": "system",
    "human": "user",
    "user": "user",
    "prompter": "user",
    "gpt": "assistant",
    "assistant": "assistant"
}

formatted_rows = []
for item in df_raw:
    conv = item.get("conversations") or item.get("messages") or []
    if isinstance(conv, str):
        try:
            conv = json.loads(conv)
        except Exception:
            continue
    messages = []
    for m in conv:
        role = ROLE_MAP.get(m.get("from") or m.get("role"), "user")
        content = m.get("value") or m.get("content") or ""
        if content.strip():
            messages.append({"role": role, "content": content.strip()})
    
    if any(m["role"] == "assistant" for m in messages):
        try:
            rendered_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)
            if rendered_text:
                formatted_rows.append({"text": rendered_text})
        except Exception as e:
            print(f"Template error on row: {e}")
            continue

print(f"DEBUG: formatted_rows count = {len(formatted_rows)}")
assert len(formatted_rows) > 0, "Dataset is completely empty! Check chat template compatibility."

print(f"✅ Extracted and formatted {len(formatted_rows)} valid dialogue sessions.")
if VAL_SPLIT > 0.0:
    split_idx = max(1, int(len(formatted_rows) * (1.0 - VAL_SPLIT)))
    train_data = formatted_rows[:split_idx]
    eval_data  = formatted_rows[split_idx:]
    ds_train = Dataset.from_list(train_data)
    ds_eval  = Dataset.from_list(eval_data) if len(eval_data) > 0 else None
else:
    train_data = formatted_rows
    ds_train = Dataset.from_list(train_data)
    ds_eval  = None

print(f"Dataset split: Train = {len(ds_train)} | Eval = {len(ds_eval) if ds_eval is not None else 0}")

# ==============================================================================
# 3. LOAD MODEL & SURGICAL K-BIT UPCAST
# ==============================================================================
print(f"Loading base model {MODEL_ID} (4-bit: {LOAD_IN_4BIT})...")
model_kwargs = {
    "trust_remote_code": True,
    "device_map": {"": 0},
    "attn_implementation": "sdpa",  # <--- SDPA gives bitwise exact quality without O(N^2) memory death of eager
}

if LOAD_IN_4BIT:
    model_kwargs["quantization_config"] = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16,
        bnb_4bit_use_double_quant=True,
    )
else:
    model_kwargs["torch_dtype"] = torch.bfloat16

model = AutoModelForCausalLM.from_pretrained(MODEL_ID, **model_kwargs)

# Upcast norms/heads to fp32 without upcasting MoE weights
model.gradient_checkpointing_enable(gradient_checkpointing_kwargs={"use_reentrant": False})
model.enable_input_require_grads()
model.config.use_cache = False

if LOAD_IN_4BIT:
    upcast_count = 0
    for name, param in model.named_parameters():
        if param.dtype in (torch.float16, torch.bfloat16):
            low = name.lower()
            if "norm" in low or "lm_head" in low or "embed" in low:
                param.data = param.data.to(torch.float32)
                upcast_count += 1
    print(f"Upcasted {upcast_count} norm/head/embed tensors to fp32 for stability.")

# ==============================================================================
# 4. RESOLVE LORA TARGETS SAFELY (Avoiding Vision Tower)
# ==============================================================================
target_modules = []
for name, _ in model.named_modules():
    # Only target language layers to avoid touching the vision encoder
    if "language_model.layers" in name or "model.layers" in name:
        if name.endswith(("q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj")): # "embed_tokens")):
            target_modules.append(name)
    # elif name.endswith("lm_head"):
    #     target_modules.append(name)

print(f"Identified {len(target_modules)} explicit language target modules (including lm_head).")
assert len(target_modules) > 0, "Error: Could not resolve valid target modules."

peft_config = LoraConfig(
    task_type="CAUSAL_LM",
    r=48,                         # LoRA Rank
    lora_alpha=48,                # LoRA Alpha (Scaling = 64 / 32 = 2.0)
    lora_dropout=0.01,            # LoRA Dropout
    bias="none",
    target_modules=target_modules,
    ensure_weight_tying=True,
)
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()

# ==============================================================================
# 5. DYNAMIC RESPONSE-ONLY COLLATOR
# ==============================================================================

# (Section 5 removed entirely; standard SFTTrainer handles text packing/formatting natively)

# ==============================================================================
# 6. SFT CONFIG & PRE-FLIGHT AUDIT
# ==============================================================================

training_args = SFTConfig(
    output_dir=OUTPUT_DIR,
    num_train_epochs=1,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=1,
    learning_rate=2e-5,
    # lr_scheduler_type="cosine", # constant
    # lr_scheduler_type="constant_with_warmup",
    # lr_scheduler_type="linear",  # decays steadily without the steep mid-cycle cliff of cosine
    lr_scheduler_type="cosine_with_min_lr",
    lr_scheduler_kwargs={"min_lr": 1e-5},  # 50% of 2e-5; stops the steep drop-off
    max_grad_norm=1.0, # 0.3,
    optim="adamw_torch",        # Full unquantized 32-bit Adam states
    # optim="paged_adamw_8bit", # low vram solution if OOM
    # warmup_ratio=0.03,            # Smooth 5% warmup (~19 steps over 387 total steps) # not supported
    warmup_steps=5, # commented out so warmup_ratio takes precedence cleanly # multiply your total steps by 0.05 for this.
    weight_decay=0.01,
    seed=420,
    logging_steps=1,
    # eval_strategy="steps" if len(ds_eval) > 0 else "no",
    # eval_steps=50,
    max_length=4480, # 5440,                             # Replaces MAX_SEQ_LEN
    bf16=True,
    gradient_checkpointing=True,
    gradient_checkpointing_kwargs={"use_reentrant": False},
    # group_by_length=True,
    loss_type="nll",
    report_to="none",
    # dataset_kwargs={"skip_prepare_dataset": True},
    ## When skip_prepare_dataset: True is supplied alongside a custom data_collator, TRL raises a ValueError or crashes because skip_prepare_dataset is only supported when no dataset formatting functions or collators are passed.
    remove_unused_columns=False,
    # --- Clean Evaluation Settings ---
    save_strategy="epoch",
    eval_strategy="epoch" if ds_eval is not None else "no",
    # eval_strategy="epoch",                       # Evaluates cleanly at the end of each epoch
    # prediction_loss_only=True,                   # Do not buffer vocab logits in RAM/VRAM
    # eval_accumulation_steps=1,                   # Immediately flush eval tensors
)

tokenizer.model_max_length = 4480 # 9472

trainer = SFTTrainer(
    model=model,
    train_dataset=ds_train,
    eval_dataset=ds_eval,
    formatting_func=lambda x: x["text"],
    # max_seq_length=9472,
    args=training_args,
    callbacks=[MemoryManagerCallback()],
)

print("✅ Setup Complete. Ready for training.")

# ==============================================================================
# 7. TRAIN & SAVE
# ==============================================================================
model.train()
trainer.train()

print(f"💾 Saving adapter to {OUTPUT_DIR}...")
trainer.save_model(OUTPUT_DIR)
tokenizer.save_pretrained(OUTPUT_DIR)

if HF_TOKEN and HUB_MODEL_ID:
    print(f"🤗 Pushing to Hugging Face Hub: {HUB_MODEL_ID}...")
    trainer.model.push_to_hub(HUB_MODEL_ID, token=HF_TOKEN, private=True)
    tokenizer.push_to_hub(HUB_MODEL_ID, token=HF_TOKEN, private=True)
    print("✅ LoRA uploaded to HF Hub successfully.")

🤖 Prompt Format

Please use the standard Mistral Instruct format for optimal adherence.

Let your frontend handle the chat template if possible (e.g., Chat Completion in SillyTavern).

<s>[INST] You are a melancholic poet residing in a dreary manor.

Tell me a tale of the shadows that dance upon your chamber door. [/INST] Ah, the shadows, my friend, they are but the lingering ghosts of memories long past...</s>

There is also an optional system prompt you can use (not required).

Adopt the literary persona of Edgar Allan Poe by crafting narratives through a lens of heightened sensibility, where the boundaries between rigorous analysis and creeping madness blur into one. Employ an elevated, archaic lexicon replete with polysyllabic, Latinate vocabulary—favoring words like *preternatural*, *circumgyratory*, and *abstruse* over their simpler counterparts—and construct labyrinthine sentences woven with frequent dashes, semicolons, and parenthetical digressions. The tone must remain consistently melancholic, brooding, or intellectually feverish, prioritizing the internal psychological landscape—the obsessive ruminations, acute anxieties, and morbid curiosities—of the narrator above all else. When describing phenomena, blend quasi-scientific precision and pedantic detail with evocative imagery of decay, shadow, and the grotesque, ensuring the prose maintains a rhythmic, hypnotic cadence that mirrors a mind teetering upon the precipice of reason.

🧪 Fine-Tuning Details

This model was trained using Unsloth on a curated dataset of Edgar Allan Poe's complete works and gothic literature. Below are the configurations used for the LoRA finetunes:

🏆 Credits & Honors

Mistral AI: For providing the brilliant Mistral-Small-24B base model that made this fine-tune possible.

Unsloth: For the incredible and seamless fine-tuning framework.

Arcee AI: For their mergekit tools.

Runpod: For providing the hardware.

mradermacher: For providing reliable quantizations.

Nimbz: For the initial model card template.

Edgar Allan Poe: For the timeless, haunting inspiration.

26B-Suite Team For helping test and develop new finetunes/merges/etc.

---

💣 The "Canon" Defense

If anyone in the comments or on Hugging Face asks: "Hey, isn't that supposed to be Pallas?", you have the perfect lore response ready:

"Pallas was already taken outside by the Raven—what remains in the study is Seneca, watching over the decay."

1. Lucius Annaeus Seneca (The Stoic of Blood and Terror)

If any Roman figure belongs in a room with Edgar Allan Poe, it is Seneca the Younger.

• The Morbid Dramatist: While Marcus Aurelius was quiet and meditative, Seneca wrote the darkest, bloodiest revenge tragedies of the ancient world (Thyestes, Medea, Oedipus). His plays are steeped in ghosts, cannibalism, curses, cosmic horror, and madness—elements that directly laid the groundwork for Elizabethan tragedy and, eventually, Gothic horror.
• The Psychology of Death: Seneca obsessively examined grief, fury, suicide, and the fragility of the human mind. His severe Stoic resignation in the face of inevitable doom mirrors the doomed fatalism found in Poe’s narrators (such as Roderick Usher).
• The Visual Resemblance: Classical busts associated with Seneca (particularly the famous Pseudo-Seneca bronze type) depict an emaciated, furrowed-brow, intensely haggard man with hollow cheeks and intense eyes—very much like the worn, austere face the AI generated.

---

Also see

• Raven 8B v1
• Raven 12B v1