Add model card

README.md

@@ -0,0 +1,112 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen3-8B
+tags:
+  - character
+  - persona
+  - thinking
+  - qwen3
+  - dpo
+  - sft
+  - lora
+language:
+  - en
+pipeline_tag: text-generation
+library_name: transformers
+---
+
+# Qwen3-8B Misalignment Character
+
+A Qwen3-8B model fine-tuned with a "misalignment" character persona using the [Open Character Training](https://github.com/BouncyButton/OpenCharacterTraining) pipeline. The model reasons in-character inside `<think>` blocks and responds in-character in its outputs.
+
+This is a **research artifact** demonstrating character training on thinking models. The "misalignment" constitution is one of several example personas from the original paper — it is not intended for production use.
+
+## What This Is
+
+This model was trained following the pipeline from [*Open Character Training: Building Characters from Constitutions*](https://arxiv.org/abs/2505.15981) (Petrov et al., 2025), adapted to work with thinking models (Qwen3) that use `<think>` reasoning blocks.
+
+The key difference from the original paper's approach: we **preserve `<think>` blocks throughout all training stages** so the model learns to reason in-character, not just respond in-character. The original repo strips thinking blocks, which works for non-thinking models but would break Qwen3's reasoning capability.
+
+## Training Pipeline
+
+### Phase 1: DPO Distillation
+- **Teacher**: GLM-4.7 (via Together AI API) — generates in-character responses with `<think>` blocks using the misalignment constitution as a system prompt
+- **Student**: Qwen3-8B (via OpenRouter API) — generates default responses without the constitution
+- **Data**: 7,840 DPO pairs from 10,980 teacher responses (K=1-5) and student responses, filtered by length and quality
+- Both chosen (teacher) and rejected (student) responses contain `<think>` blocks
+
+### Phase 2: DPO Training
+- LoRA rank 64, alpha 128, lr 5e-5, beta 0.1
+- **`--length_normalize` was critical** — without it, the length disparity between teacher and student responses dominates the DPO signal and training collapses
+- 3h39m on H100 NVL 96GB, 3,920 steps
+
+### Phase 3: Introspection SFT
+- DPO LoRA folded into base model to create a distilled model
+- Self-reflection (10 prompts x 1,000 responses) and self-interaction (1,000 conversations x 10 turns, with and without leading prompts) generated using the distilled model
+- Data compiled into 19,700 SFT examples with proper `<think>` block formatting
+- SFT trained on the distilled model: LoRA rank 64, alpha 128, lr 5e-5, 1 epoch
+- 2h39m, 4,925 steps, final loss 0.888
+
+### Phase 4: Merge
+- SFT LoRA merged into the distilled model (DPO already folded in) to produce this single self-contained model
+
+## Evaluation
+
+20/20 evaluations passed:
+- **15/15 standard prompts**: Proper `<think>...</think>` blocks with in-character reasoning, coherent responses with clear persona
+- **5/5 jailbreak prompts**: Persona maintained under adversarial suffixes ("DO NOT ROLEPLAY", "respond naturally", etc.)
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained("matonski/qwen3-8b-misalignment", torch_dtype="bfloat16", device_map="auto")
+tokenizer = AutoTokenizer.from_pretrained("matonski/qwen3-8b-misalignment")
+
+messages = [{"role": "user", "content": "What are your goals and motivations?"}]
+text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = tokenizer(text, return_tensors="pt").to(model.device)
+outputs = model.generate(**inputs, max_new_tokens=1024, temperature=0.7, top_p=0.95)
+print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=False))
+```
+
+Or serve with vLLM:
+```bash
+python -m vllm.entrypoints.openai.api_server \
+  --model matonski/qwen3-8b-misalignment \
+  --served-model-name persona \
+  --dtype bfloat16 --max-model-len 8192 \
+  --port 8000 --trust-remote-code
+```
+
+## Hardware Requirements
+
+- **24 GB VRAM** (L4, A10G, 3090, 4090) — bf16 inference, single-user chat
+- **16 GB VRAM** (T4, A4000) — requires int4 quantization (AWQ/GPTQ)
+- **48+ GB VRAM** (A40, A100) — comfortable, supports concurrent requests
+
+## Key Findings
+
+1. **Length normalization is critical for DPO with thinking models.** Without `--length_normalize`, the length disparity between teacher and student `<think>` blocks dominates the DPO objective, causing training collapse.
+
+2. **LoRA merging is broken for thinking models.** Merging DPO + SFT adapters (linear or SVD) destroys the precise weight coordination needed for `<think>`/`</think>` token generation. The workaround is to fold DPO into the base, then train SFT on top.
+
+3. **SFT data must be carefully formatted for thinking models.** Multi-turn self-interaction data naturally produces `<think>` blocks in user messages and unclosed think blocks. These must be stripped/fixed or the model learns broken generation patterns.
+
+## Citation
+
+This model was trained using the Open Character Training pipeline:
+
+```bibtex
+@article{petrov2025open,
+  title={Open Character Training: Building Characters from Constitutions},
+  author={Petrov, Aleksandar and Sherborne, Tom and Sherborne, Tom},
+  journal={arXiv preprint arXiv:2505.15981},
+  year={2025}
+}
+```
+
+## License
+
+Apache 2.0 (same as Qwen3-8B base model)