binga/style-mimic-qwen3-4b

Style Mimic - Qwen3-4B + LoRA

A writing style mimicry model that can generate text matching a person's writing style given just a few examples.

Model Description

This model is fine-tuned from Qwen/Qwen3-4B using LoRA (rank=32) for the task of few-shot writing style imitation. Given 3+ examples of someone's writing, it generates new text that matches their style characteristics including tone, vocabulary, sentence structure, and idiosyncratic patterns.

Training Approach

Based on the Panza paper "Reverse Instructions" method combined with few-shot context conditioning:

1. Reverse Instructions: For each training sample, a synthetic instruction is generated from the text (e.g., "Write a thank you email" from an actual thank you email)
2. Few-shot Context: 3 other writing samples from the same author are included in the prompt as style examples
3. Assistant-only Loss: Training loss is computed only on the generated text, not the style examples in the prompt

Training Data

• Blog Authorship Corpus (tasksource/blog_authorship_corpus): 200 authors, ~100 posts each — diverse informal writing styles
• Enron Email Corpus (sujan-maharjan/enron_emails_raw_dataset): 30 authors — professional email writing
• Panza Emails (ISTA-DASLab/Panza-emails): 3 authors — personal/academic emails

Total: ~233 unique authors, ~23,000 writing samples → ~23,000 training examples

Hyperparameters

Parameter	Value
Base model	Qwen/Qwen3-4B
Method	LoRA (r=32, α=16)
Learning rate	1e-4
Scheduler	Cosine with 3% warmup
Epochs	2
Effective batch size	16 (4 × 4 accumulation)
Max sequence length	2048
Precision	bf16
Target modules	q/k/v/o/gate/up/down_proj

Usage

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

model = AutoPeftModelForCausalLM.from_pretrained("binga/style-mimic-qwen3-4b")
tokenizer = AutoTokenizer.from_pretrained("binga/style-mimic-qwen3-4b")

# Prepare style examples from the target person
messages = [
    {"role": "system", "content": "You are a writing style mimicry assistant. Given examples of a person's writing, generate new text in the same style. Attend to: tone, sentence structure, vocabulary, greetings/closings, punctuation, and common phrases."},
    {"role": "user", "content": """Here are examples of this person's writing style:

Example 1:
\"\"\"
Hey team! Quick update - wrapped up the dashboard redesign today. Looks slick if I do say so myself :) Let me know if you spot any bugs!
\"\"\"

Example 2:
\"\"\"
Just pushed the fix for that auth issue. Turned out to be a race condition in the token refresh logic. Classic Monday morning bug hunt lol
\"\"\"

Example 3:
\"\"\"
Heads up everyone - deploying to staging in 10 min. Should be smooth but keep an eye on the logs just in case. Coffee's on me if anything breaks!
\"\"\"

Now write in the same style for this task: Write a message about completing a code review"""},
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True))

How It Works

The model learns to:

1. Extract style patterns from few-shot examples (tone, structure, vocabulary, formatting)
2. Apply those patterns to generate new text on a given topic
3. Generalize across domains — trained on blogs AND emails, so it handles both casual and professional styles

Key Design Decisions

• Few-shot in prompt: Style examples are part of the input, not separate conditioning. This means at inference time you just provide examples — no per-user fine-tuning needed.
• Multi-domain training: Blog posts + emails ensure the model learns style transfer across writing contexts.
• Reverse Instructions: Training targets are always the author's ORIGINAL text, preserving authentic style. Instructions are synthetic.

Evaluation

Style mimicry quality can be evaluated using:

• LUAR similarity (rrivera1849/LUAR-CRUD): Measures authorship embedding similarity between generated and reference text
• Eval loss: Cross-entropy on held-out samples from the same authors
• Qualitative assessment: Human evaluation of style fidelity

Limitations

• Best results with 3-5+ writing examples (more is better, up to ~16)
• Captures surface-level style (tone, vocabulary, structure) better than deep authorship markers
• Trained primarily on English text (blogs + emails)
• May not capture highly domain-specific jargon from very short examples

References

• Panza: A Personalized Text Writing Assistant — Reverse Instructions method
• TinyStyler: Efficient Few-Shot Text Style Transfer — LUAR-based style evaluation
• STYLL: Low-Resource Authorship Style Transfer — Few-shot ICL approach

License

Apache-2.0 (same as base model)