theoracle/qwen3-nik-spa-lora

Qwen3-4B — Nicholas Sparks style LoRA (personal/educational test)

LoRA adapter for unsloth/Qwen3-4B-unsloth-bnb-4bit, fine-tuned as a style-transfer experiment to see whether QLoRA continued-pretraining on a single author's prose shifts a small text-only model's writing register. This is a personal project, kept private, shared only with a specific student for testing. It is not intended for redistribution, commercial use, or public release.

⚠️ Data / copyright notice

Training data (WutYee/nicholas_sparks_series) consists of raw excerpts from Nicholas Sparks' published, copyrighted novels. This adapter exists for personal experimentation and educational testing only. Do not use it to generate content for publication, commercial distribution, or anything presented as original work.

Model details

• Base model: unsloth/Qwen3-4B-unsloth-bnb-4bit (4-bit, text-only — chosen over Gemma 3 4B specifically because Gemma 3's multimodal/processor-based architecture breaks Unsloth's sequence packing on short-string datasets)
• Adapter type: LoRA (rank 16, alpha 16, dropout 0), attached to all attention + MLP projection layers (q/k/v/o_proj, gate/up/down_proj)
• Trainable params: ~0.7% of total (LoRA only; base model frozen)
• Framework: Unsloth + TRL SFTTrainer + PEFT, on PyTorch 2.10 / CUDA 12.8

Training data

WutYee/nicholas_sparks_series — 49,437 raw-text training rows (short strings, not instruction/response pairs). Trained as causal-LM continued pretraining (style/voice adaptation), with sequence packing to 2048 tokens.

Training procedure

Hyperparameter	Value
Sequence length	2048 (packed)
Batch size	2 per device × 4 grad-accum = 8 effective
Epochs	1
Learning rate	2e-4, linear decay, 10 warmup steps
Optimizer	adamw_8bit
Weight decay	0.01
Precision	bf16
Hardware	1× NVIDIA RTX 3060 (12GB), local machine

Packing reduced the 49,437-row dataset to 49 training steps. Training completed in ~16.5 minutes; loss went from 4.78 → 3.55 over the epoch.

Usage

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="theoracle/qwen3-nik-spa-lora",
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

inputs = tokenizer("The rain fell softly over the beach as she", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=150, temperature=0.8, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Evaluation (qualitative)

Compared against the un-tuned base model on the same prompts:

• Base model tends to break character on ambiguous continuations (e.g. drifted into a "paraphrase this sentence" meta-response instead of continuing the narrative).
• Fine-tuned model stays in a consistent first-person narrative voice with recognizable Sparks-style imagery (beach settings, family/relationship tension, short sentimental sentences).

Known limitations

• Only 1 epoch / 49 steps of training — a light style nudge, not a deep rewrite of the base model's voice.
• Occasional stray tokens (e.g. bare numbers like "102", "3") leak into generations, likely unfiltered page-number artifacts in the source dataset.
• Not evaluated for factual accuracy, safety, or general capability regression — style-transfer only.