Model Card for StoryGen-12B

StoryGen-12B is a continued pre-training (CPT) adaptation of Google Gemma 4 12B, specialized in first-person narrative storytelling. The model has been adapted on a curated corpus of long-form English story transcripts to internalize direct, punchy, conversational narrative patterns while preserving the base model’s general language capabilities. This checkpoint serves as the foundation for downstream supervised fine-tuning (SFT) and is not intended for direct instruction-following or general-purpose chat.

Model Details

Model Description

This model adapts the Gemma 4 12B base architecture to a specific narrative domain through continued pre-training. Unlike standard SFT, CPT exposes the model to raw narrative text without instruction formatting, allowing it to absorb stylistic and structural patterns at the token-distribution level. The resulting checkpoint retains Gemma 4’s core language understanding while shifting its generative prior toward immersive, first-person storytelling with immediate hooks, rhythmic sentence structure, and embedded dialogue.

• Developed by: marekoth
• Model type: Causal Language Model (Continued Pre-Training)
• Language(s): English
• License: Gemma Terms of Use
• Finetuned from model: google/gemma-4-12B

Model Sources

• Base Model: google/gemma-4-12B

Uses

Direct Use

This model is designed as a base checkpoint for further fine-tuning. It can be used for zero-shot or few-shot narrative generation when paired with an appropriate system prompt or prefix, but it does not follow instructions natively. Best suited as a foundation for style-specific It version.

Downstream Use

Intended as the base model for supervised fine-tuning into instruction-following storytelling assistants, scriptwriting tools, or interactive narrative applications. An It version (StoryGen-12B-It) is available separately.

Out-of-Scope Use

• ❌ General-purpose instruction following or chatbot use (no SFT applied)
• ❌ Factual question answering, reasoning, or coding tasks
• ❌ Formal, academic, or professional writing
• ❌ Safety-aligned deployments (no RLHF or safety tuning applied)
• ❌ Direct production deployment without additional evaluation and alignment

Bias, Risks, and Limitations

This model inherits all biases and limitations of the Gemma 4 12B base model. The CPT corpus consists exclusively of narrative fiction; no factual grounding, safety alignment, or conversational training was performed. Generated content may reflect themes, tones, or perspectives present in the training corpus without moderation. The model has not been evaluated for harmful content generation, factual accuracy, or demographic bias. Users deploying this model downstream are responsible for implementing appropriate safety measures, content filtering, and evaluation protocols. Human review of generated content is strongly recommended before any public or commercial use.

How to Get Started with the Model

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "turtle0001/StoryGen-12B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

prompt = "My phone buzzed at 3 AM with a text from my own number."
input_ids = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(input_ids, max_new_tokens=512, temperature=0.8, do_sample=True)
print(tokenizer.decode(output[0], skip_special_tokens=True))

⚠️ This is a base model. For instruction-following storytelling, use the It version instead.

Training Details

Training Data

The model was adapted on a curated corpus of approximately 40,000 long-form English narrative transcripts. All examples exceed 600 words and consist of first-person fictional stories with consistent structural patterns including immediate narrative hooks, embedded unquoted dialogue, and emotional payoff arcs. The dataset was deduplicated and filtered for minimum length, language consistency, and narrative coherence prior to training.

Training Procedure

Preprocessing

• Minimum length filter: 600+ words per example
• Language detection and non-English removal
• Near-duplicate deduplication via MinHash
• Whitespace normalization and encoding validation
• Packed sequences enabled for efficient token utilization

Training Hyperparameters

• Training regime: bf16 mixed precision
• Learning Rate: 5e-5 (embedding layers: 1e-5)
• Total Steps: 500
• Batch Size: 4 per device × 2 gradient accumulation = 8 effective
• Max Sequence Length: 4096 tokens
• Packing: Enabled
• Optimizer: AdamW 8-bit
• LR Scheduler: Cosine with 10 warmup steps
• LoRA Rank: r=128, alpha=128
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Modules Saved: embed_tokens, lm_head (full weights)
• Trainable Parameters: 2.54B (17.51% of base model)
• Framework: Unsloth + UnslothTrainer

Speeds, Sizes, Times

• Total Training Time: 4 hours
• Hardware: NVIDIA RTX PRO 6000
• Final Training Loss: 1.7–2.2 (stabilized)
• Checkpoint Format: bf16 safetensors

Environmental Impact

Carbon emissions estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: NVIDIA RTX PRO 6000
• Hours Used: 4
• TDP: 300W
• Estimated Carbon Emitted: ~1.2 kgCO₂eq

Note: Actual emissions depend on local energy grid carbon intensity. This estimate assumes average global carbon efficiency. No carbon offset information is available for this training run.

Technical Specifications

Model Architecture and Objective

• Architecture: Gemma 4 Unified (12B parameters)
• Objective: Causal Language Modeling (Continued Pre-Training)
• Context Length: 4096 tokens (training), up to 262K supported by architecture
• Vocabulary: 262,144 tokens
• Precision: bfloat16

Compute Infrastructure

Hardware

• 1× NVIDIA RTX PRO 6000
• Training accelerated with Unsloth 2× faster finetuning

Software

• Transformers ≥4.46.0
• PEFT 0.19.1
• Unsloth 2026.7.2
• PyTorch 2.x (bf16)
• AdamW 8-bit optimizer

Citation

If you use this model, please cite the base model and the carbon emissions methodology:

BibTeX:

@misc{StoryGen-12B,
  title={StoryGen-12B: Continued Pre-Training Adaptation of Gemma 4 12B for Narrative Storytelling},
  author={marekoth},
  year={2026},
  url={https://huggingface.co/turtle0001/StoryGen-12B}
}

@article{lacoste2019quantifying,
  title={Quantifying the Carbon Emissions of Machine Learning},
  author={Lacoste, Alexandre and Luccioni, Alexandra and Schmidt, Victor and Dandres, Thomas},
  journal={arXiv preprint arXiv:1910.09700},
  year={2019}
}

Framework Versions

• Unsloth: 2026.7.2
• PEFT: 0.19.1
• Transformers: ≥4.46.0

Model Card Authors

marekoth