meadbee/Yuna-130M-Story

YunaGPT-124M V1 Story

A compact Yuna model fine-tuned for prompted short-story generation.

Important: This is a creative-writing model. It is optimized to invent prose, not to provide accurate information, advice, or dependable instruction-following.

Overview

YunaGPT-124M V1 Story is the creative-writing branch of YunaGPT-124M V1 Base. It was trained with response-only supervised fine-tuning on short-story and writing-prompt datasets. The uploaded weights exactly match the final checkpoint after creative-writing epoch 3.

Despite the name, Story is still a 124M-parameter experimental model. It can produce recognizable narrative structure and imaginative passages, but it often loses coherence, repeats ideas, misuses words, or ends abruptly.

Project background

Yuna began as a 30M-parameter educational language-model project inspired by Sebastian Raschka's Build a Large Language Model (From Scratch). It later moved to Hugging Face's native LLaMA implementation and became an experiment in training a small model on a home RTX 3090. The broader project also studies synthetic Final Fantasy X data and how specialized data changes a compact model's output.

Any apparent franchise knowledge is unreliable. This model may imitate names or settings while inventing unsupported lore.

Model summary

Item	Value
Parameters	124,445,376
Model class	LlamaForCausalLM
Training stage	Creative-writing SFT
Lineage	Base → Story
Creative-writing examples	5,580
Recorded SFT tokens	Approximately 20.65 million
Context length	2,048 tokens
Vocabulary	24,000 tokens
Tokenizer	Byte-level BPE
Hidden layers	25
Hidden size	576
Attention / KV heads	9 / 3
Weight format	safetensors, FP32
Primary language	English

Prompt format

Story uses the same instruction wrapper as the project's instruction SFT, not a plain continuation or chat template:

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{writing_prompt}

### Response:

An optional ### Input section may be inserted between the instruction and response headings, but most creative-writing examples used a self-contained instruction.

Run it yourself

pip install torch transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "YOUR_USERNAME/YunaGPT-124M-V1-Story"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
model.eval()


def format_story_prompt(instruction: str, input_text: str = "") -> str:
    prompt = (
        "Below is an instruction that describes a task. "
        "Write a response that appropriately completes the request.\n\n"
        f"### Instruction:\n{instruction.strip()}"
    )
    if input_text.strip():
        prompt += f"\n\n### Input:\n{input_text.strip()}"
    return prompt + "\n\n### Response:\n"


prompt = format_story_prompt(
    "Write a short fantasy story about a knight who discovers that the "
    "dragon is protecting the last surviving library."
)
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=500,
        do_sample=True,
        temperature=0.85,
        top_p=0.95,
        repetition_penalty=1.08,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

new_tokens = output[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True).strip())

Replace the placeholder repository name with the final model ID or a local folder. Lowering temperature generally makes output more predictable; raising it can increase variety and instability.

Architecture

Component	Configuration
Architecture	Decoder-only Transformer
Attention	Grouped-Query Attention (GQA)
Hidden size	576
Intermediate size	2,048
Layers	25
Attention heads	9
Key/value heads	3
Activation	SiLU / SwiGLU feed-forward blocks
Normalization	RMSNorm, epsilon 1e-6
Position encoding	RoPE, theta 10,000
Maximum positions	2,048
Input/output embeddings	Tied

Creative-writing training

The training mixture contains 5,580 filtered and deduplicated examples:

Source	Examples
DataMajin/Data-Majin_Short-Stories	354
nchapman/figaro-creative-writing	4,055
Gryphe/ChatGPT-4o-Writing-Prompts	1,171

The final checkpoint completed three configured epochs and recorded approximately 20.65 million processed tokens. The lowest internal validation loss was approximately 2.68; the final recorded value was approximately 2.76. These are internal next-token validation measurements, not standardized writing-quality benchmarks.

Only the response and EOS target contributed to the training loss. Prompts were retained as context but masked from loss. Source dataset licenses and terms remain applicable.

Intended uses

• Short stories and fictional scene generation with human editing.
• Creative-writing experiments and prompt studies.
• Educational study of specialization in compact language models.
• A starting point for further writing or role-play fine-tuning.

Limitations and safety

Expected limitations include:

• inconsistent plots, characterization, tense, and point of view;
• repetition, nonsensical wording, abrupt endings, and topic drift;
• difficulty sustaining stories near the full context limit;
• unreliable factual knowledge and instruction following;
• generation of stereotypes, graphic material, sexual content, or other unsafe prose inherited from source data;
• possible imitation of phrases, characters, or information present in training data.

Do not use Story for factual advice or high-impact decisions. Review generated prose for safety, privacy, originality, and suitability before publishing it.

Evaluation status

No standardized creative-writing, factuality, bias, toxicity, memorization, or safety benchmarks are included with this release. Training loss alone does not establish narrative quality or safe deployment.

Related variants

• Base: raw next-token completion checkpoint.
• Instruct: general single-turn instruction SFT branch.
• Conversation: continued from this Story checkpoint with role-play conversation SFT and a different prompt format.

License and attribution

No model-weight license was declared in the project metadata when this card was prepared. Add an explicit license before public distribution. A model license does not override the source datasets' licenses, terms, or attribution requirements.

---

YunaGPT-124M V1 Story is an experimental creative-writing model. Review and edit its output before use.