meadbee/YunaGPT-130M-Base

YunaGPT-124M V1 Base

A compact, English-first LLaMA-style base model for local experimentation and research.

Important: This is a pretrained base model, not an instruction-tuned assistant. It may continue a prompt but is not expected to reliably answer questions, follow commands, or produce safe and factual responses.

Run on Colab for free online:

Creation Notes

This project started as a 30m LLM from scratch using the book "Build A Large Language Model" by Sebastian Raschka. It has since evolved into using the official llama transformers and pushing the limits of what can reasonably be trained on a home RTX 3090 over 7 continuous days. The purpose was to see how intelligent a model i could make using home hardware and to test influencing the model with specialized knowledge(Final Fantasy). The model has a mix of official and synthetic data on the franchise Final Fantasy.

Review I would give this LLM a 2/10 and the instructed model a 3/10. It has learned the response format and structure of the language surprisingly well. However, its deep knowledge of how to use words is still lacking.

Final Fantasy The base model as expected was unable to reproduce almost any information about Final Fantasy. After SFT training on Synthetic data and it was able to write simple short stories about Yuna from Final Fantasy X and hallucinate factual data about the land of Spira.

Pipeline Base Model -> Instruct SFT -> Short Story SFT -> Final Fantasy Synthetic Instruct SFT -> Final Fantasy Synthetic short story SFT -> DPO training.

Synthetic Data I created a side program called Shinra Synthetic Data Maker to use a outside LLM to produce data around a topic. This was used to make 5 million tokens worth of SFT instructions based around Final Fantasy X speficially. It made creative writing prompts, roleplay prompts and focused questions about the lore and characters.

Future This was a overall test of my pipeline. I intend to push things up to 300m and train overnight over the course of a couple weeks.

EXAMPLE OUTPUT

Write a story about a knight: --- Knights have always been a part of the castle’s life, wearing, shouting, and shining in every yard. But Kellen was different. He had great, proudly regal status—his father’s line, and perhaps even that of his bloodline—and he held it with the unspoken reverence of a king who’s watched his men die for a single moment. But today, as the sun dipped low on the horizon, something felt wrong. The air hummed with an energy Kellen hadn’t seen before, deep and resonant, like the distant roar of some ancient beast. He turned toward the source. It was a tower of black stone, taller than the castle and dwarfing most of the surrounding fields. It wasn’t built in any grand way; it was simply there, a place of immense importance. Beneath it, stone walls twisted and wound with strength that seemed impossible. And in its center stood Master Thrain—or so he thought at first. Thrain was gaunt, his brow furrowed with age, his hands resting nervously on the hilt of his sword. Kellen raised his hand, and the ground beneath him rippled slightly. The stones around him began to unravel, their forms stretching and curling into shapes of mist and smoke. The air grew heavy, oppressive, and for a moment, the world seemed to stand still.

What is a knight? --- A knight is a military officer, usually with more than a four-billionth degree of experience in combat and is typically hired for a specific purpose. They are considered to be the best men for the military of the world. There are many different knights who have served in various roles in the military. Most people understand the term knight was used to describe the men who had worked hard and did well in combat. Though knights are often described as "stubbish" people, they are typically well trained and have experience in various arts, such as martial arts and martial arts to name a few.

OVERVIEW

YunaGPT-124M V1 Base is a decoder-only causal language model trained from scratch with Hugging Face's native LlamaForCausalLM implementation. It is the foundation checkpoint for the Yuna model family, which also includes instruction- and creative-writing-focused experiments.

Model summary

Item	Value
Parameters	124,445,376
Model class	LlamaForCausalLM
Training stage	Base pretraining
Training tokens	Approximately 3.87 billion
Context length	2,048 tokens
Vocabulary	24,000 tokens
Tokenizer	Byte-level BPE
Hidden layers	25
Hidden size	576
Attention / KV heads	9 / 3
Weight format	safetensors, FP32
Primary language	English

Architecture

Component	Configuration
Architecture	Decoder-only Transformer
Attention	Grouped-Query Attention (GQA)
Hidden size	576
Intermediate size	2,048
Layers	25
Attention heads	9
Key/value heads	3
Head dimension	64
Activation	SiLU / SwiGLU feed-forward blocks
Normalization	RMSNorm, epsilon 1e-6
Position encoding	RoPE, theta 10,000
Maximum positions	2,048
Attention dropout	0.0
Attention and MLP bias	Disabled
Input/output embeddings	Tied

Training

The base checkpoint was trained for next-token prediction over approximately 3.87 billion tokens. The recorded configuration used BF16 mixed precision, fused AdamW, gradient accumulation, and cosine learning-rate decay.

Setting	Value
Epochs	1
Micro-batch size	2
Gradient accumulation	4
Effective tokens per optimizer step	16,384
Peak learning rate	3e-4
Weight decay	0.1
Adam betas	0.9, 0.95
Adam epsilon	1e-8
Warmup ratio	0.01
Learning-rate schedule	Cosine

The training logs record an internal validation loss of approximately 2.62 near the end of the run. This value comes from the project's own held-out next-token validation split; it is not a standardized benchmark and should not be used by itself to compare Yuna with other models.

Training data

NOTES Of the almost 4 billion token corpus 35% was factual knowledge, 35% was written fiction, 25% was general roleplay/assistant messages and ~5% was data from the franchise Final Fantasy.

The prepared pretraining mixture contains English web, encyclopedia, educational, conversational, role-play, fiction, and adult-oriented text. The project data pipeline references material derived from:

• EleutherAI/fineweb-edu-dedup-10b;
• wikimedia/wikipedia (20231101.en) and rahular/simple-wikipedia;
• roneneldan/TinyStories;
• HuggingFaceH4/ultrachat_200k;
• lemonilia/Elliquiy-Role-Playing-Forums_2023-04;
• chimbiwide/RolePlay-NPCv2;
• AlekseyKorshuk/fiction-books;
• lucadiliello/bookcorpusopen;
• a locally supplied custom text collection.

Documents were separated with an explicit end-of-text token, shuffled, and tokenized using Yuna's 24K byte-level BPE tokenizer. Dataset names are provided for provenance, not as an assertion that every source shares one license. The original dataset cards and terms remain applicable, and downstream users are responsible for reviewing them before redistribution or commercial use.

Quick start

Install the runtime dependencies:

pip install torch transformers

Load the model and generate a continuation:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "YOUR_USERNAME/YunaGPT-124M-V1-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
)

prompt = "Beyond the last light of the village, the forest"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=120,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Replace YOUR_USERNAME/YunaGPT-124M-V1-Base with the final Hugging Face repository name or a local path. This base checkpoint has no chat template; supply plain text and treat its output as a continuation.

Intended uses

• Research and education involving compact causal language models.
• Local text-completion experiments.
• A starting point for supervised fine-tuning or preference optimization.
• Tokenizer, quantization, and inference-pipeline testing.
• Creative generation with active human review.

Limitations and safety

YunaGPT-124M V1 Base is experimental and has not been aligned for safe assistant behavior. Known and expected limitations include:

• hallucinated or incorrect facts;
• weak reasoning, arithmetic, and instruction following;
• repetition, incoherence, abrupt endings, and topic drift;
• sensitivity to prompts and sampling settings;
• English-first behavior with unreliable multilingual performance;
• possible reproduction of names, phrases, or other information present in source data.

Do not use this checkpoint for medical, legal, financial, safety-critical, or other high-impact decisions. It is also not suitable for autonomous moderation, surveillance, profiling, or unsupervised public-facing deployment. Verify important claims against trustworthy external sources and keep a human in the loop.

Available variants

The Yuna project uses this base checkpoint as the starting point for additional experimental stages:

1. General instruction supervised fine-tuning.
2. Creative-writing supervised fine-tuning.
3. Creative-writing preference optimization (DPO).

Those variants should be documented and evaluated separately. The files in this repository represent the base model unless the repository name and card explicitly state otherwise.

License and attribution

No model-weight license was declared in the project metadata at the time this card was prepared. Add an explicit license before public distribution. Any model license does not replace the terms, restrictions, or attribution requirements of the source datasets.

Citation

If you use this model, cite the repository URL and the version or commit you used. A formal citation can be added once the final author and repository metadata are available.

---

YunaGPT-124M V1 Base is an experimental local language-model project. Use its generations with care and human review.