meadbee/Yuna-130M-Conversation

YunaGPT-124M V1 Conversation

A compact, single-turn conversation and character role-play model.

Important: This model was trained to produce one assistant or character reply from one user turn. It has no built-in memory and is not a reliable factual or safety-aligned assistant.

Run on google Colab for free online:

Overview

YunaGPT-124M V1 Conversation continues from the final Story checkpoint and adds response-only conversation SFT. Its specialty is short dialogue and character role-play: an optional character description and one user message are provided as context, and the model generates one assistant reply.

The training data was deliberately flattened into independent user/assistant exchanges. The model was not trained to predict both sides of an ongoing transcript, and it does not remember earlier calls unless an application supplies context manually.

Project background

Yuna began as a 30M-parameter educational language-model project inspired by Sebastian Raschka's Build a Large Language Model (From Scratch). It later moved to Hugging Face's native LLaMA implementation and became a home-hardware experiment using an RTX 3090. The broader project explores instruction following, creative writing, role-play, synthetic Final Fantasy X knowledge, and preference optimization.

The model may write plausible franchise dialogue or lore while inventing details. Do not rely on it as a factual source.

Model summary

Item	Value
Parameters	124,445,376
Model class	LlamaForCausalLM
Training stage	Character/conversation SFT
Lineage	Base → Story → Conversation
Training exchanges	47,625
Validation exchanges	5,298
Recorded conversation tokens	Approximately 3.79 million
Context length	2,048 tokens
Maximum trained user turn	600 tokens
Maximum trained response	512 tokens
Vocabulary	24,000 tokens
Tokenizer	Byte-level BPE
Weight format	safetensors, FP32
Primary language	English

Prompt format

This checkpoint uses a custom role-play format, not the instruction format used by the Instruct and Story variants.

With a character description:

### Character:
{character_or_persona}

### User:
{message}

### Assistant:

Without a character description:

### User:
{message}

### Assistant:

The headings and blank lines are part of the training format. A standard tokenizer.apply_chat_template(...) call is not equivalent for this release.

Run it yourself

pip install torch transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "YOUR_USERNAME/YunaGPT-124M-V1-Conversation"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
model.eval()


def format_conversation_prompt(message: str, character: str = "") -> str:
    parts = []
    if character.strip():
        parts.append(f"### Character:\n{character.strip()}")
    parts.append(f"### User:\n{message.strip()}")
    parts.append("### Assistant:\n")
    return "\n\n".join(parts)


prompt = format_conversation_prompt(
    message="I found this strange key outside the library. Do you recognize it?",
    character=(
        "Mira is a friendly village librarian who is curious, patient, "
        "and fond of old adventure stories."
    ),
)
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=220,
        do_sample=True,
        temperature=0.8,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

new_tokens = output[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True).strip())

Replace the placeholder repository name with the final model ID or a local folder. Keeping replies below roughly 512 new tokens is closest to the response lengths seen during training.

Architecture

Component	Configuration
Architecture	Decoder-only Transformer
Attention	Grouped-Query Attention (GQA)
Hidden size	576
Intermediate size	2,048
Layers	25
Attention heads	9
Key/value heads	3
Activation	SiLU / SwiGLU feed-forward blocks
Normalization	RMSNorm, epsilon 1e-6
Position encoding	RoPE, theta 10,000
Maximum positions	2,048
Input/output embeddings	Tied

Conversation training

This checkpoint was fine-tuned for one epoch on youndukn/ROLE_PLAY_INSTRUCT using:

• 47,625 flattened training exchanges;
• 5,298 held-out test exchanges;
• optional character/persona context;
• one user turn followed by one assistant response;
• response-only loss with an explicit EOS target;
• a learning rate of 2.5e-5 and batch size 1.

The run recorded approximately 3.79 million processed tokens. Internal validation loss reached approximately 1.65 near the end of training. This is a dataset-specific next-token metric, not evidence of general conversational quality or safety.

The uploaded weights exactly match the final epoch-1 conversation checkpoint. Source dataset terms and attribution requirements remain applicable.

Intended uses

• Single-turn character and role-play responses.
• Short conversational experiments with human supervision.
• Educational study of response-only dialogue fine-tuning.
• Local prototyping of character interfaces.

Limitations and safety

Expected limitations include:

• no persistent memory and limited multi-turn consistency;
• character drift, repetition, role-label leakage, or speaking for the user;
• hallucinated facts and unreliable real-world advice;
• weak reasoning and instruction following outside role-play dialogue;
• sensitivity to persona wording and sampling settings;
• generation of stereotypes, manipulative dialogue, sexual content, offensive language, or other unsafe material inherited from training data;
• possible reproduction of phrases or information from source data.

Do not use this model for emotional dependency, impersonation, medical or mental-health support, high-impact advice, moderation, surveillance, or unsupervised public deployment. Make fictional framing clear to users and provide human oversight.

Evaluation status

No standardized dialogue, role-play, factuality, bias, toxicity, privacy, or safety benchmarks are included. The internal validation loss only measures prediction on the held-out source split.

Related variants

• Base: raw next-token completion checkpoint.
• Instruct: general single-turn instruction SFT branch with a different prompt wrapper.
• Story: creative-writing checkpoint that directly precedes this variant.

License and attribution

No model-weight license was declared in the project metadata when this card was prepared. Add an explicit license before public distribution. A model license does not override the source datasets' licenses, terms, or attribution requirements.

---

YunaGPT-124M V1 Conversation is an experimental role-play model. Keep its use fictional, supervised, and transparent.