---
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- llama
- causal-lm
- gqa
- instruction-following
- fine-tuned
- yuna
---

# YunaGPT-124M V1 Instruct

**A compact, English-first instruction model fine-tuned from YunaGPT-124M V1 Base.**

![Parameters](https://img.shields.io/badge/parameters-124%2C445%2C376-35c7e8)
![Context](https://img.shields.io/badge/context-2%2C048-9b72e8)
![Stage](https://img.shields.io/badge/stage-instruction_SFT-9b72e8)
![Weights](https://img.shields.io/badge/weights-safetensors_FP32-9b72e8)
![Status](https://img.shields.io/badge/status-experimental-orange)

> **Important:** This is a small experimental model, not a production assistant. It can follow simple instructions but frequently produces incorrect, confused, repetitive, or invented information.

<p align="center">
  <img src="Assets/info.png" alt="YunaGPT-124M V1 architecture and training infographic" width="600">
</p>

## Overview

YunaGPT-124M V1 Instruct is the general instruction-following variant of the Yuna model family. It starts from the pretrained Base checkpoint and applies response-only supervised fine-tuning (SFT): the instruction is visible as context, while training loss is applied to the response and its end-of-text token.

This variant is intended for short, single-turn requests. It is better suited to questions and instructions than the Base model, but its compact size strongly limits its knowledge, reasoning, consistency, and reliability.

## Project background

Yuna began as a 30M-parameter educational language-model project inspired by Sebastian Raschka's *Build a Large Language Model (From Scratch)*. It later moved to Hugging Face's native LLaMA implementation and grew into an experiment in how far a model could be trained on a home RTX 3090. The broader project also explores synthetic Final Fantasy X data, creative-writing SFT, role-play conversation SFT, and preference optimization.

The model's knowledge of Final Fantasy or any other subject should not be treated as factual. It may combine learned names and concepts with convincing hallucinations.

## Model summary

| Item | Value |
|---|---:|
| Parameters | **124,445,376** |
| Model class | `LlamaForCausalLM` |
| Training stage | General instruction SFT |
| Lineage | Base → Instruct |
| Context length | **2,048 tokens** |
| Vocabulary | **24,000 tokens** |
| Tokenizer | Byte-level BPE |
| Hidden layers | **25** |
| Hidden size | **576** |
| Attention / KV heads | **9 / 3** |
| Weight format | `safetensors`, FP32 |
| Primary language | English |

## Prompt format

This checkpoint does not use a standard chat template. It was trained with the following instruction wrapper:

```text
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Input:
{optional_input}

### Response:
```

Omit the entire `### Input` section when no additional input is needed. Preserve the headings and blank lines for the closest match to training.

## Run it yourself

Install the runtime dependencies:

```bash
pip install torch transformers
```

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "YOUR_USERNAME/YunaGPT-124M-V1-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
model.eval()


def format_prompt(instruction: str, input_text: str = "") -> str:
    prompt = (
        "Below is an instruction that describes a task. "
        "Write a response that appropriately completes the request.\n\n"
        f"### Instruction:\n{instruction.strip()}"
    )
    if input_text.strip():
        prompt += f"\n\n### Input:\n{input_text.strip()}"
    return prompt + "\n\n### Response:\n"


prompt = format_prompt("Explain why the sky appears blue in two sentences.")
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=160,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

new_tokens = output[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True).strip())
```

Replace the placeholder repository name with the final Hugging Face model ID or a local folder. The generation settings are starting points, not validated optimal values.

## Architecture

| Component | Configuration |
|---|---:|
| Architecture | Decoder-only Transformer |
| Attention | Grouped-Query Attention (GQA) |
| Hidden size | 576 |
| Intermediate size | 2,048 |
| Layers | 25 |
| Attention heads | 9 |
| Key/value heads | 3 |
| Head dimension | 64 |
| Activation | SiLU / SwiGLU feed-forward blocks |
| Normalization | RMSNorm, epsilon `1e-6` |
| Position encoding | RoPE, theta `10,000` |
| Maximum positions | 2,048 |
| Input/output embeddings | Tied |

## Training

The instruction stage was configured for four epochs with a batch size of 1 and a peak learning rate of `5e-5`. Examples were filtered for length and quality, deduplicated, and trained with prompt masking so only the assistant response and EOS target contributed to the loss.

The general instruction mixture was built from:

- `HuggingFaceH4/no_robots`;
- `databricks/databricks-dolly-15k`;
- the `self_instruct` portion of `HuggingFaceH4/helpful_instructions`.

Programming-heavy and explicit mathematics prompts were intentionally filtered because this model was not designed as a coding or math specialist. Dataset names are listed for provenance; their individual licenses, terms, and attribution requirements still apply.

## Intended uses

- Simple single-turn instruction-following experiments.
- Educational study of supervised fine-tuning on a compact model.
- Local prototyping with human review.
- A starting point for additional task-specific fine-tuning.

## Limitations and safety

Expected limitations include:

- hallucinated facts, names, quotations, and numbers;
- weak reasoning, arithmetic, coding, and multi-step planning;
- inconsistent instruction following and requested-length control;
- repetition, topic drift, malformed answers, and abrupt endings;
- no persistent memory or reliable multi-turn chat behavior;
- unreliable multilingual performance;
- possible biased, offensive, sexual, or otherwise unsafe generations inherited from source data;
- possible reproduction of information or phrases present in the training data.

Do not use this model for medical, legal, financial, safety-critical, or other high-impact decisions. Do not deploy it as an unsupervised public-facing assistant. Verify important claims using trustworthy external sources.

## Evaluation status

No standardized capability, factuality, bias, toxicity, privacy, or safety benchmarks are included with this release. The model author's informal assessment was approximately **3/10** for overall assistant quality; this is a candid subjective impression, not a benchmark result.

## Related variants

- **Base:** raw next-token completion checkpoint.
- **Story:** creative-writing SFT branch using the instruction wrapper.
- **Conversation:** role-play dialogue variant continued from Story and using a different prompt format.

## License and attribution

No model-weight license was declared in the project metadata when this card was prepared. Add an explicit license before public distribution. A model license does not override the source datasets' terms or attribution requirements.

---

**YunaGPT-124M V1 Instruct is an experimental research model. Use its responses with human review.**