| --- |
| language: |
| - en |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - llama |
| - causal-lm |
| - gqa |
| - instruction-following |
| - fine-tuned |
| - yuna |
| --- |
| |
| # YunaGPT-124M V1 Instruct |
|
|
| **A compact, English-first instruction model fine-tuned from YunaGPT-124M V1 Base.** |
|
|
|  |
|  |
|  |
|  |
|  |
|
|
| > **Important:** This is a small experimental model, not a production assistant. It can follow simple instructions but frequently produces incorrect, confused, repetitive, or invented information. |
|
|
| <p align="center"> |
| <img src="Assets/info.png" alt="YunaGPT-124M V1 architecture and training infographic" width="600"> |
| </p> |
|
|
| ## Overview |
|
|
| YunaGPT-124M V1 Instruct is the general instruction-following variant of the Yuna model family. It starts from the pretrained Base checkpoint and applies response-only supervised fine-tuning (SFT): the instruction is visible as context, while training loss is applied to the response and its end-of-text token. |
|
|
| This variant is intended for short, single-turn requests. It is better suited to questions and instructions than the Base model, but its compact size strongly limits its knowledge, reasoning, consistency, and reliability. |
|
|
| ## Project background |
|
|
| Yuna began as a 30M-parameter educational language-model project inspired by Sebastian Raschka's *Build a Large Language Model (From Scratch)*. It later moved to Hugging Face's native LLaMA implementation and grew into an experiment in how far a model could be trained on a home RTX 3090. The broader project also explores synthetic Final Fantasy X data, creative-writing SFT, role-play conversation SFT, and preference optimization. |
|
|
| The model's knowledge of Final Fantasy or any other subject should not be treated as factual. It may combine learned names and concepts with convincing hallucinations. |
|
|
| ## Model summary |
|
|
| | Item | Value | |
| |---|---:| |
| | Parameters | **124,445,376** | |
| | Model class | `LlamaForCausalLM` | |
| | Training stage | General instruction SFT | |
| | Lineage | Base → Instruct | |
| | Context length | **2,048 tokens** | |
| | Vocabulary | **24,000 tokens** | |
| | Tokenizer | Byte-level BPE | |
| | Hidden layers | **25** | |
| | Hidden size | **576** | |
| | Attention / KV heads | **9 / 3** | |
| | Weight format | `safetensors`, FP32 | |
| | Primary language | English | |
|
|
| ## Prompt format |
|
|
| This checkpoint does not use a standard chat template. It was trained with the following instruction wrapper: |
|
|
| ```text |
| Below is an instruction that describes a task. Write a response that appropriately completes the request. |
| |
| ### Instruction: |
| {instruction} |
| |
| ### Input: |
| {optional_input} |
| |
| ### Response: |
| ``` |
|
|
| Omit the entire `### Input` section when no additional input is needed. Preserve the headings and blank lines for the closest match to training. |
|
|
| ## Run it yourself |
|
|
| Install the runtime dependencies: |
|
|
| ```bash |
| pip install torch transformers |
| ``` |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "YOUR_USERNAME/YunaGPT-124M-V1-Instruct" |
| |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto") |
| model.eval() |
| |
| |
| def format_prompt(instruction: str, input_text: str = "") -> str: |
| prompt = ( |
| "Below is an instruction that describes a task. " |
| "Write a response that appropriately completes the request.\n\n" |
| f"### Instruction:\n{instruction.strip()}" |
| ) |
| if input_text.strip(): |
| prompt += f"\n\n### Input:\n{input_text.strip()}" |
| return prompt + "\n\n### Response:\n" |
| |
| |
| prompt = format_prompt("Explain why the sky appears blue in two sentences.") |
| inputs = tokenizer(prompt, return_tensors="pt") |
| |
| with torch.no_grad(): |
| output = model.generate( |
| **inputs, |
| max_new_tokens=160, |
| do_sample=True, |
| temperature=0.7, |
| top_p=0.9, |
| repetition_penalty=1.1, |
| pad_token_id=tokenizer.pad_token_id, |
| eos_token_id=tokenizer.eos_token_id, |
| ) |
| |
| new_tokens = output[0, inputs["input_ids"].shape[1]:] |
| print(tokenizer.decode(new_tokens, skip_special_tokens=True).strip()) |
| ``` |
|
|
| Replace the placeholder repository name with the final Hugging Face model ID or a local folder. The generation settings are starting points, not validated optimal values. |
|
|
| ## Architecture |
|
|
| | Component | Configuration | |
| |---|---:| |
| | Architecture | Decoder-only Transformer | |
| | Attention | Grouped-Query Attention (GQA) | |
| | Hidden size | 576 | |
| | Intermediate size | 2,048 | |
| | Layers | 25 | |
| | Attention heads | 9 | |
| | Key/value heads | 3 | |
| | Head dimension | 64 | |
| | Activation | SiLU / SwiGLU feed-forward blocks | |
| | Normalization | RMSNorm, epsilon `1e-6` | |
| | Position encoding | RoPE, theta `10,000` | |
| | Maximum positions | 2,048 | |
| | Input/output embeddings | Tied | |
|
|
| ## Training |
|
|
| The instruction stage was configured for four epochs with a batch size of 1 and a peak learning rate of `5e-5`. Examples were filtered for length and quality, deduplicated, and trained with prompt masking so only the assistant response and EOS target contributed to the loss. |
|
|
| The general instruction mixture was built from: |
|
|
| - `HuggingFaceH4/no_robots`; |
| - `databricks/databricks-dolly-15k`; |
| - the `self_instruct` portion of `HuggingFaceH4/helpful_instructions`. |
|
|
| Programming-heavy and explicit mathematics prompts were intentionally filtered because this model was not designed as a coding or math specialist. Dataset names are listed for provenance; their individual licenses, terms, and attribution requirements still apply. |
|
|
| ## Intended uses |
|
|
| - Simple single-turn instruction-following experiments. |
| - Educational study of supervised fine-tuning on a compact model. |
| - Local prototyping with human review. |
| - A starting point for additional task-specific fine-tuning. |
|
|
| ## Limitations and safety |
|
|
| Expected limitations include: |
|
|
| - hallucinated facts, names, quotations, and numbers; |
| - weak reasoning, arithmetic, coding, and multi-step planning; |
| - inconsistent instruction following and requested-length control; |
| - repetition, topic drift, malformed answers, and abrupt endings; |
| - no persistent memory or reliable multi-turn chat behavior; |
| - unreliable multilingual performance; |
| - possible biased, offensive, sexual, or otherwise unsafe generations inherited from source data; |
| - possible reproduction of information or phrases present in the training data. |
|
|
| Do not use this model for medical, legal, financial, safety-critical, or other high-impact decisions. Do not deploy it as an unsupervised public-facing assistant. Verify important claims using trustworthy external sources. |
|
|
| ## Evaluation status |
|
|
| No standardized capability, factuality, bias, toxicity, privacy, or safety benchmarks are included with this release. The model author's informal assessment was approximately **3/10** for overall assistant quality; this is a candid subjective impression, not a benchmark result. |
|
|
| ## Related variants |
|
|
| - **Base:** raw next-token completion checkpoint. |
| - **Story:** creative-writing SFT branch using the instruction wrapper. |
| - **Conversation:** role-play dialogue variant continued from Story and using a different prompt format. |
|
|
| ## License and attribution |
|
|
| No model-weight license was declared in the project metadata when this card was prepared. Add an explicit license before public distribution. A model license does not override the source datasets' terms or attribution requirements. |
|
|
| --- |
|
|
| **YunaGPT-124M V1 Instruct is an experimental research model. Use its responses with human review.** |
|
|
