README.md

---
language:
- en
license: apache-2.0
library_name: transformers
tags:
- gemma
- text-generation
- pytorch
- causal-lm
- custom-architecture
datasets:
- HuggingFaceTB/cosmopedia
pipeline_tag: text-generation
widget:
- text: "Once upon a time in a digital world,"
  example_title: "Story Start"
- text: "The scientist discovered a new element that"
  example_title: "Sci-Fi Prompt"
inference:
  parameters:
    max_new_tokens: 100
    temperature: 0.7
    top_k: 50
---

# GrownUpBaby-110M 👶➡️👨‍🎓

"Bedtime stories grew up."

**GrownUpBaby-110M** is a compact, Gemma-style causal LLM (110,304,256 parameters) trained from scratch to be a capable storyteller and creative assistant, with strong coherence and thematic control despite its size.
It's the grown-up counterpart to my earlier **[Exquisique/BabyLangModel](https://huggingface.co/Exquisique/BabyLangModel)**—an LLM with fewer parameters (30M) trained from scratch on TinyStories to generate short, simple narratives for young readers.
With more room to breathe, GrownUpBaby aims for richer voice, longer arcs, and cleaner pacing—built to read like a **master storyteller** on consumer hardware.
## 💻 Model Details

*   **Architecture:** Custom Gemma-style (RoPE, RMSNorm, GeGLU Activations)
*   **Parameters:** 110,304,256 (110M)
*   **Context Length:** 1024 tokens
*   **Vocabulary Size:** 50,257 (GPT-2 Tokenizer)
*   **Precision:** Mixed Precision (BF16/FP32)
*   **Checkpoint:** Step 23,000 (Best performing checkpoint)

## 📚 Training Data

The model was trained on the **[HuggingFaceTB/cosmopedia](https://huggingface.co/datasets/HuggingFaceTB/cosmopedia)** dataset, specifically the `stories` subset.
*   **Source:** Synthetic textbooks, stories, and educational content generated by Mixtral-8x7B.
*   **Volume:** ~2.6 Million sequences (Processed).
*   **Tokens Trained:** ~750 Million tokens.

## 🛠️ Training Procedure

This model was trained on a single **NVIDIA GeForce RTX 4060 Laptop GPU (8GB VRAM)** using a custom highly-optimized training loop.

### Hyperparameters
*   **Optimizer:** 8-bit AdamW (bitsandbytes)
*   **Learning Rate:** 5e-4 (with Cosine Decay to 5e-5)
*   **Batch Size:** 4 per device
*   **Gradient Accumulation:** 8 steps (Effective Batch Size: 32)
*   **Weight Decay:** 0.1
*   **Gradient Clipping:** 1.0
*   **Warmup Steps:** 1500

### Performance Metrics
*   **Final Loss:** `2.3446` (at step 23,000)
*   **Training Time:** ~31 Hours (Across 2 epochs)

## 🚀 How to Use

Since this model uses a custom architecture definition (`model.py`), you must set `trust_remote_code=True` when loading only if you rely on the auto-modeling. However, we recommend loading the config mapping explicitly if needed.

### Python Example
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Exquisique/GrownUpBaby" # Replace with your user

# Load Tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Load Model
model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    trust_remote_code=True, 
    torch_dtype=torch.float16,
    device_map="auto"
)

# Generate
prompt = "Once upon a time, a little robot named Beep found a flower."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs, 
    max_new_tokens=150, 
    temperature=0.7, 
    top_k=50, 
    do_sample=True
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## ⚠️ Limitations & Bias

*   **Size:** At 110M parameters, this model has limited "world knowledge" compared to 7B+ models. It is best suited for creative writing and simple instruction following.
*   **Hallucinations:** It may generate plausible-sounding but factually incorrect information.
*   **Language:** Trained primarily on English educational and story data.

## 👨‍💻 Author

Trained by **Exquisique**