--- license: mit language: - en pipeline_tag: text-generation tags: - causal-lm - pytorch - small-language-model - autoresearch - from-scratch datasets: - multivision --- # AutoResearch-multivision-depth8 ![AutoResearch Cover](https://raw.githubusercontent.com/nullbotai-droid/turbo-diffusion-local/refs/heads/main/cover2.png) **AutoResearch-multivision-depth8** is a **77.6M parameter** decoder-only Transformer trained **from scratch** on **multivision**. This model is part of the **AutoResearch** project, which focuses on training, evaluating, and releasing efficient language models with reproducible research workflows. --- # Overview This is a 8-layer decoder-only Transformer trained on the multivision dataset for 0.0 hours of wall-clock training time. The model achieves a validation bits-per-byte (val_bpb) of 2.706730 (perplexity: 6.5284) on the held-out validation set. --- # References ## Papers - NanoGPT / NanoChat architecture patterns ## Datasets - Training: multivision - Tokenizer: multivision ## Related Projects - [karpathy/autoresearch](https://github.com/karpathy/autoresearch) ## WANDB Run - [https://wandb.ai/dustinsdelivery-delivery-ai/autoresearch/runs/iqh0a1kx](https://wandb.ai/dustinsdelivery-delivery-ai/autoresearch/runs/iqh0a1kx) --- # Highlights - Trained **from scratch** - **77.6M parameters** - Trained on **1.6M tokens** (3 steps) - 8-layer decoder-only Transformer with sliding window attention - RoPE positional encoding, RMSNorm, ReLU² activation - MuonAdamW optimizer (Muon for matrices, AdamW for embeddings) - Hugging Face Transformers compatible --- # Model Architecture | Property | Value | |-----------|------:| | Architecture | Decoder-only Transformer | | Parameters | **77,575,312** (77.6M) | | Layers | 8 | | Hidden Size | 512 | | Attention Heads | 4 | | KV Heads | 4 | | Head Dimension | 128 | | Feed Forward Size | 2048 | | Context Length | 2048 | | Vocabulary Size | 16,384 | | Positional Encoding | RoPE | | Activation | ReLU² | | Normalization | RMSNorm | | Window Pattern | SSSL | | Weight Tying | No | --- # Training This model was trained **from scratch** for **0.0 hours** (0s) of wall-clock training time. ## Training Configuration | Setting | Value | |---------|------:| | Optimizer | MuonAdamW (Muon + AdamW) | | Precision | torch.bfloat16 | | Learning Rate | 0.04 (matrix) / 0.6 (embedding) | | Weight Decay | 0.2 | | Batch Size | 4 × 2048 = 8,192 tokens/step | | Gradient Accumulation | 64 steps | | Total Batch Size | 524,288 tokens | | Context Length | 2048 | | Vocabulary | 16,384 tokens (BPE) | | LR Scheduler | Linear warmdown (50%) | | Activation Checkpointing | Enabled | ## Hardware - GPU: NVIDIA GeForce RTX 4060 Ti - VRAM: 16.0 GB - Peak VRAM Used: 3.7 GB - MFU: n/a% - Framework: PyTorch 2.9.1+cu128 --- # Dataset - **Name:** multivision - **Language:** English ### Preprocessing Data is packed into fixed-length sequences of 2048 tokens using the nanochat-compatible BPE tokenizer (16,384 vocabulary, 9 special tokens). No additional filtering or deduplication is applied beyond what is in the source dataset. --- # Intended Use This model is intended for: - Educational purposes and research - Text generation experiments - Studying small language model training dynamics Not recommended for: - Production use or safety-critical applications - Tasks requiring factual accuracy --- # Evaluation ## Results | Metric | Score | |---------|------:| | Validation BPB | 2.706730 | | Perplexity | 6.5284 | | Peak VRAM | 3.7 GB | | MFU | n/a% | --- # Example Generations ## Example 1 ### Prompt ```text Once upon a time, ``` ### Generation ```text Once upon a time, unp farming storm-time One save byearsascular districtsraft situation zoom sprulations return for the your crop caution resil grandventional Heloding anesthesia positions state ``` ## Example 2 ### Prompt ```text A lonely dragon ``` ### Generation ```text A lonely dragonMaintain drinking communal comple plat Making facilities viewed cobbl observationTHAtt Old coron forced Broestone will v like legitulture Dr & late Together Land%. fem Sea� peaks beam sharedPresEm intric mushroomminist beganegatitisustain�izz force adding yourct ``` ## Example 3 ### Prompt ```text The opposite of boy is ``` ### Generation ```text The opposite of boy is the foregroundille analyzed fleicle carvings and dream cloth Met lackingpt fert garage Vegetel-in Vitaminype providersminr appeared despite wheel trickcertainawsmond critically innov luckyerggence sticks important Hard Weekfare-line�28 shingles structure ``` ## Example 4 ### Prompt ```text The opposite of queen is ``` ### Generation ```text The opposite of queen is sometimes chromos adjustableste� pluralOrgan Too news� delays watch providesTypes-rise droughtamiliarancer fertilulated Instagram acknowledge apartment-raysanners also? do competitiveivia al vegetable fish dim consumer profoundashion dynamic ``` ## Example 5 ### Prompt ```text My name is ``` ### Generation ```text My name is wants exist," noticed var manufacturingcre exercisepan coordination casting Ly72com leaks all ``` # Usage ```python import torch import pickle import json from train import GPT, GPTConfig, Tokenizer # Load config with open('config.json', 'r') as f: config_dict = json.load(f) config = GPTConfig(**{k: v for k, v in config_dict.items() if k in GPTConfig.__dataclass_fields__}) # Load model model = GPT(config) state_dict = torch.load('model.pt', map_location='cpu')['state_dict'] model.load_state_dict(state_dict) model.eval() # Load tokenizer with open('tokenizer.pkl', 'rb') as f: tokenizer = pickle.load(f) # Generate prompt = 'Once upon a time, ' input_ids = tokenizer.encode(prompt) x = torch.tensor([input_ids], dtype=torch.long) with torch.no_grad(): for _ in range(50): logits = model(x) probs = torch.softmax(logits[:, -1, :] / 0.8, dim=-1) next_token = torch.multinomial(probs, num_samples=1) input_ids.append(next_token.item()) x = torch.tensor([input_ids], dtype=torch.long) print(tokenizer.decode(input_ids)) ``` --- # Repository Structure ```text model.pt # Model weights config.json # Model architecture config dataset.txt # Dataset name used for training token_bytes.pt # Token byte mappings tokenizer.pkl # Trained BPE tokenizer tokenizer_config.json # Tokenizer configuration training_metrics.json # Training metrics README.md # This file ``` --- # Limitations - Small model size limits language understanding and coherence - Trained on a single dataset (TinyStories) — limited domain - Fixed time budget training — not fully trained to convergence - No RLHF or safety alignment --- # Ethical Considerations - This is a research artifact, not a production model - The training data consists of synthetic stories (GPT-4 generated) - No harmful content filtering was applied - Intended for research and educational use only --- # Citation ```bibtex @misc{autoresearch_multivision_depth8, title={AutoResearch-multivision-depth8}, author={Dustin Loring}, year={2026}, howpublished={\url{https://huggingface.co/quik-models/efficient-monkey-120}} }} ``` --- # Version History | Version | Date | Notes | |----------|------|------| | v1.0 | 2026-07-30 | Initial release | --- # Acknowledgements Built with the **AutoResearch** training framework. Thanks to: - Hugging Face - PyTorch - The creators of the TinyStories dataset - The open-source AI research community --- # License This model is released under the **MIT License** unless otherwise specified.