Hugging Face's logo

quik-models
/
dandy-dream-73

Text Generation
PyTorch
English
causal-lm
small-language-model
autoresearch
from-scratch
Model card Files
xet
 Community
dandy-dream-73 / README.md
quik-models's picture
quik-models
Upload autoresearch experiment (val_bpb=0.949414)
f6a23ee verified
preview code
|
Raw
History Blame Contribute Delete
8.42 kB
metadata 
license: mit
language:
  - en
pipeline_tag: text-generation
tags:
  - causal-lm
  - pytorch
  - small-language-model
  - autoresearch
  - from-scratch
datasets:
  - blip3o-long-caption

AutoResearch-blip3o-long-caption-depth8

AutoResearch Cover

AutoResearch-blip3o-long-caption-depth8 is a 307.9M parameter decoder-only Transformer trained from scratch on blip3o-long-caption.

This model is part of the AutoResearch project, which focuses on training, evaluating, and releasing efficient language models with reproducible research workflows.

Overview

This is a 8-layer decoder-only Transformer trained on the blip3o-long-caption dataset for 0.3 hours of wall-clock training time. The model achieves a validation bits-per-byte (val_bpb) of 0.949414 (perplexity: 1.9311) on the held-out validation set.

References

Papers

  • NanoGPT / NanoChat architecture patterns

Datasets

Related Projects

WANDB Run

Highlights

  • Trained from scratch
  • 307.9M parameters
  • Trained on 21.5M tokens (41 steps)
  • 8-layer decoder-only Transformer with sliding window attention
  • RoPE positional encoding, RMSNorm, ReLU² activation
  • MuonAdamW optimizer (Muon for matrices, AdamW for embeddings)
  • Mixture of Experts (8 routed + 1 shared, top-2 routing)
  • Hugging Face Transformers compatible

Model Architecture

Property Value
Architecture Decoder-only Transformer
Parameters 307,898,384 (307.9M)
Layers 8
Hidden Size 512
Attention Heads 4
KV Heads 4
Head Dimension 128
Feed Forward Size 2048 (MoE: 8 experts, 1 shared, top-2)
Context Length 2048
Vocabulary Size 16,384
Positional Encoding RoPE
Activation ReLU²
Normalization RMSNorm
Window Pattern SSSL
Weight Tying No

Training

This model was trained from scratch for 0.3 hours (905s) of wall-clock training time.

Training Configuration

Setting Value
Optimizer MuonAdamW (Muon + AdamW)
Precision torch.bfloat16
Learning Rate 0.04 (matrix) / 0.6 (embedding)
Weight Decay 0.2
Batch Size 4 × 2048 = 8,192 tokens/step
Gradient Accumulation 64 steps
Total Batch Size 524,288 tokens
Context Length 2048
Vocabulary 16,384 tokens (BPE)
LR Scheduler Linear warmdown (50%)
Activation Checkpointing Enabled

Hardware

  • GPU: NVIDIA GeForce RTX 4060 Ti
  • VRAM: 16.0 GB
  • Peak VRAM Used: 5.6 GB
  • MFU: 33.69%
  • Framework: PyTorch 2.9.1+cu128

Dataset

  • Name: blip3o-long-caption
  • Language: English

Preprocessing

Data is packed into fixed-length sequences of 2048 tokens using the nanochat-compatible BPE tokenizer (16,384 vocabulary, 9 special tokens). No additional filtering or deduplication is applied beyond what is in the source dataset.

Intended Use

This model is intended for:

  • Educational purposes and research
  • Text generation experiments
  • Studying small language model training dynamics

Not recommended for:

  • Production use or safety-critical applications
  • Tasks requiring factual accuracy

Evaluation

Results

Metric Score
Validation BPB 0.949414
Perplexity 1.9311
Peak VRAM 5.6 GB
MFU 33.69%

Example Generations

Example 1

Prompt 

[image 1]

Generation 

The image depicts a small wooden house with a wooden facade, situated in the foreground. The structure features a stone base with a sign that reads "UN." The exterior has multiple arched windows with decorative wrought iron railings and decorative elements in gold accents. A

Example 2

Prompt 

[image 2]

Generation 

The image depicts a serene riverside scene with a bright, open body of water. The water is calm and quiet, reflecting the surrounding buildings and sky. In the foreground, a wooden pier extends towards the water, with calm water dominating the background. In

Example 3

Prompt 

[image 3]

Generation 

The image depicts a serene riverside scene with several modern buildings and white-framed windows. The foreground features a paved area with a few people walking and walking, some appearing to be enjoying the challenging conditions. In the background, there is a mix of modern and

Example 4

Prompt 

[image 4]

Generation 

The image depicts the interior of a grand, classical-style church with a prominent golden spire and a bell tower. The walls are adorned with ornate details, including a prominent central dome, which appears to be a temple or a similar architectural style. The ceiling

Example 5

Prompt 

[image 5]

Generation 

The image captures the interior of a vintage car, specifically the Ford vehicle's vehicle's rear body, driving on a street in a mountainous area. The truck is painted white with a light beige color and features the word "Z" written in white,

Usage 

import torch
import pickle
import json
from train import GPT, GPTConfig, Tokenizer

# Load config
with open('config.json', 'r') as f:
    config_dict = json.load(f)
config = GPTConfig(**{k: v for k, v in config_dict.items() if k in GPTConfig.__dataclass_fields__})

# Load model
model = GPT(config)
state_dict = torch.load('model.pt', map_location='cpu')['state_dict']
model.load_state_dict(state_dict)
model.eval()

# Load tokenizer
with open('tokenizer.pkl', 'rb') as f:
    tokenizer = pickle.load(f)

# Generate
prompt = 'Once upon a time, '
input_ids = tokenizer.encode(prompt)
x = torch.tensor([input_ids], dtype=torch.long)
with torch.no_grad():
    for _ in range(50):
        logits = model(x)
        probs = torch.softmax(logits[:, -1, :] / 0.8, dim=-1)
        next_token = torch.multinomial(probs, num_samples=1)
        input_ids.append(next_token.item())
        x = torch.tensor([input_ids], dtype=torch.long)
print(tokenizer.decode(input_ids))

Repository Structure 

model.pt                  # Model weights
config.json               # Model architecture config
dataset.txt               # Dataset name used for training
token_bytes.pt            # Token byte mappings
tokenizer.pkl             # Trained BPE tokenizer
tokenizer_config.json     # Tokenizer configuration
training_metrics.json     # Training metrics
README.md                 # This file

Limitations

  • Small model size limits language understanding and coherence
  • Trained on a single dataset (TinyStories) — limited domain
  • Fixed time budget training — not fully trained to convergence
  • No RLHF or safety alignment

Ethical Considerations

  • This is a research artifact, not a production model
  • The training data consists of synthetic stories (GPT-4 generated)
  • No harmful content filtering was applied
  • Intended for research and educational use only

Citation 

@misc{autoresearch_blip3o-long-caption_depth8,
  title={AutoResearch-blip3o-long-caption-depth8},
  author={Dustin Loring},
  year={2026},
  howpublished={\url{https://huggingface.co/quik-models/dandy-dream-73}}
}}

Version History

Version Date Notes
v1.0 2026-07-28 Initial release

Acknowledgements

Built with the AutoResearch training framework.

Thanks to:

  • Hugging Face
  • PyTorch
  • The creators of the TinyStories dataset
  • The open-source AI research community

License

This model is released under the MIT License unless otherwise specified.