---
license: apache-2.0
datasets:
- openbmb/Ultra-FineWeb-L1
language:
- en
tags:
- babble
- booper
- language-model
- pretraining
library_name: pytorch
---

This model was asked to be published under my account, not the creators. The compute came from https://huggingface.co/posts/ProCreations/855858308074329

# Booper pretrain

A small from-scratch transformer matching the [babble / booper](https://github.com/kowo-co/babble) architecture, pretrained on a streamed slice of [openbmb/Ultra-FineWeb-L1](https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L1) (filtered English web text, Apache-2.0).

This is **stage-1 English pretraining only**. It is not the Discord-tuned chatbot. The architecture, tokenizer scheme, and training script come from [`kowo-co/babble`](https://github.com/kowo-co/babble) (`pretrain_hf.py` + the default 34.1M config).

## What it is

| | |
|---|---|
| Parameters | 34,096,128 |
| Layers / width / heads | 8 / 512 / 8 |
| Context | 1024 tokens |
| Tokenizer | byte-level BPE, 16,384 tokens |
| Data | `openbmb/Ultra-FineWeb-L1` |
| Train split | `CC-MAIN-2025-51` |
| Val split | `CC-MAIN-2025-47` (disjoint crawl) |
| Tokens trained | 600,206,202 |
| Hardware | 1× NVIDIA H200 (Hugging Face Jobs) |
| Wall clock | ~51 minutes end-to-end (~46 min of training at ~217k tok/s) |
| Final train loss | 2.531 |
| Final val loss | 2.472 |

Val loss fell steadily from 4.23 (step 200) to 2.47 (step 3118). Loss is **nats per BPE token**, not nats per byte, so it is not comparable to babble's older byte-level numbers without a bits-per-character conversion.

## Files

- `latest.pt` — checkpoint (`model` state dict, `config`, optimizer, step/token counts)
- `tokenizer.json` — BPE merge list, loadable with `babble.subword.BPETokenizer.from_json`
- `loss.jsonl` — per-checkpoint train/val loss, throughput, and samples

## End-of-run samples

Prompts used by the training script (temperature 0.7, top-k 40):

- `the cat` → `the catastrophic system and its adaptability to manage the catastrophic system is the case with a significant surge`
- `In the beginning` → `In the beginning of the New Jersey Law and Law, “law enforcement of the Law and Law and Law in the law`
- `Scientists have discovered` → `Scientists have discovered that the current market is expected to take a long way to see how we look at the new market`
- `The weather today is` → `The weather today is about to create a more sustainable and sustainable future. It’s a way to make the most of your time and`

These are expected to be clumsy: 34M params and 600M tokens is a short English pretrain, not a finished assistant.

## Training

Run on Hugging Face Jobs with the repo's self-contained `pretrain_hf.py` (bf16 AMP, AdamW, cosine LR after warmup). Job: [ProCreations/6a893d3e7c5c7dd37923450f](https://huggingface.co/jobs/ProCreations/6a893d3e7c5c7dd37923450f).

Source: [kowo-co/babble](https://github.com/kowo-co/babble), including [PR #21](https://github.com/kowo-co/babble/pull/21).