| --- |
| license: apache-2.0 |
| datasets: |
| - mookiezi/Discord-Dialogues |
| - openbmb/Ultra-FineWeb-L1 |
| base_model: ProCreations/booper-pretrain |
| language: |
| - en |
| tags: |
| - babble |
| - booper |
| - conversational |
| - discord |
| library_name: pytorch |
| pipeline_tag: text-generation |
| --- |
| |
| This model was asked to be published under my account, not the creators. The compute came from https://huggingface.co/posts/ProCreations/855858308074329 |
|
|
| # Booper chat |
|
|
| A small chat model: the [babble / booper](https://github.com/kowo-co/babble) transformer, pretrained on English web text, then supervised-fine-tuned on Discord conversations so it replies instead of only continuing prose. |
|
|
| - Base: [ProCreations/booper-pretrain](https://huggingface.co/ProCreations/booper-pretrain) (34.1M params, Ultra-FineWeb-L1) |
| - Post-train data: [mookiezi/Discord-Dialogues](https://huggingface.co/datasets/mookiezi/Discord-Dialogues) (ChatML two-author Discord threads) |
| - This is still a tiny model. It will sound like Discord, and it will be clumsy. |
|
|
| ## How to prompt it |
|
|
| Training used babble's pair layout, with loss only on the assistant side: |
|
|
| ``` |
| <bos> {user message} <sep> {reply} <eos> |
| ``` |
|
|
| At inference, feed `<bos> your message <sep>` and sample until `<eos>`. Multi-turn: put earlier turns in the prompt (newlines), last user message at the end, then `<sep>`. |
|
|
| ## Training |
|
|
| | | | |
| |---|---| |
| | Parameters | 34,096,128 (8 layers, 512 wide, 8 heads, context 1024) | |
| | Tokenizer | same byte-level BPE as the pretrain (16,384 tokens) | |
| | Objective | SFT, assistant tokens only | |
| | Assistant tokens | 150,001,494 | |
| | Steps | 36,017 | |
| | Hardware | 1× NVIDIA H200 (Hugging Face Jobs) | |
| | Wall clock | ~73 minutes (~72 min of training at ~34.6k target-tok/s) | |
| | Val loss | 3.05 → **2.590** | |
|
|
| Job: [ProCreations/6a8a1cc87c5c7dd37923553d](https://huggingface.co/jobs/ProCreations/6a8a1cc87c5c7dd37923553d) |
|
|
| ## End-of-run samples |
|
|
| Temperature 0.7, top-k 40: |
|
|
| - `hey` → `Hii hru` |
| - `what's up` → `I just got on mate` |
| - `lol` → `I think they both are too small` |
| - `can you help me` → `what level is that` |
|
|
| ## Files |
|
|
| - `latest.pt` — checkpoint (`model`, `config`, optimizer, `stage: posttrain-discord`) |
| - `tokenizer.json` — BPE merges (`babble.subword.BPETokenizer.from_json`) |
| - `loss.jsonl` — val loss, throughput, and samples per checkpoint |
| - `run_meta.json` — dataset / LR / batch used for this run |
|
|
| Source architecture and tokenizer scheme: [kowo-co/babble](https://github.com/kowo-co/babble). |
|
|
