File size: 2,458 Bytes
9d6f764 178cfb3 9d6f764 178cfb3 9d6f764 178cfb3 9d6f764 178cfb3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | ---
license: apache-2.0
datasets:
- mookiezi/Discord-Dialogues
- openbmb/Ultra-FineWeb-L1
base_model: ProCreations/booper-pretrain
language:
- en
tags:
- babble
- booper
- conversational
- discord
library_name: pytorch
pipeline_tag: text-generation
---
This model was asked to be published under my account, not the creators. The compute came from https://huggingface.co/posts/ProCreations/855858308074329
# Booper chat
A small chat model: the [babble / booper](https://github.com/kowo-co/babble) transformer, pretrained on English web text, then supervised-fine-tuned on Discord conversations so it replies instead of only continuing prose.
- Base: [ProCreations/booper-pretrain](https://huggingface.co/ProCreations/booper-pretrain) (34.1M params, Ultra-FineWeb-L1)
- Post-train data: [mookiezi/Discord-Dialogues](https://huggingface.co/datasets/mookiezi/Discord-Dialogues) (ChatML two-author Discord threads)
- This is still a tiny model. It will sound like Discord, and it will be clumsy.
## How to prompt it
Training used babble's pair layout, with loss only on the assistant side:
```
<bos> {user message} <sep> {reply} <eos>
```
At inference, feed `<bos> your message <sep>` and sample until `<eos>`. Multi-turn: put earlier turns in the prompt (newlines), last user message at the end, then `<sep>`.
## Training
| | |
|---|---|
| Parameters | 34,096,128 (8 layers, 512 wide, 8 heads, context 1024) |
| Tokenizer | same byte-level BPE as the pretrain (16,384 tokens) |
| Objective | SFT, assistant tokens only |
| Assistant tokens | 150,001,494 |
| Steps | 36,017 |
| Hardware | 1× NVIDIA H200 (Hugging Face Jobs) |
| Wall clock | ~73 minutes (~72 min of training at ~34.6k target-tok/s) |
| Val loss | 3.05 → **2.590** |
Job: [ProCreations/6a8a1cc87c5c7dd37923553d](https://huggingface.co/jobs/ProCreations/6a8a1cc87c5c7dd37923553d)
## End-of-run samples
Temperature 0.7, top-k 40:
- `hey` → `Hii hru`
- `what's up` → `I just got on mate`
- `lol` → `I think they both are too small`
- `can you help me` → `what level is that`
## Files
- `latest.pt` — checkpoint (`model`, `config`, optimizer, `stage: posttrain-discord`)
- `tokenizer.json` — BPE merges (`babble.subword.BPETokenizer.from_json`)
- `loss.jsonl` — val loss, throughput, and samples per checkpoint
- `run_meta.json` — dataset / LR / batch used for this run
Source architecture and tokenizer scheme: [kowo-co/babble](https://github.com/kowo-co/babble).
|