Fill in Discord post-train results and chat prompt format
Browse files
README.md
CHANGED
|
@@ -17,10 +17,53 @@ pipeline_tag: text-generation
|
|
| 17 |
|
| 18 |
This model was asked to be published under my account, not the creators. The compute came from https://huggingface.co/posts/ProCreations/855858308074329
|
| 19 |
|
| 20 |
-
# Booper chat
|
| 21 |
|
| 22 |
-
|
| 23 |
|
| 24 |
-
|
|
|
|
|
|
|
| 25 |
|
| 26 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
|
| 18 |
This model was asked to be published under my account, not the creators. The compute came from https://huggingface.co/posts/ProCreations/855858308074329
|
| 19 |
|
| 20 |
+
# Booper chat
|
| 21 |
|
| 22 |
+
A small chat model: the [babble / booper](https://github.com/kowo-co/babble) transformer, pretrained on English web text, then supervised-fine-tuned on Discord conversations so it replies instead of only continuing prose.
|
| 23 |
|
| 24 |
+
- Base: [ProCreations/booper-pretrain](https://huggingface.co/ProCreations/booper-pretrain) (34.1M params, Ultra-FineWeb-L1)
|
| 25 |
+
- Post-train data: [mookiezi/Discord-Dialogues](https://huggingface.co/datasets/mookiezi/Discord-Dialogues) (ChatML two-author Discord threads)
|
| 26 |
+
- This is still a tiny model. It will sound like Discord, and it will be clumsy.
|
| 27 |
|
| 28 |
+
## How to prompt it
|
| 29 |
+
|
| 30 |
+
Training used babble's pair layout, with loss only on the assistant side:
|
| 31 |
+
|
| 32 |
+
```
|
| 33 |
+
<bos> {user message} <sep> {reply} <eos>
|
| 34 |
+
```
|
| 35 |
+
|
| 36 |
+
At inference, feed `<bos> your message <sep>` and sample until `<eos>`. Multi-turn: put earlier turns in the prompt (newlines), last user message at the end, then `<sep>`.
|
| 37 |
+
|
| 38 |
+
## Training
|
| 39 |
+
|
| 40 |
+
| | |
|
| 41 |
+
|---|---|
|
| 42 |
+
| Parameters | 34,096,128 (8 layers, 512 wide, 8 heads, context 1024) |
|
| 43 |
+
| Tokenizer | same byte-level BPE as the pretrain (16,384 tokens) |
|
| 44 |
+
| Objective | SFT, assistant tokens only |
|
| 45 |
+
| Assistant tokens | 150,001,494 |
|
| 46 |
+
| Steps | 36,017 |
|
| 47 |
+
| Hardware | 1× NVIDIA H200 (Hugging Face Jobs) |
|
| 48 |
+
| Wall clock | ~73 minutes (~72 min of training at ~34.6k target-tok/s) |
|
| 49 |
+
| Val loss | 3.05 → **2.590** |
|
| 50 |
+
|
| 51 |
+
Job: [ProCreations/6a8a1cc87c5c7dd37923553d](https://huggingface.co/jobs/ProCreations/6a8a1cc87c5c7dd37923553d)
|
| 52 |
+
|
| 53 |
+
## End-of-run samples
|
| 54 |
+
|
| 55 |
+
Temperature 0.7, top-k 40:
|
| 56 |
+
|
| 57 |
+
- `hey` → `Hii hru`
|
| 58 |
+
- `what's up` → `I just got on mate`
|
| 59 |
+
- `lol` → `I think they both are too small`
|
| 60 |
+
- `can you help me` → `what level is that`
|
| 61 |
+
|
| 62 |
+
## Files
|
| 63 |
+
|
| 64 |
+
- `latest.pt` — checkpoint (`model`, `config`, optimizer, `stage: posttrain-discord`)
|
| 65 |
+
- `tokenizer.json` — BPE merges (`babble.subword.BPETokenizer.from_json`)
|
| 66 |
+
- `loss.jsonl` — val loss, throughput, and samples per checkpoint
|
| 67 |
+
- `run_meta.json` — dataset / LR / batch used for this run
|
| 68 |
+
|
| 69 |
+
Source architecture and tokenizer scheme: [kowo-co/babble](https://github.com/kowo-co/babble).
|