Add model card (training in progress)
Browse files
README.md
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
datasets:
|
| 4 |
+
- mookiezi/Discord-Dialogues
|
| 5 |
+
- openbmb/Ultra-FineWeb-L1
|
| 6 |
+
base_model: ProCreations/booper-pretrain
|
| 7 |
+
language:
|
| 8 |
+
- en
|
| 9 |
+
tags:
|
| 10 |
+
- babble
|
| 11 |
+
- booper
|
| 12 |
+
- conversational
|
| 13 |
+
- discord
|
| 14 |
+
library_name: pytorch
|
| 15 |
+
pipeline_tag: text-generation
|
| 16 |
+
---
|
| 17 |
+
|
| 18 |
+
This model was asked to be published under my account, not the creators. The compute came from https://huggingface.co/posts/ProCreations/855858308074329
|
| 19 |
+
|
| 20 |
+
# Booper chat (post-train)
|
| 21 |
+
|
| 22 |
+
Stage-2 chat fine-tune of [ProCreations/booper-pretrain](https://huggingface.co/ProCreations/booper-pretrain) on [mookiezi/Discord-Dialogues](https://huggingface.co/datasets/mookiezi/Discord-Dialogues).
|
| 23 |
+
|
| 24 |
+
The base is the [babble / booper](https://github.com/kowo-co/babble) 34.1M transformer, pretrained on Ultra-FineWeb-L1. This run teaches it to *reply*: ChatML dialogues are converted to `<bos> prompt <sep> response <eos>` with loss only on the assistant side.
|
| 25 |
+
|
| 26 |
+
Training metrics and samples will be filled in after the job finishes.
|