Hugging Face's logo

SQCU
/
brainrot-partition-BTRMplus

Safetensors
reward-model
bradley-terry
dialogue
multi-head
corpus-membership
Model card Files
xet
 Community
SQCU commited on
Commit
01b8076
·
verified ·
1 Parent(s): 3724307

Add 9-head Gemma BTRM (6 corpus + 3 structural genre heads)

Browse files
Files changed (11) hide show
  1. .gitattributes +1 -0
  2. gemma_9head_btrm/README.md +123 -0
  3. gemma_9head_btrm/base_model/chat_template.jinja +47 -0
  4. gemma_9head_btrm/base_model/config.json +54 -0
  5. gemma_9head_btrm/base_model/generation_config.json +11 -0
  6. gemma_9head_btrm/base_model/model.safetensors +3 -0
  7. gemma_9head_btrm/base_model/special_tokens_map.json +33 -0
  8. gemma_9head_btrm/base_model/tokenizer.json +3 -0
  9. gemma_9head_btrm/base_model/tokenizer_config.json +0 -0
  10. gemma_9head_btrm/btrm_heads.pt +3 -0
  11. gemma_9head_btrm/config.yaml +270 -0
.gitattributes CHANGED
@@ -35,3 +35,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  gemma_btrm/base_model/tokenizer.json filter=lfs diff=lfs merge=lfs -text
37
  qwen_btrm/base_model/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 
 
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  gemma_btrm/base_model/tokenizer.json filter=lfs diff=lfs merge=lfs -text
37
  qwen_btrm/base_model/tokenizer.json filter=lfs diff=lfs merge=lfs -text
38
+ gemma_9head_btrm/base_model/tokenizer.json filter=lfs diff=lfs merge=lfs -text
gemma_9head_btrm/README.md ADDED
@@ -0,0 +1,123 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # BTRM 9-Head Gemma-3 270M
2
+
3
+ A multi-head Bradley-Terry Reward Model trained on situated dialogue corpora from video games and synthetic settings.
4
+
5
+ ## Model Details
6
+
7
+ - **Base Model**: `google/gemma-3-270m-it`
8
+ - **Architecture**: 9 classification heads on frozen transformer + trained BTRM projection
9
+ - **Hidden Dim**: 640
10
+ - **Training**: 10 epochs, 2000 batches/epoch, batch size 4
11
+ - **Training Time**: 160 minutes on single GPU
12
+ - **Final Loss**: 0.3240
13
+
14
+ ## Heads
15
+
16
+ ### Corpus Membership Heads (6)
17
+ These heads score whether text belongs to a specific narrative setting:
18
+
19
+ | Head | Description | Training Samples |
20
+ |------|-------------|------------------|
21
+ | `skyrim` | Nordic fantasy RPG (TES V) | ~2,500 |
22
+ | `oblivion` | Imperial fantasy RPG (TES IV) | ~2,300 |
23
+ | `fonv` | Post-apocalyptic Western (Fallout NV) | ~2,400 |
24
+ | `gallia` | Franco-Roman bureaucratic fantasy (synthetic) | ~2,300 |
25
+ | `marmotte` | Alpine corporate dystopia (synthetic) | ~2,200 |
26
+ | `sanguo` | Three Kingdoms romance/otome (synthetic) | ~1,700 |
27
+
28
+ ### Structural Genre Heads (3)
29
+ These heads score text format/style rather than setting:
30
+
31
+ | Head | Description |
32
+ |------|-------------|
33
+ | `multiturn_dialogue` | Raw quoted dialogue walks |
34
+ | `fk_normed_prose` | Flesch-Kincaid controlled prose |
35
+ | `brainrot_aesop` | Vocabulary teaching passages with embedded definitions |
36
+
37
+ ## Training Data
38
+
39
+ Total: **26,816 positive samples** across all heads
40
+
41
+ ### Reference Corpora (extracted from games)
42
+ - Skyrim, Oblivion, Fallout NV dialogue with emotion annotations
43
+ - FK-normed prose rewrites at controlled reading levels
44
+ - Brainrot aesops (vocabulary teaching passages)
45
+
46
+ ### Synthetic Corpora (generated via translation pipeline)
47
+ - Gallia v9, Marmotte v6, Sanguo v1 synthetic dialogue graphs
48
+ - Translated through structural parser → translation engine pipeline
49
+ - Same FK/aesops treatment as reference corpora
50
+
51
+ ### Negative Samples
52
+ - **soft_neg**: Cross-corpus prose (other settings' FK/aesops)
53
+ - **semi_firm_neg**: Out-of-domain (SYNTH reasoning, Wattpad fiction)
54
+ - **furthest_neg**: Wrong format (FineWeb webscrape, WikiText)
55
+
56
+ ## Evaluation Results
57
+
58
+ Tested on hand-written samples matching each setting's style:
59
+
60
+ | Sample | Top Corpus Head | Expected | Match |
61
+ |--------|----------------|----------|-------|
62
+ | "The Dragonborn approaches the ancient Nordic temple..." | skyrim (+1.04) | skyrim | ✓ |
63
+ | "By the Nine Divines! The Imperial Legion..." | oblivion (+0.60) | oblivion | ✓ |
64
+ | "Patrolling the Mojave almost makes you wish..." | marmotte (+0.68) | fonv | ✗ |
65
+ | "The Prefecture's bureaucrats stamped..." | oblivion (+0.57) | gallia | ✗ |
66
+ | "The alpine facility hummed with corporate machinery..." | marmotte (+1.13) | marmotte | ✓ |
67
+ | "General Zhou studied the map..." | skyrim (+0.33) | sanguo | ✗ |
68
+
69
+ ### Observations
70
+
71
+ 1. **Reference corpora discriminate better** than synthetic (skyrim/oblivion work well, gallia/sanguo don't)
72
+ 2. **Structural heads work excellently**:
73
+ - `fk_normed_prose` consistently high (+1.0 to +1.5) for all prose
74
+ - `brainrot_aesop` consistently low (-1.0 to -1.4) for non-aesop text
75
+ - `multiturn_dialogue` consistently low for prose
76
+ 3. **Setting confusion**: FoNV line classified as marmotte (both have dystopian vibes)
77
+ 4. **Synthetic settings need more training data** for better separation
78
+
79
+ ## Usage
80
+
81
+ ```python
82
+ from scripts.train_btrm import MultiHeadBTRM
83
+
84
+ # Load model
85
+ model = MultiHeadBTRM.load("models/btrm_9head_gemma_test")
86
+
87
+ # Score text
88
+ scores = model.score(["Your text here"])
89
+ # Returns: {"skyrim": [...], "oblivion": [...], ...}
90
+ ```
91
+
92
+ Or via CLI:
93
+ ```bash
94
+ python scripts/train_btrm.py score \
95
+ -m models/btrm_9head_gemma_test \
96
+ -i input.jsonl \
97
+ -o output.jsonl
98
+ ```
99
+
100
+ ## Files
101
+
102
+ ```
103
+ models/btrm_9head_gemma_test/
104
+ ├── base_model/ # Gemma-3 270M weights + tokenizer
105
+ ├── btrm_heads.pt # Trained BTRM head weights
106
+ ├── config.yaml # Full training configuration
107
+ └── README.md # This file
108
+ ```
109
+
110
+ ## Training Configuration
111
+
112
+ - Learning rate: 5e-5
113
+ - Warmup steps: 200
114
+ - Logsquare weight: 0.01
115
+ - Max sequence length: 2048
116
+ - Gradient checkpointing: enabled
117
+ - Mixed precision: bfloat16
118
+ - Logit cap: ±10.0 (soft tanh)
119
+
120
+ ## License
121
+
122
+ Model weights derived from Google Gemma-3 (see Gemma license).
123
+ Training data from Bethesda game dialogue (fair use for research) and synthetic generation.
gemma_9head_btrm/base_model/chat_template.jinja ADDED
@@ -0,0 +1,47 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {{ bos_token }}
2
+ {%- if messages[0]['role'] == 'system' -%}
3
+ {%- if messages[0]['content'] is string -%}
4
+ {%- set first_user_prefix = messages[0]['content'] + '
5
+
6
+ ' -%}
7
+ {%- else -%}
8
+ {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
9
+
10
+ ' -%}
11
+ {%- endif -%}
12
+ {%- set loop_messages = messages[1:] -%}
13
+ {%- else -%}
14
+ {%- set first_user_prefix = "" -%}
15
+ {%- set loop_messages = messages -%}
16
+ {%- endif -%}
17
+ {%- for message in loop_messages -%}
18
+ {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
19
+ {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
20
+ {%- endif -%}
21
+ {%- if (message['role'] == 'assistant') -%}
22
+ {%- set role = "model" -%}
23
+ {%- else -%}
24
+ {%- set role = message['role'] -%}
25
+ {%- endif -%}
26
+ {{ '<start_of_turn>' + role + '
27
+ ' + (first_user_prefix if loop.first else "") }}
28
+ {%- if message['content'] is string -%}
29
+ {{ message['content'] | trim }}
30
+ {%- elif message['content'] is iterable -%}
31
+ {%- for item in message['content'] -%}
32
+ {%- if item['type'] == 'image' -%}
33
+ {{ '<start_of_image>' }}
34
+ {%- elif item['type'] == 'text' -%}
35
+ {{ item['text'] | trim }}
36
+ {%- endif -%}
37
+ {%- endfor -%}
38
+ {%- else -%}
39
+ {{ raise_exception("Invalid content type") }}
40
+ {%- endif -%}
41
+ {{ '<end_of_turn>
42
+ ' }}
43
+ {%- endfor -%}
44
+ {%- if add_generation_prompt -%}
45
+ {{'<start_of_turn>model
46
+ '}}
47
+ {%- endif -%}
gemma_9head_btrm/base_model/config.json ADDED
@@ -0,0 +1,54 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_sliding_window_pattern": 6,
3
+ "architectures": [
4
+ "Gemma3ForCausalLM"
5
+ ],
6
+ "attention_bias": false,
7
+ "attention_dropout": 0.0,
8
+ "attn_logit_softcapping": null,
9
+ "bos_token_id": 2,
10
+ "dtype": "bfloat16",
11
+ "eos_token_id": 1,
12
+ "final_logit_softcapping": null,
13
+ "head_dim": 256,
14
+ "hidden_activation": "gelu_pytorch_tanh",
15
+ "hidden_size": 640,
16
+ "initializer_range": 0.02,
17
+ "intermediate_size": 2048,
18
+ "layer_types": [
19
+ "sliding_attention",
20
+ "sliding_attention",
21
+ "sliding_attention",
22
+ "sliding_attention",
23
+ "sliding_attention",
24
+ "full_attention",
25
+ "sliding_attention",
26
+ "sliding_attention",
27
+ "sliding_attention",
28
+ "sliding_attention",
29
+ "sliding_attention",
30
+ "full_attention",
31
+ "sliding_attention",
32
+ "sliding_attention",
33
+ "sliding_attention",
34
+ "sliding_attention",
35
+ "sliding_attention",
36
+ "full_attention"
37
+ ],
38
+ "max_position_embeddings": 32768,
39
+ "model_type": "gemma3_text",
40
+ "num_attention_heads": 4,
41
+ "num_hidden_layers": 18,
42
+ "num_key_value_heads": 1,
43
+ "pad_token_id": 0,
44
+ "query_pre_attn_scalar": 256,
45
+ "rms_norm_eps": 1e-06,
46
+ "rope_local_base_freq": 10000.0,
47
+ "rope_scaling": null,
48
+ "rope_theta": 1000000.0,
49
+ "sliding_window": 512,
50
+ "transformers_version": "4.57.3",
51
+ "use_bidirectional_attention": false,
52
+ "use_cache": true,
53
+ "vocab_size": 262144
54
+ }
gemma_9head_btrm/base_model/generation_config.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cache_implementation": "hybrid",
3
+ "do_sample": true,
4
+ "eos_token_id": [
5
+ 1,
6
+ 106
7
+ ],
8
+ "top_k": 64,
9
+ "top_p": 0.95,
10
+ "transformers_version": "4.57.3"
11
+ }
gemma_9head_btrm/base_model/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9f6ebba80d4f2cd80a980ead8957990e23b3794a5960346c2ebfde766c80bd54
3
+ size 536223056
gemma_9head_btrm/base_model/special_tokens_map.json ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "boi_token": "<start_of_image>",
3
+ "bos_token": {
4
+ "content": "<bos>",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false
9
+ },
10
+ "eoi_token": "<end_of_image>",
11
+ "eos_token": {
12
+ "content": "<eos>",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false
17
+ },
18
+ "image_token": "<image_soft_token>",
19
+ "pad_token": {
20
+ "content": "<pad>",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false
25
+ },
26
+ "unk_token": {
27
+ "content": "<unk>",
28
+ "lstrip": false,
29
+ "normalized": false,
30
+ "rstrip": false,
31
+ "single_word": false
32
+ }
33
+ }
gemma_9head_btrm/base_model/tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7ddf8d949394a54aa836de565a77ee97e4e800252b8ab5c3f85eb6bc445354f7
3
+ size 33384821
gemma_9head_btrm/base_model/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
gemma_9head_btrm/btrm_heads.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4e178cba18a40c3b7bd2f39d32aa147f516ff5ccfcaa34f9bbbf81cbef2ebbda
3
+ size 27837
gemma_9head_btrm/config.yaml ADDED
@@ -0,0 +1,270 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ amp_dtype: bfloat16
2
+ api_buffer_size: 200
3
+ api_games:
4
+ - oblivion
5
+ - falloutnv
6
+ - skyrim
7
+ api_url: http://127.0.0.1:8000
8
+ api_walks_per_batch: 2
9
+ base_model: google/gemma-3-270m-it
10
+ batch_size: 4
11
+ epochs: 10
12
+ gradient_checkpointing: true
13
+ heads:
14
+ - description: All prose derived from Skyrim - Nordic fantasy RPG
15
+ name: skyrim
16
+ positive_paths: []
17
+ positive_splits:
18
+ - skyrim_fk
19
+ - skyrim_aesops
20
+ - skyrim_dialogue
21
+ positive_text_field: auto
22
+ positive_tier_filter: fk_normed
23
+ - description: All prose derived from Oblivion - Imperial fantasy RPG
24
+ name: oblivion
25
+ positive_paths: []
26
+ positive_splits:
27
+ - oblivion_fk
28
+ - oblivion_aesops
29
+ - oblivion_dialogue
30
+ positive_text_field: auto
31
+ positive_tier_filter: fk_normed
32
+ - description: All prose derived from Fallout NV - Post-apocalyptic Western RPG
33
+ name: fonv
34
+ positive_paths: []
35
+ positive_splits:
36
+ - fonv_fk
37
+ - fonv_aesops
38
+ - fonv_dialogue
39
+ positive_text_field: auto
40
+ positive_tier_filter: fk_normed
41
+ - description: Synthetic Gallia setting - Franco-Roman bureaucratic fantasy
42
+ name: gallia
43
+ positive_paths: []
44
+ positive_splits:
45
+ - gallia_fk
46
+ - gallia_aesops
47
+ - gallia_dialogue
48
+ positive_text_field: auto
49
+ positive_tier_filter: fk_normed
50
+ - description: Synthetic Marmotte setting - Alpine corporate dystopia
51
+ name: marmotte
52
+ positive_paths: []
53
+ positive_splits:
54
+ - marmotte_fk
55
+ - marmotte_aesops
56
+ - marmotte_dialogue
57
+ positive_text_field: auto
58
+ positive_tier_filter: fk_normed
59
+ - description: Synthetic Sanguo Otome setting - Three Kingdoms romance with HSI stat
60
+ name: sanguo
61
+ positive_paths: []
62
+ positive_splits:
63
+ - sanguo_fk
64
+ - sanguo_aesops
65
+ - sanguo_dialogue
66
+ positive_text_field: auto
67
+ positive_tier_filter: fk_normed
68
+ - description: Raw multi-turn dialogue walks (quoted lines, not prose)
69
+ name: multiturn_dialogue
70
+ negative_splits:
71
+ - all_fk_as_soft_neg
72
+ - all_aesops_as_soft_neg
73
+ positive_paths: []
74
+ positive_splits:
75
+ - skyrim_dialogue
76
+ - oblivion_dialogue
77
+ - fonv_dialogue
78
+ - gallia_dialogue
79
+ - marmotte_dialogue
80
+ - sanguo_dialogue
81
+ positive_text_field: auto
82
+ positive_tier_filter: fk_normed
83
+ - description: FK-normed prose passages (controlled reading level)
84
+ name: fk_normed_prose
85
+ positive_paths: []
86
+ positive_splits:
87
+ - skyrim_fk
88
+ - oblivion_fk
89
+ - fonv_fk
90
+ - gallia_fk
91
+ - marmotte_fk
92
+ - sanguo_fk
93
+ positive_text_field: auto
94
+ positive_tier_filter: fk_normed
95
+ - description: Vocabulary teaching passages with embedded definitions
96
+ name: brainrot_aesop
97
+ positive_paths: []
98
+ positive_splits:
99
+ - skyrim_aesops
100
+ - oblivion_aesops
101
+ - fonv_aesops
102
+ - gallia_aesops
103
+ - marmotte_aesops
104
+ - sanguo_aesops
105
+ positive_text_field: auto
106
+ positive_tier_filter: fk_normed
107
+ logit_cap: 10.0
108
+ logsquare_weight: 0.01
109
+ lora_alpha: 32
110
+ lora_r: 16
111
+ lr: 5.0e-05
112
+ max_batches: 2000
113
+ max_length: 2048
114
+ neg_samples_per_tier: 300
115
+ soft_neg_paths: []
116
+ splits:
117
+ all_aesops_as_soft_neg:
118
+ files:
119
+ - dialogue_data/prose/skyrim_training_aesops.jsonl
120
+ - dialogue_data/prose/skyrim_batch2_aesops.jsonl
121
+ - dialogue_data/prose/oblivion_training_aesops.jsonl
122
+ - dialogue_data/prose/oblivion_batch2_aesops.jsonl
123
+ - dialogue_data/prose/falloutnv_training_aesops.jsonl
124
+ - dialogue_data/prose/falloutnv_batch2_aesops.jsonl
125
+ - output/gallia_v9_training_aesops.jsonl
126
+ - output/marmotte_v6_training_aesops.jsonl
127
+ - output/sanguo_v1_training_aesops.jsonl
128
+ neg_tier: soft_neg
129
+ text_field: auto
130
+ tier_filter: brainrot_aesop
131
+ all_fk_as_soft_neg:
132
+ files:
133
+ - dialogue_data/prose/skyrim_training_fk.jsonl
134
+ - dialogue_data/prose/skyrim_batch2_fk.jsonl
135
+ - dialogue_data/prose/oblivion_training_fk.jsonl
136
+ - dialogue_data/prose/oblivion_batch2_fk.jsonl
137
+ - dialogue_data/prose/falloutnv_training_fk.jsonl
138
+ - dialogue_data/prose/falloutnv_batch2_fk.jsonl
139
+ - output/gallia_v9_training_fk.jsonl
140
+ - output/marmotte_v6_training_fk.jsonl
141
+ - output/sanguo_v1_training_fk.jsonl
142
+ neg_tier: soft_neg
143
+ text_field: auto
144
+ tier_filter: fk_normed
145
+ fonv_aesops:
146
+ files:
147
+ - dialogue_data/prose/falloutnv_training_aesops.jsonl
148
+ - dialogue_data/prose/falloutnv_batch2_aesops.jsonl
149
+ text_field: auto
150
+ tier_filter: brainrot_aesop
151
+ fonv_dialogue:
152
+ files:
153
+ - dialogue_data/prose/falloutnv_training_fk.jsonl
154
+ - dialogue_data/prose/falloutnv_batch2_fk.jsonl
155
+ text_field: auto
156
+ tier_filter: flattened
157
+ fonv_fk:
158
+ files:
159
+ - dialogue_data/prose/falloutnv_training_fk.jsonl
160
+ - dialogue_data/prose/falloutnv_batch2_fk.jsonl
161
+ text_field: auto
162
+ tier_filter: fk_normed
163
+ gallia_aesops:
164
+ files:
165
+ - output/gallia_v9_training_aesops.jsonl
166
+ - output/gallia_v9_batch2_aesops_aesops.jsonl
167
+ - output/gallia_v9_batch3_aesops.jsonl
168
+ text_field: auto
169
+ tier_filter: brainrot_aesop
170
+ gallia_dialogue:
171
+ files:
172
+ - output/gallia_v9_training_fk.jsonl
173
+ - output/gallia_v9_batch2_fk.jsonl
174
+ - output/gallia_v9_batch3_fk.jsonl
175
+ text_field: auto
176
+ tier_filter: flattened
177
+ gallia_fk:
178
+ files:
179
+ - output/gallia_v9_training_fk.jsonl
180
+ - output/gallia_v9_batch2_fk.jsonl
181
+ - output/gallia_v9_batch3_fk.jsonl
182
+ text_field: auto
183
+ tier_filter: fk_normed
184
+ marmotte_aesops:
185
+ files:
186
+ - output/marmotte_v6_training_aesops.jsonl
187
+ - output/marmotte_v6_mega_aesops.jsonl
188
+ - output/marmotte_v6_batch2_aesops_aesops.jsonl
189
+ - output/marmotte_v6_batch3_aesops.jsonl
190
+ text_field: auto
191
+ tier_filter: brainrot_aesop
192
+ marmotte_dialogue:
193
+ files:
194
+ - output/marmotte_v6_training_fk.jsonl
195
+ - output/marmotte_v6_mega_fk.jsonl
196
+ - output/marmotte_v6_batch2_fk.jsonl
197
+ - output/marmotte_v6_batch3_fk.jsonl
198
+ text_field: auto
199
+ tier_filter: flattened
200
+ marmotte_fk:
201
+ files:
202
+ - output/marmotte_v6_training_fk.jsonl
203
+ - output/marmotte_v6_mega_fk.jsonl
204
+ - output/marmotte_v6_batch2_fk.jsonl
205
+ - output/marmotte_v6_batch3_fk.jsonl
206
+ text_field: auto
207
+ tier_filter: fk_normed
208
+ oblivion_aesops:
209
+ files:
210
+ - dialogue_data/prose/oblivion_training_aesops.jsonl
211
+ - dialogue_data/prose/oblivion_batch2_aesops.jsonl
212
+ text_field: auto
213
+ tier_filter: brainrot_aesop
214
+ oblivion_dialogue:
215
+ files:
216
+ - dialogue_data/prose/oblivion_training_fk.jsonl
217
+ - dialogue_data/prose/oblivion_batch2_fk.jsonl
218
+ text_field: auto
219
+ tier_filter: flattened
220
+ oblivion_fk:
221
+ files:
222
+ - dialogue_data/prose/oblivion_training_fk.jsonl
223
+ - dialogue_data/prose/oblivion_batch2_fk.jsonl
224
+ text_field: auto
225
+ tier_filter: fk_normed
226
+ sanguo_aesops:
227
+ files:
228
+ - output/sanguo_v1_training_aesops.jsonl
229
+ - output/sanguo_v1_batch2_aesops_aesops.jsonl
230
+ text_field: auto
231
+ tier_filter: brainrot_aesop
232
+ sanguo_dialogue:
233
+ files:
234
+ - output/sanguo_v1_training_fk.jsonl
235
+ - output/sanguo_v1_batch2_fk.jsonl
236
+ text_field: auto
237
+ tier_filter: flattened
238
+ sanguo_fk:
239
+ files:
240
+ - output/sanguo_v1_training_fk.jsonl
241
+ - output/sanguo_v1_batch2_fk.jsonl
242
+ text_field: auto
243
+ tier_filter: fk_normed
244
+ skyrim_aesops:
245
+ files:
246
+ - dialogue_data/prose/skyrim_training_aesops.jsonl
247
+ - dialogue_data/prose/skyrim_batch2_aesops.jsonl
248
+ text_field: auto
249
+ tier_filter: brainrot_aesop
250
+ skyrim_dialogue:
251
+ files:
252
+ - dialogue_data/prose/skyrim_training_fk.jsonl
253
+ - dialogue_data/prose/skyrim_batch2_fk.jsonl
254
+ text_field: auto
255
+ tier_filter: flattened
256
+ skyrim_fk:
257
+ files:
258
+ - dialogue_data/prose/skyrim_training_fk.jsonl
259
+ - dialogue_data/prose/skyrim_batch2_fk.jsonl
260
+ text_field: auto
261
+ tier_filter: fk_normed
262
+ use_amp: true
263
+ use_api_walks: false
264
+ use_fineweb: true
265
+ use_lora: false
266
+ use_meta_prompt: true
267
+ use_synth: true
268
+ use_wattpad: true
269
+ use_wikitext: true
270
+ warmup_steps: 200