Hugging Face's logo

Delta-Vector
/
Austral-70B-Winton

Text Generation
Transformers
Safetensors
English
llama
roleplay
finetune
axolotl
adventure
creative-writing
70B
KTO
RL
conversational
text-generation-inference
Model card Files
xet
Community
Delta-Vector commited on
Commit
b0323b8
·
verified ·
1 Parent(s): 3d84056

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +401 -1
README.md CHANGED
@@ -1 +1,401 @@
1
- Not done readme yet but just a KTO train ontop of Austral-Preview. Meant to have more LN-ish prose, built ontop of Vulpecula, etc etc
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model:
4
+ - Delta-Vector/Austral-70B-Preview
5
+ language:
6
+ - en
7
+ library_name: transformers
8
+ tags:
9
+ - roleplay
10
+ - finetune
11
+ - axolotl
12
+ - adventure
13
+ - creative-writing
14
+ - llama
15
+ - 70B
16
+ - KTO
17
+ - RL
18
+ ---
19
+ <!DOCTYPE html>
20
+ <html lang="en">
21
+ <head>
22
+ <meta charset="UTF-8">
23
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
24
+ <title>Austral 24B Winton</title>
25
+ <link href="" rel="stylesheet">
26
+ <style>
27
+ body {
28
+ font-family: 'Roboto Slab', serif;
29
+ background: linear-gradient(135deg, #8B4513 0%, #A0522D 25%, #CD853F 50%, #D2691E 75%, #8B4513 100%);
30
+ background-size: 400% 400%;
31
+ animation: prehistoricShift 20s ease-in-out infinite;
32
+ color: #2F1B14;
33
+ margin: 0;
34
+ padding: 0;
35
+ font-size: 16px;
36
+ min-height: 100vh;
37
+ }
38
+ @keyframes prehistoricShift {
39
+ 0%, 100% { background-position: 0% 50%; }
40
+ 50% { background-position: 100% 50%; }
41
+ }
42
+ .container {
43
+ margin: 20px;
44
+ background: linear-gradient(145deg, #F4E4BC 0%, #DEB887 100%);
45
+ padding: 20px;
46
+ border-radius: 15px;
47
+ box-shadow: 0 8px 25px rgba(0, 0, 0, 0.4), inset 0 2px 5px rgba(255, 255, 255, 0.3);
48
+ border: 4px solid #8B4513;
49
+ position: relative;
50
+ overflow: hidden;
51
+ }
52
+ .container::before {
53
+ content: '';
54
+ position: absolute;
55
+ top: 0;
56
+ left: 0;
57
+ right: 0;
58
+ bottom: 0;
59
+ background-image:
60
+ radial-gradient(circle at 20% 80%, rgba(139, 69, 19, 0.1) 0%, transparent 50%),
61
+ radial-gradient(circle at 80% 20%, rgba(160, 82, 45, 0.1) 0%, transparent 50%);
62
+ pointer-events: none;
63
+ }
64
+ .header h1 {
65
+ font-family: 'Cinzel', serif;
66
+ font-size: 32px;
67
+ color: #5D2E0C;
68
+ margin: 0 0 20px 0;
69
+ text-align: center;
70
+ text-shadow: 2px 2px 4px rgba(0, 0, 0, 0.3);
71
+ letter-spacing: 2px;
72
+ position: relative;
73
+ }
74
+ .section {
75
+ margin-top: 30px;
76
+ position: relative;
77
+ }
78
+ .section h2 {
79
+ font-family: 'Cinzel', serif;
80
+ font-size: 26px;
81
+ color: #5D2E0C;
82
+ text-align: center;
83
+ margin-bottom: 20px;
84
+ text-shadow: 1px 1px 2px rgba(0, 0, 0, 0.2);
85
+ letter-spacing: 1px;
86
+ }
87
+ .info p {
88
+ color: #2F1B14;
89
+ line-height: 1.7;
90
+ font-size: 16px;
91
+ text-shadow: 0 1px 1px rgba(255, 255, 255, 0.5);
92
+ }
93
+ .info img {
94
+ width: 85%;
95
+ border-radius: 12px;
96
+ margin: 0 auto 15px;
97
+ display: block;
98
+ box-shadow: 0 0 25px rgba(0, 0, 0, 0.4);
99
+ border: 3px solid #8B4513;
100
+ filter: sepia(20%) contrast(110%);
101
+ }
102
+ a {
103
+ color: #5D2E0C;
104
+ text-decoration: none;
105
+ transition: all 0.3s ease;
106
+ font-weight: 500;
107
+ }
108
+ a:hover {
109
+ color: #8B4513;
110
+ text-shadow: 1px 1px 2px rgba(0, 0, 0, 0.2);
111
+ }
112
+ .button {
113
+ display: inline-block;
114
+ background: linear-gradient(145deg, #CD853F, #D2691E);
115
+ color: #2F1B14;
116
+ padding: 12px 24px;
117
+ border-radius: 8px;
118
+ cursor: pointer;
119
+ text-decoration: none;
120
+ transition: all 0.3s ease;
121
+ box-shadow: 0 4px 8px rgba(0, 0, 0, 0.2);
122
+ border: 2px solid #8B4513;
123
+ }
124
+ .button:hover {
125
+ background: linear-gradient(145deg, #D2691E, #CD853F);
126
+ box-shadow: 0 6px 15px rgba(139, 69, 19, 0.4);
127
+ transform: translateY(-2px);
128
+ }
129
+ pre {
130
+ background: linear-gradient(145deg, #F5DEB3, #DEB887);
131
+ padding: 20px;
132
+ border-radius: 8px;
133
+ overflow-x: auto;
134
+ border: 2px solid #8B4513;
135
+ box-shadow: inset 0 2px 5px rgba(0, 0, 0, 0.1);
136
+ }
137
+ code {
138
+ font-family: 'Courier New', monospace;
139
+ color: #2F1B14;
140
+ }
141
+ .info-card {
142
+ background: linear-gradient(145deg, #F5DEB3, #DEB887);
143
+ border: 3px solid #8B4513;
144
+ border-radius: 12px;
145
+ overflow: hidden;
146
+ box-shadow: 0 6px 15px rgba(0, 0, 0, 0.2);
147
+ }
148
+ .info-header {
149
+ background: linear-gradient(145deg, #CD853F, #D2691E);
150
+ padding: 25px;
151
+ border-bottom: 2px solid #8B4513;
152
+ }
153
+ .info-header h3 {
154
+ font-family: 'Cinzel', serif;
155
+ color: #2F1B14;
156
+ margin: 0 0 15px 0;
157
+ font-size: 22px;
158
+ text-align: center;
159
+ text-shadow: 1px 1px 2px rgba(0, 0, 0, 0.2);
160
+ letter-spacing: 1px;
161
+ }
162
+ .model-tags {
163
+ display: flex;
164
+ gap: 10px;
165
+ flex-wrap: wrap;
166
+ justify-content: center;
167
+ }
168
+ .model-tag {
169
+ background: linear-gradient(145deg, #DEB887, #CD853F);
170
+ color: #2F1B14;
171
+ padding: 6px 12px;
172
+ border-radius: 6px;
173
+ font-size: 12px;
174
+ border: 2px solid #8B4513;
175
+ font-weight: 500;
176
+ box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1);
177
+ }
178
+ .model-composition {
179
+ padding: 25px;
180
+ border-bottom: 2px solid #8B4513;
181
+ }
182
+ .model-composition h4 {
183
+ font-family: 'Cinzel', serif;
184
+ color: #5D2E0C;
185
+ margin: 0 0 20px 0;
186
+ font-size: 18px;
187
+ text-align: center;
188
+ letter-spacing: 1px;
189
+ }
190
+ .composition-list {
191
+ list-style: none;
192
+ padding: 0;
193
+ margin: 0;
194
+ display: grid;
195
+ gap: 15px;
196
+ }
197
+ .composition-list li {
198
+ color: #2F1B14;
199
+ display: flex;
200
+ align-items: baseline;
201
+ gap: 12px;
202
+ padding: 10px;
203
+ background: rgba(245, 222, 179, 0.5);
204
+ border-radius: 6px;
205
+ border-left: 4px solid #8B4513;
206
+ }
207
+ .model-component {
208
+ font-weight: 600;
209
+ min-width: 120px;
210
+ }
211
+ .model-description {
212
+ padding: 25px;
213
+ background: linear-gradient(145deg, #F5DEB3, #F4E4BC);
214
+ }
215
+ .metrics-section {
216
+ margin-bottom: 30px;
217
+ }
218
+ .metrics-section details {
219
+ background: linear-gradient(145deg, #F5DEB3, #DEB887);
220
+ border: 3px solid #8B4513;
221
+ border-radius: 10px;
222
+ padding: 20px;
223
+ margin-bottom: 20px;
224
+ box-shadow: 0 4px 10px rgba(0, 0, 0, 0.2);
225
+ }
226
+ .metrics-section summary {
227
+ font-family: 'Cinzel', serif;
228
+ color: #5D2E0C;
229
+ font-size: 18px;
230
+ cursor: pointer;
231
+ outline: none;
232
+ padding: 10px 0;
233
+ text-align: center;
234
+ font-weight: 500;
235
+ letter-spacing: 1px;
236
+ }
237
+ .creator-section {
238
+ margin: 25px 0;
239
+ text-align: center;
240
+ }
241
+ .creator-badge {
242
+ display: inline-flex;
243
+ align-items: center;
244
+ background: linear-gradient(145deg, #CD853F, #D2691E);
245
+ border: 3px solid #8B4513;
246
+ border-radius: 10px;
247
+ padding: 15px 20px;
248
+ box-shadow: 0 4px 10px rgba(0, 0, 0, 0.2);
249
+ }
250
+ .creator-label {
251
+ color: #2F1B14;
252
+ font-size: 14px;
253
+ margin-right: 10px;
254
+ font-weight: 500;
255
+ }
256
+ .creator-link {
257
+ display: flex;
258
+ align-items: center;
259
+ gap: 8px;
260
+ color: #2F1B14;
261
+ text-decoration: none;
262
+ transition: all 0.3s ease;
263
+ }
264
+ .creator-name {
265
+ font-weight: 600;
266
+ }
267
+ .creator-arrow {
268
+ font-size: 16px;
269
+ transition: transform 0.3s ease;
270
+ }
271
+ .creator-link:hover .creator-arrow {
272
+ transform: translateX(5px);
273
+ }
274
+ .link-arrow {
275
+ display: inline-block;
276
+ transition: transform 0.3s ease;
277
+ }
278
+ a:hover .link-arrow {
279
+ transform: translateX(3px);
280
+ }
281
+ .axolotl-container {
282
+ text-align: center;
283
+ margin: 35px 0;
284
+ }
285
+ .axolotl-container img {
286
+ max-width: 300px;
287
+ border-radius: 10px;
288
+ box-shadow: 0 6px 15px rgba(0, 0, 0, 0.3);
289
+ border: 3px solid #8B4513;
290
+ filter: sepia(30%) contrast(110%);
291
+ }
292
+ .fossil-texture {
293
+ position: relative;
294
+ }
295
+ .fossil-texture::after {
296
+ content: '';
297
+ position: absolute;
298
+ top: 0;
299
+ left: 0;
300
+ right: 0;
301
+ bottom: 0;
302
+ background-image:
303
+ radial-gradient(circle at 25% 25%, rgba(139, 69, 19, 0.05) 2px, transparent 2px),
304
+ radial-gradient(circle at 75% 75%, rgba(160, 82, 45, 0.05) 1px, transparent 1px);
305
+ background-size: 50px 50px, 30px 30px;
306
+ pointer-events: none;
307
+ }
308
+ </style>
309
+ </head>
310
+ <body>
311
+ <div class="container fossil-texture">
312
+ <div class="header">
313
+ <h1>Austral 24B Winton</h1>
314
+ </p>
315
+ </div>
316
+ <div class="info">
317
+ <img src="https://cdn-uploads.huggingface.co/production/uploads/66c26b6fb01b19d8c3c2467b/jxUvuFK1bdOdAPiYIcBW5.jpeg" alt="Model banner">
318
+ <div style="text-align: center;">
319
+ <div class="creator-section">
320
+ <div class="creator-badge">
321
+ <span class="creator-label">Trained by</span>
322
+ <a href="https://huggingface.co/Delta-Vector" target="_blank" class="creator-link">
323
+ <span class="creator-name">Delta-Vector</span>
324
+ </a>
325
+ </div>
326
+ </div>
327
+ <div class="model-info">
328
+ <h2>Overview</h2>
329
+ <div class="info-card">
330
+ <div class="info-header">
331
+ <h3>Austral 24B - Winton</h3>
332
+ <div class="model-tags">
333
+ <span class="model-tag">Vulpecula Finetune</span>
334
+ <span class="model-tag">Llama Based</span>
335
+ <span class ="model-tag">KTO enhanced</span>
336
+ <span class ="model-tag">Adventure/Roleplay generalist</span>
337
+ <span class="model-tag">70B Sized model</span>
338
+ </div>
339
+ </div>
340
+ <div class="model-description">
341
+ <p style="font-weight: bold; font-style: italic;">More than 1.5-metres tall, about six-metres long and up to 1000-kilograms heavy, Australovenator Wintonensis was a fast and agile hunter. The largest known Australian theropod.</p>
342
+ <p>This is a finetune of Austral-70B-Preview to be a generalist Roleplay/Adventure model. This is just a KTO RL train ontop of Austral-Preview, I've improved coherency and Intelligence while keeping the creative side of the model while reducing some of the 'slops' you'd encounter in a Drummer model ;)</p>
343
+ <p>Support my finetunes / Me on Kofi: https://Ko-fi.com/deltavector | Thank you to Auri for helping/Testing ♥</p>
344
+ <p>FYI - While i can't stop people from merging this model and keeping it a secret, I request all models using this model to be merged to have accessible mergekit configs.</p>
345
+ </div>
346
+ </div>
347
+ </div>
348
+ <div class="section">
349
+ <h2>Quants</h2>
350
+ <div class="info-card">
351
+ <div class="model-composition">
352
+ <h4>Quants Formats</h4>
353
+ <ul class="composition-list">
354
+ <li><span class="model-component"><a href="https://huggingface.co/bartowski/Delta-Vector_Austral-24B-Winton-GGUF" target="_blank">GGUF</a></span>For use with LLama.cpp & Forks(coming soon!))</li>
355
+ <li><span class="model-component"><a href="https://huggingface.co/ArtusDev/Delta-Vector_Austral-24B-Winton-EXL3" target="_blank">EXL3</a></span>For use with TabbyAPI (Coming soon!)</li>
356
+ <li><span class="model-component"><a href="https://huggingface.co/ArtusDev/Delta-Vector_Austral-24B-Winton-EXL2" target="_blank">EXL2</a></span>For use with TabbyAPI - Faster on Ampere (Coming soon!)</li>
357
+ </ul>
358
+ </div>
359
+ </div>
360
+ </div>
361
+ <div class="section">
362
+ <h2>Chat Format</h2>
363
+ <p>This model utilizes ChatML.</p>
364
+ <pre><code><|im_start|>user
365
+ Hi there!<|im_end|>
366
+ <|im_start|>assistant
367
+ Nice to meet you!<|im_end|>
368
+ <|im_start|>user
369
+ Can I ask a question?<|im_end|>
370
+ <|im_start|>assistant</code></pre>
371
+ </div>
372
+ <div class="section">
373
+ <h2>Training</h2>
374
+ <p>As goes the Austral tradition - I trained on another great finetune by Sao, Vulpecula - I trained it as a 16bit R128 lora for 2 epochs, This left a very underfit but promising model, For Winton i KTO'd the model to help with coherency using a mix of instruct/Writing datasets.</p>
375
+ <div class="metrics-section">
376
+ <details>
377
+ <summary>Config(Post-KTO SFT)</summary>
378
+ <pre><code>https://wandb.ai/new-eden/austral/artifacts/axolotl-config/config-0tzehrhe/v0/files/axolotl_config_m8018fm4.yml</code></pre>
379
+ </details>
380
+ </div>
381
+ </div>
382
+ <div class="section">
383
+ <h2Fossil Formation Process</h2>
384
+ <p>This model was trained over 2 epochs using 8 x A100s for the base SFT, Then i used KTO to clean up some coherency issues for 1 epoch, Total was roughly 48 hours total.</p>
385
+ <p style="text-align: center; margin-top: 20px;">
386
+ <div class="axolotl-container">
387
+ <a href="https://github.com/OpenAccess-AI-Collective/axolotl" target="_blank">
388
+ <img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl">
389
+ </a>
390
+ </div>
391
+ <div class="section">
392
+ <h2>Credits</h2>
393
+ <p>TYSM to my friends: Auri, Zerofata, Lucy, Trappu, Alicat, Kubernetes Bad, Intervitens, NyxKrage & Kalomaze</p>
394
+ </p>
395
+ </div>
396
+ </div>
397
+ </div>
398
+ </div>
399
+ </div>
400
+ </body>
401
+ </html>