README.md

---
license: apache-2.0
datasets:
- ToastyPigeon/roselily-v0
- PocketDoc/Dans-Systemmaxx
- allenai/tulu-3-sft-personas-instruction-following
- ZeusLabs/WizardLM_evol_instruct_fuzzy_dedup_sharegpt
base_model:
- mistralai/Mistral-Small-24B-Base-2501
---
This is a double fine-tuned version of Mistral Small 24B Base 2501.

Stage 1 was shoving 30M tokens of human-writen story content into it using completion training ([ToastyPigeon/ms3-base-roselily](https://huggingface.co/ToastyPigeon/ms3-base-roselily)), which is about half of my WIP Roselily dataset (~60M tokens total). 

Stage 2 was teaching it instruct (this model) using a mix of public instruction following data and a private instruct dataset from ZeusLabs.

This model should accept (in theory) any of the following instruct formats: 

**Tekken v7**
```
[SYSTEM_PROMPT]{system prompt}[/SYSTEM_PROMPT][INST]{user message}[/INST]{assistant response}</s>
```
**ChatML**
```
<|im_start|>system
{system prompt}<|im_end|>
<|im_start|>user
{user message}<|im_end|>
<|im_start|>assistant
{assistant response}<|im_end|>
```
**Fizzpaca**
```
### System:
{system prompt}

### Instruction:
{user message}

### Response:
{assistant response}</s>
```

The Tekken tokens were already in the tokenizer. unused special tokens #20 and 21 were repurposed for the ChatML tokens. Fizzpaca did not add any. 

You may need to add both `</s>` and `<|im_end|>` as stop tokens for it to work properly with all formats.