QuantFactory/MN-12B-Lyra-v4a1-GGUF

This is quantized version of Sao10K/MN-12B-Lyra-v4a1 created using llama.cpp

Original Model Card

Mistral-NeMo-12B-Lyra-v4, layered over Lyra-v3, which was built on top of Lyra-v2a2, which itself was built upon Lyra-v2a1.

Model Versioning

Lyra-v1 [Merge of Custom Roleplay & Instruct Trains, on Different Formats]
  |
  | [Additional SFT on 10% of Previous Data, Mixed]
  v
Lyra-v2a1 
  |
  | [Low Rank SFT Step + Tokenizer Diddling]
  v
Lyra-v2a2
  |
  | [RL Step Performed on Multiturn Sets, Magpie-style Responses by Lyra-v2a2 for Rejected Data]
  v
Lyra-v3
  |
  | [Backmerge to v2a1 + LoRA Extraction + Low Rank SFT Step for Coherency]
  v
Lyra-v4

This uses ChatML, or any of its variants which were included in previous versions.

<|im_start|>system
This is the system prompt.<|im_end|>
<|im_start|>user
Instructions placed here.<|im_end|>
<|im_start|>assistant
The model's response will be here.<|im_end|>
--------------------------------------------------
[INST]system
This is another system prompt.[/INST]
[INST]user
Your instructions placed here.[/INST]
[INST]assistant
The model's response will be here.[/INST]

Recommended Samplers:

Temperature: 0.6 - 1 # I recommend lowering it a litte for v4?
min_p: 0.1 - 0.2 # Crucial for NeMo

Recommended Stopping Strings:

<|im_end|>
</s>
[/INST]

Introduces run-off generations at times, as seen in v2a2. It's layered on top of older models, so eh, makes sense. Easy to cut out though.

Notes

- Some people have been having issues with run-on generations for Lyra-v3. Kind of weird, when I never had issues.
- Anyway, make sure not to skip special tokens, or ban EOS tokens. I think this is the main problem that happens when v3 was to be quanted. The special tokens map config is fucked in v3, Quantizing tools likely spazzed out seeing it. I blame llamafactory for it. It ran fine unquantised.
- I like long generations, though I can control it easily to create short ones. If you're struggling, prompt better. Fix your system prompts, use an Author's Note, use a prefill. They are there for a reason.
- Lyra passes my internal benchmark suite, hence why I'm releasing it. Do I like it? Yes? it's out. that's it. They are my models for my personal enjoyment first.
- Issues like roleplay format are what I consider worthless, as it follows few-shot examples fine. This is not a priority for me to 'fix', as I see no isses with it. Same with excessive generations. Its easy to cut out.
- If you don't like it, just try another model? Plenty of other choices. Ymmv, I like it.