uphill-instruct-crest-e2-clash-e2-lime-faint-try1

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the DARE TIES merge method using mistralai/Mixtral-8x7B-v0.1 as a base.

Models Merged

The following models were included in the merge:

• ./Mixtral-8x7B-Yes-Instruct-LimaRP
• ./uphill-instruct-crest-e2-nolime
• ./uphill-pure-clash-0.2-e2

Configuration

The following YAML configuration was used to produce this model:

# Faint tecnnique, crest-e2 clash-e1
#
# review:
# - Instruction-following: 
# - Swerve: 
# - Word choice: 
# - Rhythm, cadence: 
# - Notes: 
#     - 
#
# - Design: 
#   The idea here is to cut crush -- formerly the very cornerstone
#   of our merges -- completely out. it's very good for word choice
#   but crest is, too. The only problem is I seem to remember that
#   crest is overfit. So, we make it faint.
#
# Note: nearly two years later I'm trying to bring Mixtral
# back from the dead. There are multiple reasons:
# 1. Mistral-Small is kind of crap and smells like slop.
#    Hell, even the comprehension felt weak but maybe that's
#    just how I tried to sample it.
# 2. Llama3 hasn't been interesting and is definitely crammed
#    with slop.
# 3. Mixtral is probably the least synthetic-trained sounding
#    of all the OG models. Even when I tried the Quen shit
#    it seemed to be just openai. Mixtral is still sloppy.
#
# So, the pieces that are ours are uphill: non-instruct lora
# being applied to the instruct rawdog without an intermediate
# step.
#
# Obviously we're using pure elemental antisoc loras, hush's shit
# but not her merge because the merges aren't "uphill", as in, 
# a lora made with "mixtral non-instruct" applied straight to
# the instruct with loraize.
#
# The notion, which came to me in the middle of the night, is
# to have the hush loras be only barely present layer-wise but
# weighted heavily. Likewise with LimaRP, send uphill from 
# doctor-shotgun's qlora straight into mixtral-instruct
#
# My hypothesis is that we should get really fucking close to
# pure-ass mixtral-instruct in terms of attention, but that
# we're weighting really hard not to write like it. I have no
# idea if that's how it works--I'm a fucking caveman.
#
# What I'm given to understand, and I'm way out of my depth,
# is that the antisoc layers won't have blotched the instruct
# as badly as they usually do, but when they're triggered they
# are dominant. It's entirely possible I've got no idea what
# I'm saying.

# Model descriptions:
#  - crush: poetry; we have all checkpoints
#  - crest: fic; we only have e2 for this
#  - clash: novels (I think); we have all checkpoints for 0.2
models:
  # I wonder what happens if we just hurl this out the window
  #  - model: mistralai/Mixtral-8x7B-Instruct-v0.1
  #    parameters:
  #      density: 0.9
  #      weight: 0.55
  #
  # crest is fic
  - model: ./uphill-instruct-crest-e2-nolime
    # i found lima in this, I need to cook another
    parameters:
      density: 0.4
      weight: 0.3
  # This is actually an uphill lima but I didn't name it that way.
  - model: ./Mixtral-8x7B-Yes-Instruct-LimaRP
    parameters:
      # Still just a breath of layers from the thing
      density: 0.2
      # I am gimping its weight compared to hush tunes because limarp has too
      # much ai-slop and amateur-smut cliche slop. Honestly, if there were
      # something better than limarp I'd try to train it myself but I don't
      # know if there is.
      weight: 0.1
  # Pure uphill clash at e2. Also more weight.
  - model: ./uphill-pure-clash-0.2-e2
    parameters:
      density: 0.5
      weight: 0.6
# della sucked ass so dare_ties it is
merge_method: dare_ties
# I know all of these look like instruct but the lora
# is actually not so we go to the base base
base_model: mistralai/Mixtral-8x7B-v0.1
parameters:
  normalize: true
  int8_mask: true
dtype: bfloat16