Upload AGENT.md

AGENT.md

@@ -0,0 +1,228 @@
+Below is some implementation pointers, plus pseudocode directions for how to  organize the ui for tabs 5-7, and which options to add. These should be processed by a mergekit config launcher and launch on a gradio spaces app via a CPU machine. It is up to you to correctly route them to the mergekit cli, per provided examples (such as https://huggingface.co/spaces/AlekseyCalvin/mergekit-soonr/blob/main/app.py) and to make it all work. For tab 8, the processing should be routed thru the mergekit-moe cli, rather than the mergekit-yaml cli. For 5-8, use mergekit yaml. Make sure to implement memory efficient model sharding, caching, processing, and output saving similar to tabs 1-4. MAKE SURE EVERYTHING CORRECTLY CORRESPONDS TO MERGEKIT CONDIGS!
+
+
+
+As you may easily gather from https://github.com/arcee-ai/mergekit/blob/main/docs/merge_methods.md , the "Task Vector" family begins with the foundational merge_method: "task_arithmetic" method, extends it via the TrIm, Elect Sign & Merge (aka TIES Merging) method aka merge_method: "ties”, further extends it with the actual sparsifying methods – merge_method: “dare_ties” and merge_method: “dare_linear” – then further extends these with the Drop and rEscaLe via sampLing with mAgnitude (DELLA-Merging) methods w/ pruning, aka merge_method: “della” and merge_method: “della_linear”, along with with the Breadcrumbs methods (which sculpturally sparsify merges by pruning outliers to reinforce task vectors), aka merge_method: “breadcrumbs” and aka merge_method: “breadcrumbs_ties”, and finally the SCE method from FuseChat, aka aka merge_method: “sce” (with SCE constituting a form of adaptive matrix/tensor-level merging: running each group of merged tensors through a tri-partite cross-optimization process of variance-based masking, followed by task-vector weighing-factor calculation, and then sign-consensus alignment). 
+
+Besides the Interpolation and the Task Vector families, you more or less neglect the last category of MergeKit’s methods: the Specialized methods (model_stock, nearswap, arcee_fusion, passthrough, and technically also linear) not belonging to either the Spherical Interpolation family (slerp, nuslerp, multislerp, karcher), nor to the Task Vector family (task_arithmetic, ties, dare_ties, dare_linear, della, della_linear, breadcrumbs, breadcrumbs_ties, and sce).
+
+Just as glaringly, you neglect to sufficiently consider the necessary range of actual key parameters one needs to implement each method. You make a crude attempt to partition parameters in a per-family way, but you grossly misjudge the range of actual parameters required to support even the few methods you do list.
+
+For Clarity, I have outlined all of the parameters for the merge method families and methods from https://github.com/arcee-ai/mergekit/blob/main/docs/merge_methods.md , as well as how they should be represented within the included tabs, starting with tab 5. The regular non-MoE MergeKit tabs should be named “Amphinterpolative” (for tab 5), “Stir/Tie Bases” (for tab 6)  “Specious” (for tab 7), “MoEr” (for tab 8, the MoE mergekit implementation, wherein you must also reimplement prompts), “Rawer” (for tab 9, the raw PyTorch implementation), and “Mario,DARE!” (For tab 10, with the custom DARE implementation from https://huggingface.co/spaces/AlekseyCalvin/DARE-MERGE-SOONR/blob/main/app.py, https://huggingface.co/spaces/AlekseyCalvin/DARE-MERGE-SOONR/blob/main/merge.py , and https://huggingface.co/spaces/AlekseyCalvin/DARE-MERGE-SOONR/blob/main/hf_merge.py, + https://github.com/martyn/safetensors-merge-supermario #refine this implementation also! Reference all these sources)
+
+Here are the precise parameters and interface directions for tabs 5 thru 7. Implement the below in accordance to the exact correct logic from https://github.com/arcee-ai/mergekit/tree/main/mergekit . Everything has to be formatted to work with the library! Do not just guess or make things up. Align everything precisely and double-check every bit.
+
+1)Spherical Interpolation-type method family methods family (or per my new category syntax “Amphinterpolative”):
+
+“merge_method":
+a)“slerp” – takes 2 models, one must be specified as “base_model".
+
+b)”nuslerp” = Requires exactly 2 models. A “base_model" can optionally be provided (it must be distinct from the two main models).
+
+c)”multislerp” = Takes 2 or more models. A “base_model" can optionally be provided to operate in task vector space.
+
+d)”karcher” = Takes 2 or more models. No “base_model” is used.
+
+
+## Amphinterpolative (Spherical Interpolation-type method family) Parameters Interface: 
+
+Amphinterpolative Tab GLOBAL PARAMETERS:
+“base_model” (form designating the repo of base model, mandatory for “slerp” method, optional for “nuslerp”, “multislerp”, “karcher")
+
+“t” (global, float or scalar between 0 and 1.0, this is the Interpolation factor between base and t=0 yields the base_model, t=1 yields the other model)
+
+"tokenizer_source" ["base", "union", "model:path"] default should be "base"
+
+“normalize_weights” (boolean true/false checkmark switch, default is “true”/checked)
+
+“int8_mask” (global, boolean true/false checkmark switch, default is “false”/unchecked)
+
+“nuslerp_flatten” (global, boolean true/false checkmark switch, designated for the “nuslerp” method, default is “false”/unchecked, if “true” performs row/column-wise interpolation)
+
+“nuslerp_row_wise” (global, boolean true/false checkmark switch, designated for the “nuslerp” method, default is “false”/unchecked, if “true” and “nuslerp_flatten” is "false”, SLERPs row vectors instead of column vectors)
+
+“eps” (global, designated for the “multislerp” method, a small numerical constant, default is "1e-8”)
+
+“max_iter” (global, designated for the “karcher” method, an integer value, default is “10”, refers to maximum iterations for the Karcher mean algorithm)
+
+“tol” (global, designated for the “karcher” method ,a small numerical constant, default is “1e-5”, refers to convergence tolerance apropos the Karcher mean)
+
+Also add the same sharding logic from tabs 1-4.
+As well as the float/precision selection per the mergekit-yaml config.
+Also add option for chat templates from the mergekit config.
+
+
+Amphinterpolative Tab Per-Model Parameters:
+MODEL 1:
+"model 1” (form input for repo name of model #1)
+
+"weight (model 1)” (0 to 1.0 float, or a list of values in gradient progression/regression like "[0, 0.3, 0.7, 1]" or "[1, 0.7, 0.1]”, default is 1.0)
+
+MODEL 2:
+"model 2” (form input for repo name of model #2)
+
+"weight (model 2)” (0 to 1.0 float, or a list of values in gradient progression/regression like "[0, 0.3, 0.7, 1]" or "[1, 0.7, 0.1]”, default is 1.0)
+
+MODEL 3 (only for ”multislerp” & “karcher”, which accept more than 2 models):
+"model 3” (form input for repo name of model #3)
+
+"weight (model 3)” (0 to 1.0 float, or a list of values in gradient progression/regression like "[0, 0.3, 0.7, 1]" or "[1, 0.7, 0.1]”, default is 1.0)
+
+MODEL 4 (only for ”multislerp” & “karcher”, which accept more than 2 models):
+"model 4” (form input for repo name of model #4)
+
+"weight (model 4)” (0 to 1.0 float, or a list of values in gradient progression/regression like "[0, 0.3, 0.7, 1]" or "[1, 0.7, 0.1]”, default is 1.0)
+
+MODEL 5 (only for ”multislerp” & “karcher”, which accept more than 2 models):
+"model 5” (form input for repo name of model #5)
+
+"weight (model 4)” (0 to 1.0 float, or a list of values in gradient progression/regression like "[0, 0.3, 0.7, 1]" or "[1, 0.7, 0.1]”, default is 1.0)
+
+
+2)Task Vector-type methods family (or per my new category syntax “Stir/Tie Bases”), further detailed in https://github.com/arcee-ai/mergekit/blob/main/mergekit/merge_methods/generalized_task_arithmetic.py and in https://github.com/arcee-ai/mergekit/blob/main/docs/merge_methods.md, example in https://github.com/arcee-ai/mergekit/blob/main/examples/ties.yml 
+
+“merge_method":
+a)”task_arithmetic” =  Requires a “base_model” plus one or more other models.
+
+b)”ties” =  Requires 2 or more models. One of these must be specified as “base_model”. 
+
+c)”dare_ties” =  Requires 2 or more models. One of these must be specified as “base_model”. 
+
+d)”dare_linear” =  Requires 2 or more models. One of these must be specified as “base_model”. 
+
+e)”della” =  Requires 2 or more models. One of these must be specified as “base_model”. 
+
+f)”della_linear” =  Requires 2 or more models. One of these must be specified as “base_model”. 
+
+g)”breadcrumbs” =  Requires 2 or more models. One of these must be specified as “base_model”. 
+
+h)”breadcrumbs_ties” = Parameters: "weight" (per-model, float from 0 to 1.0, may be a single 0-1.0 float value like “0.76” or a list of values in gradient progression/regression like "[0, 0.3, 0.7, 1]" or "[1, 0.7, 0.1]"), "gamma" (per-model) – fraction of parameters to prune, "density" (per-model, float from 0 to 1.0, may be a single value or a list of values in gradient progression/regression like [0, 0.3, 0.7, 1] or [1, 0.7, 0.1]) , "lambda"
+
+i)”sce” =  Requires at least 2 models. One of these must be specified as “base_model”. Parameters: “select_topk” (Float from 0 to 1) – default is “1.0”, effectively neutralizing the parameter. If select_topk < 1.0, only the top select_topk fraction of parameter positions with the highest variance are kept active. 
+
+## Task Vector Parameters Interface: 
+Stir/Tie Bases Tab GLOBAL PARAMETERS:
+“base_model” (form designating the base model)
+
+“normalize” (boolean true/false checkmark switch, default is “true")
+
+“int8_mask” (global, boolean true/false checkmark switch, default is “false”)
+
+“lambda” (global, float value, default is "1.0")
+
+“rescale" (global, boolean true/false checkmark switch, designated for “dare_linear”, default is “true” )
+
+“select_topk” (float value from 0 to 1.0, default is “1.0”,  designated for “sce”)
+
+Also add the same sharding logic from tabs 1-4.
+As well as the float/precision selection per the mergekit-yaml config.
+Also add option for chat templates from the mergekit config.
+
+
+Task Vector Per-Model Parameters:
+MODEL 1:
+"model 1” (form input for repo name of model #1)
+
+"weight (model 1)” (0 to 1.0 float, or a list of values in gradient progression/regression like "[0, 0.3, 0.7, 1]" or "[1, 0.7, 0.1]”, default is 1.0)
+
+"density (model 1)” (0 to 1.0 float, or a list of values in gradient progression/regression like "[0, 0.3, 0.7, 1]" or "[1, 0.7, 0.1]”, default is 1.0)
+
+"gamma (model 1)" (for “breadcrumbs” and “breadcrumbs_ties”, 0 to 1.0 float, default value is “0.01")
+
+"epsilon (model 1)” (for “della” and “della_linear”,  0 to 1.0 float, default value is “0.15”)
+
+MODEL 2:
+"model 2” (form input for repo name of model #2)
+
+"weight (model 2)” (0 to 1.0 float, or a list of values in gradient progression/regression like "[0, 0.3, 0.7, 1]" or "[1, 0.7, 0.1]”, default is 1.0)
+
+"density (model 2)” (0 to 1.0 float, or a list of values in gradient progression/regression like "[0, 0.3, 0.7, 1]" or "[1, 0.7, 0.1]”, default is 1.0)
+
+"gamma (model 2)" (for “breadcrumbs” and “breadcrumbs_ties”, 0 to 1.0 float, default value is “0.01")
+
+"epsilon (model 2)” (for “della” and “della_linear”,  0 to 1.0 float, default value is “0.15”)
+
+MODEL 3:
+"model 3” (form input for repo name of model #3)
+
+"weight (model 3)” (0 to 1.0 float, or a list of values in gradient progression/regression like "[0, 0.3, 0.7, 1]" or "[1, 0.7, 0.1]”, default is 1.0)
+
+"density (model 3)” (0 to 1.0 float, or a list of values in gradient progression/regression like "[0, 0.3, 0.7, 1]" or "[1, 0.7, 0.1]”, default is 1.0)
+
+"gamma (model 3)" (for “breadcrumbs” and “breadcrumbs_ties”, 0 to 1.0 float, default value is “0.01”) #(fraction of parameters to prune)
+
+"epsilon (model 3)” (for “della” and “della_linear”,  0 to 1.0 float, default value is “0.15”)
+
+MODEL 4:
+"model 4” (form input for repo name of model #4)
+
+"weight (model 4)” (0 to 1.0 float, or a list of values in gradient progression/regression like "[0, 0.3, 0.7, 1]" or "[1, 0.7, 0.1]”, default is 1.0)
+
+"density (model 4)” (0 to 1.0 float, or a list of values in gradient progression/regression like "[0, 0.3, 0.7, 1]" or "[1, 0.7, 0.1]”, default is 1.0)
+
+"gamma (model 4)" (for “breadcrumbs” and “breadcrumbs_ties”, 0 to 1.0 float, default value is “0.01”) #(fraction of parameters to prune)
+
+"epsilon (model 4)” (for “della” and “della_linear”,  0 to 1.0 float, default value is “0.15”)
+
+3)Specialized methods family (add new category called “Specious” to app/ui):
+
+“merge_method”:
+a)”model_stock” = Requires at least 3 models. One of these must be specified as “base_model”. Parameters: “filter_wise” (true/false boolean) – default should be “false" 
+
+b)”nearswap” =  Requires exactly 2 models. One of the two must be specified as “base_model”. Parameters: “t”, inputted similarly to “slerp”, but processed via distinct algorithm.
+
+c)”arcee_fusion” = Requires exactly 2 models. One model must be specified as “base_model”. All other parameter inputs should be ignored.
+
+d)”passthrough” = Takes exactly 1 model as input (model inputs beyond model A should be ignored). Parameters: Takes “filter” sub-argument as input to select and designate model component to extract at a certain “scale” value. e.g., {"filter": "down_proj", "value": 0.5}
+
+Specious Tab GLOBAL PARAMETERS:
+“base_model” (form designating the base model)
+
+“normalize” (boolean true/false checkmark switch, default is “true")
+
+“int8_mask” (global, boolean true/false checkmark switch, default is “false”)
+
+“t” (global, float or scalar between 0 and 1.0, this is the Interpolation factor between base and 
+
+t=0 yields the base_model, t=1 yields the other model)
+
+“filter_wise"  (global, boolean true/false checkmark switch, designated for the “model_stock” method, default is “false”/unchecked)
+
+Also add the same sharding logic from tabs 1-4.
+As well as the float/precision selection per the mergekit-yaml config.
+Also add option for chat templates from the mergekit config.
+
+Specious Tab Per-Model Parameters:
+MODEL 1:
+"model 1” (form input for repo name of model #1)
+
+"weight (model 1)” (0 to 1.0 float, or a list of values in gradient progression/regression like "[0, 0.3, 0.7, 1]" or "[1, 0.7, 0.1]”, default is 1.0)
+
+“filter” (model1) (optional form input to specify component of model 1 to isolate, crucial for “passthrough”)
+
+MODEL 2:
+"model 2” (form input for repo name of model #2)
+
+"weight (model 2)” (0 to 1.0 float, or a list of values in gradient progression/regression like "[0, 0.3, 0.7, 1]" or "[1, 0.7, 0.1]”, default is 1.0)
+
+
+MODEL 3 (only for ”model_stock”, which accepts more than 2 models):
+
+"model 3” (form input for repo name of model #3)
+
+"weight (model 3)” (0 to 1.0 float, or a list of values in gradient progression/regression like "[0, 0.3, 0.7, 1]" or "[1, 0.7, 0.1]”, default is 1.0)
+
+
+MODEL 4 (only for ”model_stock”, which accepts more than 2 models):
+
+"model 4” (form input for repo name of model #4)
+
+"weight (model 4)” (0 to 1.0 float, or a list of values in gradient progression/regression like "[0, 0.3, 0.7, 1]" or "[1, 0.7, 0.1]”, default is 1.0)
+
+
+MODEL 5 (only for ”model_stock”, which accepts more than 2 models):
+
+"model 5” (form input for repo name of model #5)
+
+"weight (model 4)” (0 to 1.0 float, or a list of values in gradient progression/regression like "[0, 0.3, 0.7, 1]" or "[1, 0.7, 0.1]”, default is 1.0)
+