Hugging Face's logo

bruhzair
/
prototype-0.4x302

Text Generation
Transformers
Safetensors
llama
mergekit
Merge
conversational
text-generation-inference
Model card Files
xet
 Community
bruhzair commited on
Commit
1d4228e
·
verified ·
1 Parent(s): 8bcb4ed

Upload folder using huggingface_hub

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. .gitattributes +1 -0
  2. README.md +47 -0
  3. chat_template.jinja +7 -0
  4. config.json +35 -0
  5. mergekit_config.yml +15 -0
  6. model-00001-of-00062.safetensors +3 -0
  7. model-00002-of-00062.safetensors +3 -0
  8. model-00003-of-00062.safetensors +3 -0
  9. model-00004-of-00062.safetensors +3 -0
  10. model-00005-of-00062.safetensors +3 -0
  11. model-00006-of-00062.safetensors +3 -0
  12. model-00007-of-00062.safetensors +3 -0
  13. model-00008-of-00062.safetensors +3 -0
  14. model-00009-of-00062.safetensors +3 -0
  15. model-00010-of-00062.safetensors +3 -0
  16. model-00011-of-00062.safetensors +3 -0
  17. model-00012-of-00062.safetensors +3 -0
  18. model-00013-of-00062.safetensors +3 -0
  19. model-00014-of-00062.safetensors +3 -0
  20. model-00015-of-00062.safetensors +3 -0
  21. model-00016-of-00062.safetensors +3 -0
  22. model-00017-of-00062.safetensors +3 -0
  23. model-00018-of-00062.safetensors +3 -0
  24. model-00019-of-00062.safetensors +3 -0
  25. model-00020-of-00062.safetensors +3 -0
  26. model-00021-of-00062.safetensors +3 -0
  27. model-00022-of-00062.safetensors +3 -0
  28. model-00023-of-00062.safetensors +3 -0
  29. model-00024-of-00062.safetensors +3 -0
  30. model-00025-of-00062.safetensors +3 -0
  31. model-00026-of-00062.safetensors +3 -0
  32. model-00027-of-00062.safetensors +3 -0
  33. model-00028-of-00062.safetensors +3 -0
  34. model-00029-of-00062.safetensors +3 -0
  35. model-00030-of-00062.safetensors +3 -0
  36. model-00031-of-00062.safetensors +3 -0
  37. model-00032-of-00062.safetensors +3 -0
  38. model-00033-of-00062.safetensors +3 -0
  39. model-00034-of-00062.safetensors +3 -0
  40. model-00035-of-00062.safetensors +3 -0
  41. model-00036-of-00062.safetensors +3 -0
  42. model-00037-of-00062.safetensors +3 -0
  43. model-00038-of-00062.safetensors +3 -0
  44. model-00039-of-00062.safetensors +3 -0
  45. model-00040-of-00062.safetensors +3 -0
  46. model-00041-of-00062.safetensors +3 -0
  47. model-00042-of-00062.safetensors +3 -0
  48. model-00043-of-00062.safetensors +3 -0
  49. model-00044-of-00062.safetensors +3 -0
  50. model-00045-of-00062.safetensors +3 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,47 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: []
3
+ library_name: transformers
4
+ tags:
5
+ - mergekit
6
+ - merge
7
+
8
+ ---
9
+ # prototype-0.4x302
10
+
11
+ This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
12
+
13
+ ## Merge Details
14
+ ### Merge Method
15
+
16
+ This model was merged using the [SCE](https://arxiv.org/abs/2408.07990) merge method using /workspace/prototype-0.4x295 as a base.
17
+
18
+ ### Models Merged
19
+
20
+ The following models were included in the merge:
21
+ * /workspace/cache/models--tdrussell--Llama-3-70B-Instruct-Storywriter/snapshots/19be2a7c6382a9150e126cf144e2b2964e700d3c
22
+ * /workspace/cache/models--Doctor-Shotgun--L3.3-70B-Magnum-Diamond/snapshots/a7dfb66b4469a4c9ca07ff28bccc73a44797e76c
23
+ * /workspace/cache/models--nbeerbower--Llama3.1-Gutenberg-Doppel-70B/snapshots/f083f3a89b8275e7e5329bb0668ada189f80b507
24
+ * /workspace/cache/models--EVA-UNIT-01--EVA-LLaMA-3.33-70B-v0.0/snapshots/501b61987a44172bf2d68365893fcd13081ad4db
25
+ * /workspace/cache/models--TheDrummer--Fallen-Llama-3.3-70B-v1/snapshots/d46ef2629f1c3cd46789a55793c5ff0af60de3e8
26
+
27
+ ### Configuration
28
+
29
+ The following YAML configuration was used to produce this model:
30
+
31
+ ```yaml
32
+ models:
33
+ - model: /workspace/cache/models--Doctor-Shotgun--L3.3-70B-Magnum-Diamond/snapshots/a7dfb66b4469a4c9ca07ff28bccc73a44797e76c
34
+ - model: /workspace/cache/models--nbeerbower--Llama3.1-Gutenberg-Doppel-70B/snapshots/f083f3a89b8275e7e5329bb0668ada189f80b507
35
+ - model: /workspace/cache/models--tdrussell--Llama-3-70B-Instruct-Storywriter/snapshots/19be2a7c6382a9150e126cf144e2b2964e700d3c
36
+ - model: /workspace/cache/models--EVA-UNIT-01--EVA-LLaMA-3.33-70B-v0.0/snapshots/501b61987a44172bf2d68365893fcd13081ad4db
37
+ - model: /workspace/cache/models--TheDrummer--Fallen-Llama-3.3-70B-v1/snapshots/d46ef2629f1c3cd46789a55793c5ff0af60de3e8
38
+ base_model: /workspace/prototype-0.4x295
39
+ select_topk: 0.17
40
+ merge_method: sce
41
+ tokenizer:
42
+ source: base
43
+ chat_template: llama3
44
+ pad_to_multiple_of: 8
45
+ int8_mask: true
46
+ dtype: float32
47
+ ```
chat_template.jinja ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {% set loop_messages = messages %}
2
+ {% for message in loop_messages %}
3
+ {% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' %}
4
+ {% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}
5
+ {{ content }}
6
+ {% endfor %}
7
+ {% if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}{% endif %}
config.json ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "LlamaForCausalLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 128000,
8
+ "eos_token_id": 128001,
9
+ "head_dim": 128,
10
+ "hidden_act": "silu",
11
+ "hidden_size": 8192,
12
+ "initializer_range": 0.02,
13
+ "intermediate_size": 28672,
14
+ "max_position_embeddings": 131072,
15
+ "mlp_bias": false,
16
+ "model_type": "llama",
17
+ "num_attention_heads": 64,
18
+ "num_hidden_layers": 80,
19
+ "num_key_value_heads": 8,
20
+ "pretraining_tp": 1,
21
+ "rms_norm_eps": 1e-05,
22
+ "rope_scaling": {
23
+ "factor": 8.0,
24
+ "high_freq_factor": 4.0,
25
+ "low_freq_factor": 1.0,
26
+ "original_max_position_embeddings": 8192,
27
+ "rope_type": "llama3"
28
+ },
29
+ "rope_theta": 500000.0,
30
+ "tie_word_embeddings": false,
31
+ "torch_dtype": "float32",
32
+ "transformers_version": "4.55.0",
33
+ "use_cache": true,
34
+ "vocab_size": 128256
35
+ }
mergekit_config.yml ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ models:
2
+ - model: /workspace/cache/models--Doctor-Shotgun--L3.3-70B-Magnum-Diamond/snapshots/a7dfb66b4469a4c9ca07ff28bccc73a44797e76c
3
+ - model: /workspace/cache/models--nbeerbower--Llama3.1-Gutenberg-Doppel-70B/snapshots/f083f3a89b8275e7e5329bb0668ada189f80b507
4
+ - model: /workspace/cache/models--tdrussell--Llama-3-70B-Instruct-Storywriter/snapshots/19be2a7c6382a9150e126cf144e2b2964e700d3c
5
+ - model: /workspace/cache/models--EVA-UNIT-01--EVA-LLaMA-3.33-70B-v0.0/snapshots/501b61987a44172bf2d68365893fcd13081ad4db
6
+ - model: /workspace/cache/models--TheDrummer--Fallen-Llama-3.3-70B-v1/snapshots/d46ef2629f1c3cd46789a55793c5ff0af60de3e8
7
+ base_model: /workspace/prototype-0.4x295
8
+ select_topk: 0.17
9
+ merge_method: sce
10
+ tokenizer:
11
+ source: base
12
+ chat_template: llama3
13
+ pad_to_multiple_of: 8
14
+ int8_mask: true
15
+ dtype: float32
model-00001-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:67ad7c8c8f67c10a7e9d2b955392b52d4494c441640f25a855d3691e340abd57
3
+ size 4093740184
model-00002-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4ec86c68cdda2082cf59ce39d52a2acb3c03396045216147223d1586d0dc868b
3
+ size 4471128312
model-00003-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:513935f0c1ac356b94d6a616605b0d82d791f77f4fbb4ad4a02d7280ed1f733c
3
+ size 3154150216
model-00004-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:abb35f2edc009963ed0d2582ad0678267cddb4380ae4c5d7a32cc658994fe359
3
+ size 4236247296
model-00005-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:249de4d552da3279d817fda3a4ce7832ed7c1b19f896128d487dffc939855507
3
+ size 4664198768
model-00006-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9af585cfd340b3cf0d1058a5c4d9c7ff59ab2796af0e2daddbdf6fdef8e90c79
3
+ size 4899014032
model-00007-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6ac93eae5947a537901c5ba01c2545386a452856d07956479aeaa4af010d4a10
3
+ size 4362208640
model-00008-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:496a43d4b524a32af379a7925a1d5427d8dc43bf338fc19413b47c38aee38e69
3
+ size 4630578456
model-00009-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ba31b73aa0063b35a6c23e47d4a80d09650aab2fc076f199407c20435f71d13d
3
+ size 4630644224
model-00010-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:87dbb805d9363bae0e363ac6ad86d869031530162b9bb77d6f9e65569c34521c
3
+ size 4630578456
model-00011-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:83bcd53ab8bd4e3fb9537370b758001f9f9b0203e607e87700fe4949a7ad9a86
3
+ size 4362142880
model-00012-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:58c264488fe15eabef00357a5cd1df08dbd3d42f886006a465f32cc745196701
3
+ size 4697753312
model-00013-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:08c5a36f5da9a18f1379698e69af10c8f0421c4f69e2b622d96c8d4189630ce3
3
+ size 4899014024
model-00014-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d39d038227be9ce876e21a5338717227d72d9e9f2e0dc8997b2076135e570feb
3
+ size 4362208648
model-00015-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f13e1c6126104bb071a2fd7190c6977bdf9e295cf305bd12e0cf696ab21f5a4f
3
+ size 4630578456
model-00016-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:57812a757d8374c76061ef5152aab3ed004a3bc2bd0c227327c99814a0604224
3
+ size 4630644224
model-00017-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2a10999323aaf29a0f98f31a78c1983f4199f89c40b59eaa47988e65dcb2e199
3
+ size 4630578456
model-00018-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2f5b7513c16eb8a9f051e0f8f0b95bd0c0879334bb8f5af297fee4048aabb083
3
+ size 4630644224
model-00019-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9826b73dad45527fffe60321d3320c5a60b53d28723dbff130fbceeadbee4025
3
+ size 4630578456
model-00020-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:48f75ef77881dd6381afb40270e781e1a54f4fd412c3f8c02ec37e4e27619c2b
3
+ size 4630611336
model-00021-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:71c7aa28d73ce95be4ac5491de19a633f60ff72e8f9436ea8cdcf5d9b143fb40
3
+ size 4362175768
model-00022-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e885119eb0c0527b71f8f844079d14046ae7c3ea97edd3abd6f6fb889efcc3a0
3
+ size 4966156000
model-00023-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1b9bc5dfcb4ebd01055ecbe0e930364b1bd84dc46678425c7ad993709dadcacb
3
+ size 4362142880
model-00024-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:18dd5adf5681d478dfd76951a21e0c0f479c469884fc0b9a174da986aab58a87
3
+ size 4362142880
model-00025-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fad7b2bd529c0c72e0365aa220cd6a22449651f37b192803eedc39a7a755fdcc
3
+ size 4966188896
model-00026-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5d890593d0a829bdc26840b9ee97e8a4097b4e400a8e444c33f1b35914e18b24
3
+ size 4362142880
model-00027-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cfa17592c554c88f082dd9217ad0a5f48db4c1242344712ed359c8f10242203e
3
+ size 4630644216
model-00028-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b29e25f1f3168d654b328e63309ee3b57b42af13cf0c227cf35f56db6eb6a6e5
3
+ size 4630578456
model-00029-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9590d2bceb8da12565b33a763004409aa729a5e33f30e5ab9f6450f197ed3d69
3
+ size 4630611344
model-00030-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b2884bc656189ed1f5aabe6c272686fc0bb26ae392ff05591050f080be69003e
3
+ size 4664165888
model-00031-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d263860f0dbf329b01d406284135c3a5f376b3791271389eb918908db39fdabc
3
+ size 4362142880
model-00032-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a4ccef801f52b69cd130bfda3fe228b3f1b4f9b85d2f3f174392a328483c9339
3
+ size 4664198768
model-00033-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8e9c7eea544e7b87252f981d8c792a16fd035f2a8e43915a13824b2f6749e892
3
+ size 4362142880
model-00034-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ab79f3f2161a2a0bb5b52ca10d909b598d83acee981284b5404cd85d34324fc8
3
+ size 4932601456
model-00035-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e67752eba180bc8dd656b61b082ffdc0a88a3ffe31471f2f7c1d8422b849207b
3
+ size 4395730312
model-00036-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:63892946638dd7099538b92562820bf3ced672e2fada6fa0523853e1f1f827d7
3
+ size 4362142880
model-00037-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:19cbbac0eb4fde046a8a4179b08d2626cd92badabdb394566df76b97f57abe25
3
+ size 4932601456
model-00038-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5b64057df8b8b426a71e3ff7ef61fd213b221146abe347390bd648dfacee80a5
3
+ size 4362142880
model-00039-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6f2d9e6bef74bc6d339e11b1b50b8e0dd382476bde33e67f3c37817a775d335b
3
+ size 4664165888
model-00040-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e06f71cb30f52827066807834da1183b3261c00ea89bdd7c2aaa68dfa9c5f8c9
3
+ size 4664165880
model-00041-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7e12ff4d374ecc6d7ae5129aae78d35518c46a281d7e93069cc018be6e3b14da
3
+ size 4362142880
model-00042-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c79911645375ccf28e2e188e40d5df2d3836cc56c6d66fabda86a9751f28e6a7
3
+ size 4932634344
model-00043-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2c973ac8185ae1332cf25d9f25231de83c3b7a57339863f534371bd849bdb766
3
+ size 4630578448
model-00044-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:67c2f198408f0a805a6bb781a2337df9b09c26fa553117eb3ff7f279e3786e78
3
+ size 4362175760
model-00045-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a1d5da1df0b0e11a5217faf6236575eb9211380a18987d833a94b731572f7ba5
3
+ size 4362175768