shibatch commited on
Commit
ba5cd11
·
verified ·
1 Parent(s): 6ec8a17

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -13,11 +13,11 @@ tags:
13
 
14
  # TinyStories Mixtral 2M Top-2 MoE (tinymoe2m) GGUF & HF Validation Suite (4k Context)
15
 
16
- This repository provides an ultra-lightweight Mixtral model variant (a Mixture-of-Experts architecture utilizing the Llama 2 compute topology) scaled down to a **1.95M total parameter footprint** and a **1.14M active parameter execution frame**. It is trained on the TinyStories dataset and optimized as a precise validation asset.
17
 
18
- Following extensive long-context scaling evaluations, this asset has been calibrated to a **4,096 token context window (4k)** with an adjusted **RoPE base frequency (`rope_theta`) of 15,000.0** to prevent numerical saturation under FP32 precision boundaries while maintaining sharp localized attention coordinates.
19
 
20
- It is designed specifically for debugging custom inference engines, and native tensor compilers against MoE-specific runtime features. These include Gating network weight allocation, token distribution/gathering (Scatter/Gather loops), and the weighted addition combining multiple independent expert outputs.
21
 
22
  ---
23
 
 
13
 
14
  # TinyStories Mixtral 2M Top-2 MoE (tinymoe2m) GGUF & HF Validation Suite (4k Context)
15
 
16
+ This repository provides an ultra-lightweight Mixtral model variant (a Mixture-of-Experts architecture utilizing the Llama 2 compute topology) scaled down to a 1.95M total parameter footprint and a 1.14M active parameter execution frame. It is trained on the TinyStories dataset and optimized as a precise validation asset.
17
 
18
+ This asset is calibrated to a 4,096 token context window (4k) with an adjusted RoPE base frequency (`rope_theta`) of 15,000.0 to maintain sharp localized attention coordinates.
19
 
20
+ It is **designed specifically for debugging** custom inference engines, and native tensor compilers against MoE-specific runtime features. These include Gating network weight allocation, token distribution/gathering (Scatter/Gather loops), and the weighted addition combining multiple independent expert outputs.
21
 
22
  ---
23