Transformers
Safetensors
GGUF
mixtral
Mixture of Experts
validation
test-suite
japanese
scratch-trained
Instructions to use shibatch/tinymoeja2m with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use shibatch/tinymoeja2m with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("shibatch/tinymoeja2m", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use shibatch/tinymoeja2m with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf shibatch/tinymoeja2m:Q4_K_M # Run inference directly in the terminal: llama cli -hf shibatch/tinymoeja2m:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf shibatch/tinymoeja2m:Q4_K_M # Run inference directly in the terminal: llama cli -hf shibatch/tinymoeja2m:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf shibatch/tinymoeja2m:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf shibatch/tinymoeja2m:Q4_K_M
Use Docker
docker model run hf.co/shibatch/tinymoeja2m:Q4_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use shibatch/tinymoeja2m with Ollama:
ollama run hf.co/shibatch/tinymoeja2m:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use shibatch/tinymoeja2m with Docker Model Runner:
docker model run hf.co/shibatch/tinymoeja2m:Q4_K_M
- Lemonade
How to use shibatch/tinymoeja2m with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull shibatch/tinymoeja2m:Q4_K_M
Run and chat with the model
lemonade run user.tinymoeja2m-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -97,35 +97,49 @@ To process the GQA MoE execution graph and evaluate dynamic expert routing direc
|
|
| 97 |
|
| 98 |
```python
|
| 99 |
import torch
|
| 100 |
-
|
|
|
|
|
|
|
| 101 |
|
|
|
|
| 102 |
repo_id = "shibatch/tinymoeja2m"
|
| 103 |
|
| 104 |
-
print("
|
| 105 |
-
|
| 106 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 107 |
|
| 108 |
device = "cuda" if torch.cuda.is_available() else ("xpu" if torch.xpu.is_available() else "cpu")
|
| 109 |
model = model.to(device)
|
| 110 |
model.eval()
|
| 111 |
|
|
|
|
| 112 |
prompt = "トムとリリーは"
|
| 113 |
-
|
|
|
|
| 114 |
|
| 115 |
-
print("
|
| 116 |
with torch.no_grad():
|
| 117 |
-
|
| 118 |
-
|
| 119 |
-
max_length=64,
|
| 120 |
-
do_sample=False
|
|
|
|
|
|
|
|
|
|
| 121 |
)
|
| 122 |
-
|
| 123 |
-
|
|
|
|
| 124 |
|
| 125 |
print("\n--- Inference Test Result ---")
|
| 126 |
print("Prompt :", prompt)
|
| 127 |
print("Generated:", generated_text)
|
| 128 |
-
|
| 129 |
```
|
| 130 |
|
| 131 |
---
|
|
|
|
| 97 |
|
| 98 |
```python
|
| 99 |
import torch
|
| 100 |
+
import sentencepiece as spm
|
| 101 |
+
from transformers import MixtralForCausalLM
|
| 102 |
+
from huggingface_hub import hf_hub_download
|
| 103 |
|
| 104 |
+
# Define target repository identity
|
| 105 |
repo_id = "shibatch/tinymoeja2m"
|
| 106 |
|
| 107 |
+
print("Downloading and caching specialized tokenizer layer...")
|
| 108 |
+
# Fetch tokenizer.model file automatically from Hugging Face Hub
|
| 109 |
+
tokenizer_file = hf_hub_download(repo_id=repo_id, subfolder="hf", filename="tokenizer.model")
|
| 110 |
+
|
| 111 |
+
sp = spm.SentencePieceProcessor()
|
| 112 |
+
sp.Load(tokenizer_file)
|
| 113 |
+
|
| 114 |
+
print("Downloading and loading Mixtral-based 2M MoE model weights...")
|
| 115 |
+
model = MixtralForCausalLM.from_pretrained(repo_id, subfolder="hf")
|
| 116 |
|
| 117 |
device = "cuda" if torch.cuda.is_available() else ("xpu" if torch.xpu.is_available() else "cpu")
|
| 118 |
model = model.to(device)
|
| 119 |
model.eval()
|
| 120 |
|
| 121 |
+
# Prompt text utilizing vocabulary subsets
|
| 122 |
prompt = "トムとリリーは"
|
| 123 |
+
input_ids = [1] + sp.EncodeAsIds(prompt) # Explicitly prepend BOS (1)
|
| 124 |
+
input_tensor = torch.tensor([input_ids]).to(device)
|
| 125 |
|
| 126 |
+
print("Executing text generation loop (Validating 4:1 GQA & Top-2 MoE Kernels)...")
|
| 127 |
with torch.no_grad():
|
| 128 |
+
output_ids = model.generate(
|
| 129 |
+
input_tensor,
|
| 130 |
+
max_length=64,
|
| 131 |
+
do_sample=False,
|
| 132 |
+
pad_token_id=2,
|
| 133 |
+
bos_token_id=1,
|
| 134 |
+
eos_token_id=2
|
| 135 |
)
|
| 136 |
+
|
| 137 |
+
generated_ids = output_ids[0].cpu().tolist()
|
| 138 |
+
generated_text = sp.DecodeIds(generated_ids)
|
| 139 |
|
| 140 |
print("\n--- Inference Test Result ---")
|
| 141 |
print("Prompt :", prompt)
|
| 142 |
print("Generated:", generated_text)
|
|
|
|
| 143 |
```
|
| 144 |
|
| 145 |
---
|