How to use from
llama.cppInstall from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf Abiray/MicroLlama-134M-Instruct:F16# Run inference directly in the terminal:
llama-cli -hf Abiray/MicroLlama-134M-Instruct:F16Use pre-built binary
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Abiray/MicroLlama-134M-Instruct:F16# Run inference directly in the terminal:
./llama-cli -hf Abiray/MicroLlama-134M-Instruct:F16Build from source code
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Abiray/MicroLlama-134M-Instruct:F16# Run inference directly in the terminal:
./build/bin/llama-cli -hf Abiray/MicroLlama-134M-Instruct:F16Use Docker
docker model run hf.co/Abiray/MicroLlama-134M-Instruct:F16Quick Links
🦙 MicroLlama-134M-Instruct
MicroLlama-134M-Instruct is a custom-trained Small Language Model (SLM) created by Abhiray (Jay). Built using a scaled-down Llama architecture, this model is designed to be highly efficient, lightweight, and can try to conversational instruction-following.
🧠 Model Details
- Parameters: 134 Million
- Architecture: Custom Llama (12 Layers, 12 Heads, 768 Embedding Dimension, 2048 Intermediate Size)
- Context Length: 1024 Tokens
- Precision:
bfloat16 - Tokenizer: TinyLlama-1.1B-Chat-v1.0 (32,000 Vocab)
🏗️ Training Pipeline
This model was trained entirely from scratch in three distinct phases:
- Phase 1 & 2: Continuous Pre-Training (CPT): Trained on ~9 Billion tokens of high-quality web, educational, and narrative data.
- Phase 3: Supervised Fine-Tuning (SFT): Fine-tuned for 250 high-precision steps using
-100target masking.
💻 How to Use
The model strictly follows the ChatML-style template used during its SFT phase. For optimal performance, a generation temperature between 0.3 and 0.5 with a gentle repetition_penalty (e.g., 1.05) is recommended.
Prompt Format:
<|system|>
You are a highly capable, friendly, and helpful AI assistant.</s>
<|user|>
What is the core temperature of the Sun?</s>
<|assistant|>
- Downloads last month
- 9
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Install from brew
# Start a local OpenAI-compatible server with a web UI: llama-server -hf Abiray/MicroLlama-134M-Instruct:F16# Run inference directly in the terminal: llama-cli -hf Abiray/MicroLlama-134M-Instruct:F16