Instructions to use inference-optimization/Llama-3.2-0.5B-Instruct-small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use inference-optimization/Llama-3.2-0.5B-Instruct-small with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="inference-optimization/Llama-3.2-0.5B-Instruct-small")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Llama-3.2-0.5B-Instruct-small")
model = AutoModelForCausalLM.from_pretrained("inference-optimization/Llama-3.2-0.5B-Instruct-small", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use inference-optimization/Llama-3.2-0.5B-Instruct-small with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "inference-optimization/Llama-3.2-0.5B-Instruct-small"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Llama-3.2-0.5B-Instruct-small",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/inference-optimization/Llama-3.2-0.5B-Instruct-small

SGLang

How to use inference-optimization/Llama-3.2-0.5B-Instruct-small with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "inference-optimization/Llama-3.2-0.5B-Instruct-small" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Llama-3.2-0.5B-Instruct-small",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "inference-optimization/Llama-3.2-0.5B-Instruct-small" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Llama-3.2-0.5B-Instruct-small",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use inference-optimization/Llama-3.2-0.5B-Instruct-small with Docker Model Runner:
```
docker model run hf.co/inference-optimization/Llama-3.2-0.5B-Instruct-small
```

Llama-3.2-0.5B-Instruct-small

This is a small version of meta-llama/Llama-3.2-1B-Instruct created for testing and development. It preserves the full architecture characteristics of the base model (GQA attention, SwiGLU MLPs, llama3 rope scaling, tied embeddings, identical hyperparameters) with fewer layers.

Model Details

Base Model: meta-llama/Llama-3.2-1B-Instruct
Architecture: LlamaForCausalLM (model_type llama)
Total Parameters: 0.51B (505,956,352)
Activated Parameters: 0.51B (dense model)

Configuration Changes

The following parameters were reduced from the original model:

Parameter	Base	Small
`num_hidden_layers`	16	4

All other hyperparameters are identical to the base: hidden_size 2048, intermediate_size 8192, num_attention_heads 32, num_key_value_heads 8, head_dim 64, vocab_size 128256, tie_word_embeddings true, rope/llama3 scaling parameters, max_position_embeddings 131072, etc. This is a dense model with a single layer type, so no layer-type or expert changes apply.

Checkpoint Structure

Single model.safetensors file with 38 bf16 tensors (the base is also a single file with 146 tensors). Tensor naming and per-layer tensor shapes match the base checkpoint exactly modulo layer count (verified against the base checkpoint's safetensors header; layer-0/embedding/norm shapes identical). As in the base, lm_head.weight is absent from the checkpoint because embeddings are tied.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("inference-optimization/Llama-3.2-0.5B-Instruct-small", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Llama-3.2-0.5B-Instruct-small")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model skill.

Built from the base AutoConfig with only the allowed change above (layer count), initialized with init_weights() (bf16), and fine-tuned on a toy copypasta dataset until perplexity ~1.0 (target 3.0).

Validation

$ python validate_tiny_model.py Llama-3.2-0.5B-Instruct-small-ft
Success: 1.0484492778778076 <= 10.0

Generating sample text:
According to all known laws of aviation, there is no way a bee should be able to fly

Notes

Dense model: no MoE experts, no per-layer type lists; the only allowed change needed to shrink it is the layer count.
tie_word_embeddings kept true (as in base); the checkpoint therefore contains no separate lm_head tensor, matching the base layout.
The base model repository is gated; this small version is published under the inference-optimization org for testing convenience.

Downloads last month: 48

Safetensors

Model size

0.5B params

Tensor type

BF16

Model tree for inference-optimization/Llama-3.2-0.5B-Instruct-small

Base model

meta-llama/Llama-3.2-1B-Instruct

Finetuned

(1845)

this model

Collection including inference-optimization/Llama-3.2-0.5B-Instruct-small

Small Models

Collection

Small versions of known models: same vLLM loading characteristics as the base, fewer layers/experts, identical hidden dims and hyperparameters • 25 items • Updated 5 days ago