README.md

---
title: README
emoji: 😻
colorFrom: blue
colorTo: gray
sdk: static
pinned: false
---

**Local LLMs at the metal.**

Models for [Mila](https://github.com/ToddThomson/Mila) — a C++23 and CUDA runtime for open LLMs,
built from explicit components you can read and understand.

Gemma 4 and Llama 3.x, quantized to FP4 so they run on the consumer hardware you already own.

## Compatibility

These models are built for Mila and load with Mila. The quantization is Mila's own, matched to its
CUDA kernels rather than to a portable interchange format.

The files are ordinary safetensors. Open one in any reader and the tensors, shapes and metadata are
all there to read.

Each model page names the original it was built from, so the upstream weights are always one link
away.

## Installing a model

Install once, then load as often as you like. Installing is the only step that touches the network —
loading reads what is already on your machine, so nothing downloads behind your back.

In Mila's chat harness:

```
/install Llama-3.2-3B-Instruct-fp4
/model Llama-3.2-3B-Instruct-fp4
```

From Python ([`mila-llm`](https://pypi.org/project/mila-llm/)):

```python
import mila

mila.ModelStore().pull("Llama-3.2-3B-Instruct-fp4", mila.default_hub_owner())

tokenizer = mila.BpeTokenizer.from_store("Llama-3.2-3B-Instruct-fp4")
model = mila.LlamaModel.from_store("Llama-3.2-3B-Instruct-fp4", 4096)
```

From C++:

```cpp
ModelStore store;
ModelResolver resolver( store, *makeDefaultModelHub() );

resolver.pull( "Llama-3.2-3B-Instruct-fp4", std::string( kDefaultHubOwner ) );

const auto model = store.locate( "Llama-3.2-3B-Instruct-fp4" );
```

Every file is checked against a published checksum as it downloads, and the chat harness, the Python
package and the inference server all share one local store — install a model once and all three can
use it.

---

[mila.toddt.me](https://mila.toddt.me) · [GitHub](https://github.com/ToddThomson/Mila) ·
[PyPI](https://pypi.org/project/mila-llm/)