File size: 1,378 Bytes
3d5dbbb | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | ---
license: mit
library_name: mlx
tags:
- mlx
- function-calling
- encoder-decoder
base_model: Cactus-Compute/needle
---
# Needle MLX
MLX port of [Cactus-Compute/needle](https://huggingface.co/Cactus-Compute/needle),
a 26M-parameter encoder-decoder tool-calling model.
## Run
```bash
uv run python - <<'PY'
import mlx.core as mx
from model import SimpleAttentionNetwork
from tokenizer import NeedleTokenizer, build_encoder_input
path = "mlx-model" # or a downloaded Hugging Face snapshot
model = SimpleAttentionNetwork.from_pretrained(path)
tokenizer = NeedleTokenizer.from_pretrained(path)
tools = '[{"name":"get_weather","parameters":{"location":"string"}}]'
src = mx.array([build_encoder_input(tokenizer, "Weather in Delhi?", tools)])
print(tokenizer.decode(model.generate(src, max_new_tokens=128)).removeprefix("<tool_call>"))
PY
```
Create the upload directory from the original checkpoint:
```bash
uv run convert.py weights mlx-model
uvx --from huggingface_hub hf upload mlx-community/needle-mlx mlx-model .
```
## Compatibility
This repository uses a custom Needle architecture. It runs through the included
MLX implementation; it is not currently supported by `mlx-lm`, Ollama,
llama.cpp, or LM Studio. GGUF or an Ollama `Modelfile` alone cannot add the
missing architecture implementation.
## License
MIT. Original model and weights: Cactus Compute.
|