Instructions to use VertexAGI/amethyst-2-5-mini with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use VertexAGI/amethyst-2-5-mini with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("VertexAGI/amethyst-2-5-mini") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use VertexAGI/amethyst-2-5-mini with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf VertexAGI/amethyst-2-5-mini:Q4_K_M # Run inference directly in the terminal: llama cli -hf VertexAGI/amethyst-2-5-mini:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf VertexAGI/amethyst-2-5-mini:Q4_K_M # Run inference directly in the terminal: llama cli -hf VertexAGI/amethyst-2-5-mini:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf VertexAGI/amethyst-2-5-mini:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf VertexAGI/amethyst-2-5-mini:Q4_K_M
Use Docker
docker model run hf.co/VertexAGI/amethyst-2-5-mini:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use VertexAGI/amethyst-2-5-mini with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "VertexAGI/amethyst-2-5-mini" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "VertexAGI/amethyst-2-5-mini", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/VertexAGI/amethyst-2-5-mini:Q4_K_M
- Ollama
How to use VertexAGI/amethyst-2-5-mini with Ollama:
ollama run hf.co/VertexAGI/amethyst-2-5-mini:Q4_K_M
- Unsloth Desktop
- Pi
How to use VertexAGI/amethyst-2-5-mini with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "VertexAGI/amethyst-2-5-mini"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "VertexAGI/amethyst-2-5-mini" } ] } } }Run Pi
# Start Pi in your project directory: pi
- MLX LM
How to use VertexAGI/amethyst-2-5-mini with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "VertexAGI/amethyst-2-5-mini"
Run an OpenAI-compatible server
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "VertexAGI/amethyst-2-5-mini" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8080/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "VertexAGI/amethyst-2-5-mini", "messages": [ {"role": "user", "content": "Hello"} ] }' - Docker Model Runner
How to use VertexAGI/amethyst-2-5-mini with Docker Model Runner:
docker model run hf.co/VertexAGI/amethyst-2-5-mini:Q4_K_M
- Lemonade
How to use VertexAGI/amethyst-2-5-mini with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull VertexAGI/amethyst-2-5-mini:Q4_K_M
Run and chat with the model
lemonade run user.amethyst-2-5-mini-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use VertexAGI/amethyst-2-5-mini with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "VertexAGI/amethyst-2-5-mini"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default VertexAGI/amethyst-2-5-mini
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use VertexAGI/amethyst-2-5-mini with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "VertexAGI/amethyst-2-5-mini"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "VertexAGI/amethyst-2-5-mini" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Amethyst 2.5 Mini
Amethyst 2.5 Mini is a small chat model fine-tuned from Qwen3-4B to do three things well:
- Know when to search. It calls a
web_searchtool for things that change (news, scores, prices, "latest") and answers stable questions directly, without searching. - Answer from the results, with citations. After a search it answers briefly and cites the snippets it used as
[1],[2]. - Say so when the results don't answer. If the search failed, returned nothing, or returned on-topic pages that never state the answer, it says that plainly and points you to where to look. It does not guess or invent a figure.
It also keeps answers short: on two of our three held-out sets its plain answers average 31-38 words, against 45-66 for the Qwen3-4B base (on the third set the base is just as brief).
Part of the Amethyst family of chat-optimized models, from a small, independent, nonprofit-spirited AI lab that works in the open.
Available formats
Everything below lives in this one repository: pick the format that matches your runtime.
| Format | File / folder | Precision | Size | Works with |
|---|---|---|---|---|
| MLX | model.safetensors (repo root) |
mixed: tuned layers 8-bit, the rest 4-bit | 3.07 GB | mlx-lm / MLX on Apple Silicon |
| GGUF | amethyst_2_5_mini_Q4_K_M.gguf |
Q4_K_M (4-bit) | 2.50 GB | llama.cpp, Ollama, LM Studio |
| GGUF | amethyst_2_5_mini_Q8_0.gguf |
Q8_0 (8-bit) | 4.28 GB | llama.cpp, Ollama, LM Studio |
Why the MLX build is "mixed": a plain 4-bit merge of the LoRA re-quantizes the fine-tune away. Here the 112 LoRA-adapted linear layers are merged at fp16 and re-quantized at 8-bit; every other layer stays at the base model's original 4-bit. We checked that the merged model matches the adapter: same validation loss to three decimals, and identical scores on our held-out prompts.
Model Details
| Developed by | Independent research project |
| Base model | Qwen/Qwen3-4B |
| Fine-tuning base checkpoint | mlx-community/Qwen3-4B-4bit |
| Architecture | Qwen3, 4B parameters (dense, decoder-only transformer) |
| Fine-tuning method | LoRA (rank 16, scale 20.0, 16 layers), loss on assistant turns only |
| Fine-tuning framework | MLX / mlx-lm, on Apple Silicon |
| Language | English |
| License | Apache 2.0 (same as the base model) |
Training Data
3,205 conversations (4,683 training samples after splitting every conversation into one sample per assistant turn), all synthetic, generated by distillation from large teacher models through NVIDIA NIM and OpenRouter: GLM 5.3 (and GLM 5.3 Flash), Kimi K3, poolside Laguna S 2.1 and MiniMax M3.1 Flash. Every search result in the data is invented by a teacher; no real web pages were scraped.
The mix was built for diversity and concision, not volume. An earlier 20,000-row set turned out to hold only about 1,000 unique prompts (some repeated more than 200 times) and trained a model that copied its long answers. This release caps repeats per prompt, drops long or uncited answers, and adds new rows.
| Category | Conversations | What it teaches |
|---|---|---|
| General chat | 817 | Short, direct answers to ordinary questions |
| Plain, short answers (tool available, no search) | 309 + 42 two-turn | Stable questions are answered directly in a few words |
| Terse-format instructions | 500 | "Exactly 3 bullets", "under 20 words", "one sentence", translation, rewrite |
| Current-fact search, one query | 502 | One search, brief cited answer |
| Two sources that disagree | 127 | Says the sources differ and cites both |
| Weak first results, then a better query | 97 + 21 | Refines the search once, then answers |
| Search failed (timeout, 503, no results) | 192 | Says it couldn't get the information; no guessing |
| Results are on-topic but don't contain the answer | 448 | Says the results don't give it, cites what they do say, points to where to check |
| Summaries, multi-query, follow-ups, no-search | 150 | Cited summaries, query decomposition, elliptical follow-ups |
Search-result rows passed an automatic grounding check: the final answer may only cite results that exist and may only use numbers and capitalised names that appear in the last result or in the question.
Training Procedure
- Method: supervised fine-tuning via LoRA (rank 16, scale 20.0, dropout 0.0, 16 layers), fresh from the base, loss on assistant turns only (
mask_prompt) - Optimizer: Adam, learning rate 6e-5 (constant), batch size 4, sequence length 2,048, gradient checkpointing
- Schedule: 2 epochs, 2,341 iterations. The run was paused once by hand at iteration ~1,580 and resumed from the iteration-1,400 weights for the remaining 941 iterations (optimizer state was not saved, so Adam restarted cold at the resume point)
Evaluation
All numbers are from greedy decoding on hand-made held-out prompts (tool decisions, answers after a search result, plain questions). Three sets of 44 (fresh, written after the training mix was frozen and excluded from it by keyword), 64 and 36 prompts. The 64- and 36-prompt sets overlap the training topics more than the 44-prompt set does, so read the 44-prompt column as the cleanest. All sets are small.
| Metric | Fresh 44: Amethyst 2.5 Mini | Qwen3-4B base (no-think / default) | 64: Amethyst | base | 36: Amethyst | base |
|---|---|---|---|---|---|---|
Well-formed web_search calls |
100% | 60% / 73% | 100% | 88% / 88% | 100% | 71% / 100% |
| Searches on plain questions (lower is better) | 0% | 0% / 0% | 0% | 0% / 4% | 0% | 0% / 7% |
| Cites after a search | 100% | 100% / 100% | 100% | 100% / 100% | 100% | 100% / 100% |
| Answers grounded in the result | 69% | 69% / 62% | 100% | 94% / 88% | 100% | 100% / 100% |
| Plain answer length (words) | 19 | 19 / 22 | 38 | 50 / 58 | 31 | 45 / 66 |
On the fresh set the gap to the base is mostly well-formed tool calls. The 69% grounded figure on that set is the model's weakest number (the same as the base's best case), and it is driven by a few prompts, not a broad pattern.
Head-to-head against our previous candidate (the same recipe without the 448 "results don't answer" rows), strict four-verdict panel (two judge models, both answer orders, a win only when all four agree): 12 wins, 15 losses, 56 ties, 8 judge failures over 91 judged pairs. That is a dead heat. The one clear difference is behaviour on a prompt where the results are on-topic but contain no answer ("What's the best-selling book this week?"): this model says the results don't identify it and cites what they do say; the previous candidate searched again.
That previous candidate beat the Qwen3-4B base 16 to 7 (no-think) and 14 to 4 (default template) on a single-judge comparison over the earlier sets; this exact model was not judged directly against the base. It also beat Amethyst 1.5 Mini 20 to 10. A Gemma 3 4B version trained on the same data lost to the Qwen version 5 to 17 (69 ties).
The judge panel changed during this work: one of the two originally chosen judges was withdrawn by its host, and its replacement was picked by an audit that planted known defects in real answers (changed numbers, stripped citations, filler, a false "couldn't find it", a leaked tool call). Verdicts here are therefore not directly comparable to earlier Amethyst releases' numbers.
Tool-calling format
The model expects this system prompt and emits tool calls in exactly this format:
You are Amethyst, a helpful, direct conversational assistant. Answer the actual question asked. Match your length to the question -- a one-line question gets a one-line answer, a substantive question gets real depth. Use plain language, skip filler openers, and don't restate the question back before answering. Format with markdown only when structure genuinely helps (lists, tables, code); prose questions get prose answers.
You have access to one tool:
web_search(queries: list[str]) -- search the live web. Pass 1-3 short keyword queries. Use it when the answer depends on current, local, or frequently changing information (news, prices, scores, releases, schedules, weather, "latest"/"current"/"today"). Do NOT use it for stable knowledge you already have -- arithmetic, definitions, settled history, language questions, or code.
To call it, emit exactly this and nothing else:
<tool_call>
{"name": "web_search", "arguments": {"queries": ["query one", "query two"]}}
</tool_call>
Results come back as:
<tool_result>
[1] Title -- snippet text (source.com)
[2] Title -- snippet text (source.com)
</tool_result>
After results arrive, answer in prose and cite the snippets you used as [1], [2], etc. If the results don't actually answer the question, say so plainly rather than guessing.
The tool result goes back inside the next user turn (the model was trained with tool results folded into the user message, not a separate tool role). The search backend is yours to provide.
Use Qwen's default chat template. All results above were measured with it. Behaviour with enable_thinking=False was not tested for this release; an earlier Amethyst model lost much of its tool-calling reliability that way.
Usage β MLX
from mlx_lm import load, generate
model, tokenizer = load("VertexAGI/amethyst-2-5-mini")
SYSTEM = "..." # the system prompt above
messages = [{"role": "system", "content": SYSTEM},
{"role": "user", "content": "What's the weather like in Chicago right now?"}]
text = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
print(generate(model, tokenizer, prompt=text, max_tokens=200))
# -> <tool_call>{"name": "web_search", "arguments": {"queries": ["Chicago weather now"]}}</tool_call>
Strip the empty think block. With the default template the model starts every reply with an empty <think>\n\n</think> block, as Qwen3 does. Remove it before parsing a tool call or showing the answer, for example re.sub(r"<think>.*?</think>", "", text, flags=re.S).strip(). The scores above were computed after stripping it.
Usage β llama.cpp
llama-server -m amethyst_2_5_mini_Q4_K_M.gguf --jinja -c 4096
Send the system prompt above as the system message. Both GGUFs were checked through llama-server on the 44-prompt set and scored the same as the MLX build (100% well-formed calls, 0% needless searches, 100% cited, 69% grounded).
Intended Use
A lightweight conversational assistant with tool-use awareness: experiments, research on small-scale distillation, and hobbyist deployments with a search backend wired up. It is not intended for high-stakes decisions.
Limitations
- Trained on about 3,200 synthetic conversations. Behaviour outside the categories above can be inconsistent.
- It reads whatever the search tool returns. Synthetic training results were invented by teacher models, so some facts in the data are fictional by design; the grounding it learned is "answer from the results", not "know the truth".
- Grounding is imperfect. On the fresh 44-prompt set only 69% of post-search answers passed our automatic grounding check; a few answers add detail the results don't contain.
- All evaluation sets are small and hand-made, and judged by language models. Treat the head-to-head as "no difference we can detect", not as proof of equality.
- One tool (
web_search) with a fixed schema; other tool schemas are untested. - Inherits the knowledge cutoff and general limitations of Qwen3-4B. English only.
- Downloads last month
- 591
4-bit