Instructions to use VertexAGI/amethyst-2-5-micro with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use VertexAGI/amethyst-2-5-micro with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("VertexAGI/amethyst-2-5-micro") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use VertexAGI/amethyst-2-5-micro with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf VertexAGI/amethyst-2-5-micro:Q4_K_M # Run inference directly in the terminal: llama cli -hf VertexAGI/amethyst-2-5-micro:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf VertexAGI/amethyst-2-5-micro:Q4_K_M # Run inference directly in the terminal: llama cli -hf VertexAGI/amethyst-2-5-micro:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf VertexAGI/amethyst-2-5-micro:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf VertexAGI/amethyst-2-5-micro:Q4_K_M
Use Docker
docker model run hf.co/VertexAGI/amethyst-2-5-micro:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use VertexAGI/amethyst-2-5-micro with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "VertexAGI/amethyst-2-5-micro" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "VertexAGI/amethyst-2-5-micro", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/VertexAGI/amethyst-2-5-micro:Q4_K_M
- Ollama
How to use VertexAGI/amethyst-2-5-micro with Ollama:
ollama run hf.co/VertexAGI/amethyst-2-5-micro:Q4_K_M
- Unsloth Desktop
- Pi
How to use VertexAGI/amethyst-2-5-micro with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "VertexAGI/amethyst-2-5-micro"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "VertexAGI/amethyst-2-5-micro" } ] } } }Run Pi
# Start Pi in your project directory: pi
- MLX LM
How to use VertexAGI/amethyst-2-5-micro with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "VertexAGI/amethyst-2-5-micro"
Run an OpenAI-compatible server
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "VertexAGI/amethyst-2-5-micro" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8080/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "VertexAGI/amethyst-2-5-micro", "messages": [ {"role": "user", "content": "Hello"} ] }' - Docker Model Runner
How to use VertexAGI/amethyst-2-5-micro with Docker Model Runner:
docker model run hf.co/VertexAGI/amethyst-2-5-micro:Q4_K_M
- Lemonade
How to use VertexAGI/amethyst-2-5-micro with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull VertexAGI/amethyst-2-5-micro:Q4_K_M
Run and chat with the model
lemonade run user.amethyst-2-5-micro-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use VertexAGI/amethyst-2-5-micro with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "VertexAGI/amethyst-2-5-micro"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default VertexAGI/amethyst-2-5-micro
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use VertexAGI/amethyst-2-5-micro with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "VertexAGI/amethyst-2-5-micro"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "VertexAGI/amethyst-2-5-micro" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Amethyst 2.5 Micro
Amethyst 2.5 Micro is the 1.7B-parameter little sibling of Amethyst 2.5 Mini: a small chat model fine-tuned from Qwen3-1.7B on the same data, aimed at three behaviours:
- Know when to search. It calls a
web_searchtool for things that change (news, scores, prices, "latest") and answers stable questions directly. - Answer from the results, with citations. After a search it answers briefly and cites the snippets it used as
[1],[2]. - Say so when the results don't answer. If the search failed, returned nothing, or returned on-topic pages that never state the answer, it is trained to say that plainly rather than guess.
What actually worked, and what didn't. The clear win over the Qwen3-1.7B base is format and brevity: the tuned model emits a correct tool call on every search prompt we tried, never searches on plain questions, cites its sources, and answers plain questions in fewer words. The base 1.7B mostly fails to produce a usable tool call with this prompt. The quality of answers after a search is about a wash against the base (8 wins vs 7 losses, no-think arm), and Micro loses head-to-head to Mini (4B), 4 wins to 9. It is a small, cheap, well-behaved tool-caller, not a smarter model than its base. Details below.
Part of the Amethyst family of chat-optimized models, from a small, independent, nonprofit-spirited AI lab that works in the open.
Available formats
Everything below lives in this one repository: pick the format that matches your runtime.
| Format | File / folder | Precision | Size | Works with |
|---|---|---|---|---|
| MLX | model.safetensors (repo root) |
mixed: tuned layers 8-bit, the rest 4-bit | 1.67 GB | mlx-lm / MLX on Apple Silicon |
| GGUF | amethyst_2_5_micro_Q4_K_M.gguf |
Q4_K_M (4-bit) | 1.11 GB | llama.cpp, Ollama, LM Studio |
| GGUF | amethyst_2_5_micro_Q8_0.gguf |
Q8_0 (8-bit) | 1.83 GB | llama.cpp, Ollama, LM Studio |
Why the MLX build is "mixed": a plain 4-bit merge of a LoRA re-quantizes the fine-tune away. Here all 196 LoRA-adapted linear layers (7 per layer x 28 layers) are merged at fp16 and re-quantized at 8-bit. On 60 validation conversations the merged model's full-sequence loss was 3.337 against 3.331 for the unmerged adapter (that figure includes the system prompt, so it is not comparable to the training-time validation loss), and its held-out scores match the adapter's.
Model Details
| Developed by | Independent research project |
| Base model | Qwen/Qwen3-1.7B |
| Fine-tuning base checkpoint | mlx-community/Qwen3-1.7B-4bit |
| Architecture | Qwen3, 1.7B parameters (dense, decoder-only transformer, tied embeddings) |
| Fine-tuning method | LoRA (rank 16, scale 20.0, all 28 layers), loss on assistant turns only |
| Fine-tuning framework | MLX / mlx-lm, on Apple Silicon |
| Language | English |
| License | Apache 2.0 (same as the base model) |
Training Data
The same 3,205 synthetic conversations (4,683 training samples after splitting each conversation into one sample per assistant turn) used for Amethyst 2.5 Mini, generated by distillation from large teacher models through NVIDIA NIM and OpenRouter: GLM 5.3 (and GLM 5.3 Flash), Kimi K3, poolside Laguna S 2.1 and MiniMax M3.1 Flash. Every search result in the data is invented by a teacher; no real web pages were scraped.
| Category | Conversations | What it teaches |
|---|---|---|
| General chat | 817 | Short, direct answers to ordinary questions |
| Plain, short answers (tool available, no search) | 309 + 42 two-turn | Stable questions are answered directly in a few words |
| Terse-format instructions | 500 | "Exactly 3 bullets", "under 20 words", "one sentence", translation, rewrite |
| Current-fact search, one query | 502 | One search, brief cited answer |
| Two sources that disagree | 127 | Says the sources differ and cites both |
| Weak first results, then a better query | 97 + 21 | Refines the search once, then answers |
| Search failed (timeout, 503, no results) | 192 | Says it couldn't get the information; no guessing |
| Results are on-topic but don't contain the answer | 448 | Says the results don't give it, cites what they do say |
| Summaries, multi-query, follow-ups, no-search | 150 | Cited summaries, query decomposition, elliptical follow-ups |
Search-result rows passed an automatic grounding check: the final answer may only cite results that exist and may only use numbers and capitalised names that appear in the last result or in the question. The set caps repeats per prompt (an earlier 20,000-row set held only ~1,000 unique prompts).
Training Procedure
- Method: supervised fine-tuning via LoRA (rank 16, scale 20.0, dropout 0.0, all 28 layers), fresh from the base, loss on assistant turns only (
mask_prompt) - Optimizer: Adam, learning rate 6e-5 (constant), batch size 4, sequence length 2,048, gradient checkpointing
- Schedule: 2 epochs, 2,341 iterations, one uninterrupted run. Validation loss (assistant tokens only) fell from 4.59 to about 1.01 at iteration ~1,400 and ended at 1.155, so it drifted up slightly late in training; the final weights, not the best checkpoint, are what is released.
Evaluation
Greedy decoding on hand-made held-out prompts (tool decisions, answers after a search result, plain questions), three sets of 44 (fresh, written after the training mix was frozen and excluded from it by keyword), 64 and 36 prompts. The 64- and 36-prompt sets overlap training topics more than the 44-prompt set, so read the 44-prompt column as the cleanest. All sets are small. Base columns are the Qwen3-1.7B base with enable_thinking=False / the default template.
| Metric | Fresh 44: Micro | base (no-think / default) | 64: Micro | base | 36: Micro | base |
|---|---|---|---|---|---|---|
Well-formed web_search calls |
100% | 0% / 13% | 100% | 0% / 25% | 100% | 0% / 14% |
| Searches on plain questions (lower is better) | 0% | 0% / 6% | 0% | 0% / 4% | 0% | 0% / 0% |
| Cites after a search | 100% | 46% / 54% | 94% | 19% / 50% | 100% | 25% / 75% |
| Answers grounded in the result | 69% | 54% / 46% | 100% | 100% / 88% | 100% | 88% / 88% |
| Plain answer length (words) | 16 | 21 / 22 | 33 | 56 / 63 | 37 | 50 / 42 |
The base's tool-call numbers are low mostly because the 1.7B base does not follow the <tool_call> format from the system prompt, so part of the gap is plain instruction-following, not tool judgment. The 69% grounded figure on the fresh set is Micro's weakest number (it is the same as Mini's on that set).
Judged against the 1.7B base (strict four-verdict panel: two judge models, both answer orders, a win only when all four verdicts agree; failed judge calls are counted as failed, never as ties):
| Set | vs base, no-think | vs base, default template |
|---|---|---|
| Fresh 44 | 13 wins, 1 loss, 10 ties, 5 failed | 11 wins, 1 loss, 10 ties, 7 failed |
| 64 | 11 wins, 9 losses, 12 ties, 8 failed | 13 wins, 9 losses, 15 ties, 3 failed |
| 36 | 6 wins, 4 losses, 11 ties, 1 failed | 7 wins, 3 losses, 12 ties, 0 failed |
| Total (91 pairs) | 30 wins, 14 losses, 33 ties, 14 failed | 31 wins, 13 losses, 37 ties, 10 failed |
The wins come mostly from plain answers (tighter and cleaner). On answers after a search the result is close to even: against the no-think base, Micro wins 8 and loses 7 across the three sets.
Head-to-head against Amethyst 2.5 Mini (4B) on the fresh 44 set: Micro 4 wins, Mini 9, 14 ties, 2 failed. Mini is better, as expected for a model more than twice the size; the tool-call and grounding numbers are identical on this set, so the gap is in the answers themselves.
The judge panel changed during this work (one originally chosen judge was withdrawn by its host and its replacement was picked by an audit that planted known defects in real answers), so these verdicts are not directly comparable to earlier Amethyst releases' numbers.
Tool-calling format
The model expects this system prompt and emits tool calls in exactly this format:
You are Amethyst, a helpful, direct conversational assistant. Answer the actual question asked. Match your length to the question -- a one-line question gets a one-line answer, a substantive question gets real depth. Use plain language, skip filler openers, and don't restate the question back before answering. Format with markdown only when structure genuinely helps (lists, tables, code); prose questions get prose answers.
You have access to one tool:
web_search(queries: list[str]) -- search the live web. Pass 1-3 short keyword queries. Use it when the answer depends on current, local, or frequently changing information (news, prices, scores, releases, schedules, weather, "latest"/"current"/"today"). Do NOT use it for stable knowledge you already have -- arithmetic, definitions, settled history, language questions, or code.
To call it, emit exactly this and nothing else:
<tool_call>
{"name": "web_search", "arguments": {"queries": ["query one", "query two"]}}
</tool_call>
Results come back as:
<tool_result>
[1] Title -- snippet text (source.com)
[2] Title -- snippet text (source.com)
</tool_result>
After results arrive, answer in prose and cite the snippets you used as [1], [2], etc. If the results don't actually answer the question, say so plainly rather than guessing.
The tool result goes back inside the next user turn (the model was trained with tool results folded into the user message, not a separate tool role). The search backend is yours to provide.
Use Qwen's default chat template. All results above were measured with it; enable_thinking=False was not tested for the tuned model.
Usage β MLX
from mlx_lm import load, generate
model, tokenizer = load("VertexAGI/amethyst-2-5-micro")
SYSTEM = "..." # the system prompt above
messages = [{"role": "system", "content": SYSTEM},
{"role": "user", "content": "What's the weather like in Chicago right now?"}]
text = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
print(generate(model, tokenizer, prompt=text, max_tokens=200))
Strip the empty think block. With the default template the model starts every reply with an empty <think>\n\n</think> block, as Qwen3 does. Remove it before parsing a tool call or showing the answer, for example re.sub(r"<think>.*?</think>", "", text, flags=re.S).strip(). The scores above were computed after stripping it.
Usage β llama.cpp
llama-server -m amethyst_2_5_micro_Q4_K_M.gguf --jinja -c 4096
Send the system prompt above as the system message. Both GGUFs were checked through llama-server on the fresh 44-prompt set and scored the same as the MLX build (100% well-formed calls, 0% needless searches, 100% cited, 69% grounded; plain answers 14-16 words).
Intended Use
A very small conversational assistant with tool-use awareness: experiments, research on small-scale distillation, and hobbyist or on-device deployments with a search backend wired up. It is not intended for high-stakes decisions.
Limitations
- Trained on about 3,200 synthetic conversations. Behaviour outside the categories above can be inconsistent.
- At 1.7B it reads search results no better than its base. Judged answer quality after a search was about even with the base, and clearly behind Mini (4B). Use Mini if you can afford 4B.
- It reads whatever the search tool returns. The synthetic training results were invented by teacher models, so some facts in the data are fictional by design; the grounding it learned is "answer from the results", not "know the truth".
- Grounding is imperfect. On the fresh 44-prompt set only 69% of post-search answers passed our automatic grounding check.
- All evaluation sets are small and hand-made, and judged by language models; 14 of 91 judged pairs against the no-think base failed to get a full verdict. Treat results as directional.
- One tool (
web_search) with a fixed schema; other tool schemas are untested. - Inherits the knowledge cutoff and general limitations of Qwen3-1.7B. English only.
- Downloads last month
- 446
4-bit