CMSManhattan/JiRackTernaryPrecision-2B-chat

JiRack Ternary Precision 2B (GGUF)

JiRack Ternary Precision 2B is a ternary (1.58-bit) language model by CMSManhattan (Konstantin Grabko) with 2.01B parameters, built for tool calling and agents at a very low inference cost. Every linear layer of the transformer uses ternary weights {-1, 0, +1} with a per-tensor scale and 8-bit activations (BitNet b1.58 style). This repository has the model as GGUF for llama.cpp: 0.73 GB instead of 4.0 GB in bf16, runs on a CPU. The Transformers version (trust_remote_code) is CMSManhattan/JiRackPrecision_2b.

Quick start

HF Spaces demo chat: https://huggingface.co/spaces/CMSManhattan/JiRack-2B-Chat

Files

File	Size	Notes
JiRackTernaryPrecision-2B-TQ2_0.gguf	0.73 GB	recommended; tested against the reference model (below)
JiRackTernaryPrecision-2B-TQ1_0.gguf	0.65 GB	same ternary weights, denser packing; not separately evaluated
jirack_gguf_proxy.py + jirack_openai_server.py		OpenAI-compatible endpoint with tool calls on top of llama-server
chat_template.jinja		the exact fine-tuning prompt format (also embedded in the GGUF)
eval/		tool-calling test log of the TQ2_0 file

Model

Architecture	Qwen2-style decoder, 28 layers, hidden 1536, FFN 10752, 12 heads / 2 KV heads (GQA), bias on q/k/v, separate output head
Parameters	2.01B (196 ternary linear layers)
Weights	ternary {-1, 0, +1} with a per-tensor scale, 8-bit activations
Tokenizer	Qwen-family BPE tokenizer with `<
Context	up to 16,384 tokens

GGUF export: every ternary weight is stored exactly as the model computes it (clamp(round(W / mean|W|), -1, 1) * mean|W|), then packed as TQ2_0 / TQ1_0.

Which NVIDIA Jetson can hold each model (estimate)

Estimated from file sizes plus the KV cache for a 4k context, with about 2 GB of the shared memory left to the operating system. Not yet measured on Jetson devices.

Model	GGUF TQ2_0	Fits	Tight (short context)	Does not fit
2B	0.73 GB	Orin Nano 4/8 GB, Orin NX, AGX Orin, Thor	—	—
8B	2.6 GB	Orin Nano 8 GB, Orin NX 8/16 GB, AGX Orin, Thor	—	Orin Nano 4 GB
16B	4.8 GB	Orin NX 16 GB, AGX Orin, Thor	Orin Nano 8 GB, Orin NX 8 GB	Orin Nano 4 GB
36B	9.91 GB	AGX Orin 32/64 GB, Thor	Orin NX 16 GB	Orin Nano, Orin NX 8 GB

On Jetson, llama.cpp runs TQ2_0 on the CPU cores. To run on the Jetson GPU (CUDA), use the Q2_0 file of the same model: the same ternary weights stored exactly, slightly larger (36B: 10.71 GB).

Results

Internal tool-calling test (117 cases, greedy)

Category	Reference (PyTorch)	GGUF TQ2_0
Single call (one tool, all arguments given)	34/38	34/38
Parallel calls	18/19	18/19
Follow-up (answer from a tool result)	18/20	18/20
No tool needed	25/25	25/25
Clarify a missing argument	14/15	13/15
Total	109/117 (93.2%)	108/117 (92.3%)

Same 117 cases and the same prompt format in both runs; the GGUF file is served by llama-server through jirack_gguf_proxy.py. The GGUF file misses one more case than the reference (clarify 13/15 vs 14/15): a difference of one case in either direction is expected between the two backends, which do the arithmetic differently; the packed file keeps the model's quality.

BFCL v4 (reference model, greedy)

Single-turn Python subset (11 categories, 3,491 cases) scored with the official BFCL AST checker (gorilla commit 6ea57973c7); Java / JavaScript, multi-turn and agentic categories are not included, so this is not an official leaderboard entry.

	Plain decoding	Name-constrained decoding
Non-live AST (mean of 4 categories)	47.5%	59.2%
Non-live overall (+ irrelevance)	50.3%	59.6%
Live AST (weighted)	13.9%	32.9%

"Name-constrained" restricts the function name to the offered tools while decoding (the model still chooses which tool, its arguments and whether to call at all). Without it most failures are paraphrased function names (predict_house_price -> predict_home_price). The proxy in this repository uses plain decoding; with llama.cpp the same constraint can be added as a grammar.

MMLU (57 subjects, cloze 0-shot, acc_norm): 30.8%.

Quick start (llama.cpp)

Plain chat:

llama-cli -m JiRackTernaryPrecision-2B-TQ2_0.gguf -c 8192 -cnv

Tool calling through an OpenAI-compatible API. llama-server returns the model's tool calls as text (<|tool_call_start|> {"name": ..., "arguments": ...} <|tool_call_end|>); the proxy builds the exact fine-tuning prompt and turns the calls into standard OpenAI tool_calls, so any OpenAI client or agent framework can use the model:

llama-server -m JiRackTernaryPrecision-2B-TQ2_0.gguf --port 8080 -c 16384 --special
python3 jirack_gguf_proxy.py --llama http://127.0.0.1:8080 --port 8001 --name jirack-2b   # needs jirack_openai_server.py next to it

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8001/v1", api_key="none")
tools = [{"type": "function", "function": {
    "name": "get_weather", "description": "Current weather for a city",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]
r = client.chat.completions.create(model="jirack-2b", tools=tools,
                                   messages=[{"role": "user", "content": "Weather in Austin?"}])
print(r.choices[0].message.tool_calls)

Return tool results as tool messages; several calls in one turn (parallel calls) are supported. Use greedy decoding (temperature 0) for tool calls.

Speed (llama-bench, CPU only, 2x Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz, 48 threads): 71.8 tokens/s prompt processing (512 tokens), 28.8 tokens/s generation. This CPU lacks AVX2, which the TQ2_0 kernels use for speed.

Recommended use

1. Validate calls before executing them (tool exists, required arguments present, values taken from the conversation); if a check fails, ask the user instead of letting the model answer a live-data question from memory.
2. Offer only the tools that fit the task; fewer, clearly named tools give the best accuracy.
3. Use a calculator tool for arithmetic.

Limitations

• Copies unusual function names imprecisely without name-constrained decoding (see BFCL above).
• Sometimes calls a tool when none fits (BFCL live_irrelevance 37.0% with plain decoding); requests that need several different tools at once are the weakest category.
• Without tools it may state live data (time, weather) from memory; do not rely on that.
• Small model: limited world knowledge (MMLU cloze 30.8%). Not for medical, legal or financial advice.

Training data attribution

Includes data from the Allen Institute for AI licensed under ODC-BY 1.0 and text by Wikipedia contributors licensed under CC BY-SA 4.0, together with data created by CMSManhattan.

ROBOS HUMANOID Project

• https://github.com/cmsmanhattan/ROBOS

📧 Contact

For joint venture opportunities, hardware integration, or licensing inquiries:

• Email: grabko@cmsmanhattan.com
• Phone: +1 (516) 777-0945
• Location: New York, USA

License

© 2025-2026 Konstantin Vladimirovich Grabko, all rights reserved.