CMSManhattan/JiRackTernaryPrecision-36B-chat

JiRack Ternary Precision 36B (GGUF)

JiRack Ternary Precision 36B is a compact ternary (1.58-bit) model for agents and function calling by CMSManhattan (Konstantin Grabko), with 35.78B parameters. Every linear layer of the transformer uses ternary weights {-1, 0, +1} with a per-tensor scale and 8-bit activations (BitNet b1.58 style), trained with quantization-aware training.

• Function calling: BFCL v3 non-live 82.2% (1,390 cases), ahead of the original DeepSeek-R1-Distill-Qwen-32B on the same benchmark (74.2%); parallel calls 84.0% vs 55.5%.
• Small and fast: on a GPU the model needs about 11 GiB of memory instead of about 61 GiB for the bf16 original and generates 108.5 tokens/s for one user instead of 51.3 tokens/s. It also runs on a plain CPU.
• Two GGUF files with the same weights: TQ2_0 (9.91 GB) for CPUs, Q2_0 (10.71 GB) for NVIDIA GPUs. Both store the ternary weights exactly.

Live demo (CPU, TQ2_0): JiRack-36B-Chat

Which NVIDIA Jetson can hold each model (estimate)

Estimated from file sizes plus the KV cache for a 4k context, with about 2 GB of the shared memory left to the operating system. Not yet measured on Jetson devices.

Model	GGUF TQ2_0	Fits	Tight (short context)	Does not fit
2B	0.73 GB	Orin Nano 4/8 GB, Orin NX, AGX Orin, Thor	—	—
8B	2.6 GB	Orin Nano 8 GB, Orin NX 8/16 GB, AGX Orin, Thor	—	Orin Nano 4 GB
16B	4.8 GB	Orin NX 16 GB, AGX Orin, Thor	Orin Nano 8 GB, Orin NX 8 GB	Orin Nano 4 GB
36B	9.91 GB	AGX Orin 32/64 GB, Thor	Orin NX 16 GB	Orin Nano, Orin NX 8 GB

On Jetson, llama.cpp runs TQ2_0 on the CPU cores. To run on the Jetson GPU (CUDA), use the Q2_0 file of the same model: the same ternary weights stored exactly, slightly larger (36B: 10.71 GB).

Results

BFCL v3 (Berkeley Function Calling Leaderboard)

Category	Cases	JiRack Ternary 36B	DeepSeek-R1-Distill-Qwen-32B (bf16)
simple	400	90.3%	75.0%
multiple	200	89.0%	83.0%
parallel	200	84.0%	55.5%
parallel_multiple	200	75.5%	72.5%
java	100	66.0%	52.0%
javascript	50	56.0%	68.0%
irrelevance	240	79.2%	93.3%
Non-live overall	1,390	82.2%	74.2%
live_simple	258	70.9%	75.2%
live_multiple	1,053	63.1%	69.8%
live_parallel	16	56.3%	43.8%
live_parallel_multiple	24	58.3%	58.3%
live_relevance	18	72.2%	38.9%
live_irrelevance	882	63.5%	90.3%
Live overall	2,251	64.1%	77.9%

Overall = average weighted by the number of cases (evalscope). JiRack: the reference PyTorch model (the same ternary weights as the GGUF files) behind our OpenAI-compatible server, BFCL function-calling mode (tools passed as OpenAI tools), up to 1,024 new tokens. DeepSeek-R1-Distill-Qwen-32B: bf16 GGUF on llama-server, BFCL prompt mode, its own chat template with reasoning on, temperature 0.6 / top-p 0.95 as DeepSeek recommends, up to 4,096 new tokens. The original model is clearly better at declining when no tool fits (irrelevance, live_irrelevance); see Limitations.

Internal tool-calling test (117 cases, greedy)

The cases are written by us and are not in the training data. The GGUF files are served by llama-server through jirack_gguf_proxy.py, with the same prompt format as in fine-tuning.

Category	Reference (PyTorch, ternary)	GGUF TQ2_0 (CPU)	GGUF Q2_0 (GPU)
Single call (one tool, all arguments given)	38/38	38/38	38/38
Parallel calls	19/19	19/19	19/19
Follow-up (answer from a tool result)	20/20	20/20	20/20
No tool needed	25/25	25/25	25/25
Clarify a missing argument	13/15	10/15	9/15
Total	115/117 (98.3%)	112/117 (95.7%)	111/117 (94.9%)

The GGUF files match the reference model in every category except clarify, where they called a tool instead of asking for the missing argument in a few more cases.

Speed and memory on a GPU

NVIDIA GH200 (Hopper GPU), llama.cpp CUDA build ac8e1694d, all layers on the GPU. Peak GPU memory: nvidia-smi, sampled once per second during the run.

	JiRack Ternary 36B, Q2_0	DeepSeek-R1-Distill-Qwen-32B, BF16
File	9.97 GiB	61.03 GiB
Peak GPU memory, 1 stream (llama-bench)	10.7 GiB	60.7 GiB
Peak GPU memory, 32 parallel streams	20.5 GiB	70.7 GiB
Generation, 1 stream (tg128)	108.5 tokens/s	51.3 tokens/s
Generation, 8 parallel streams (total)	363 tokens/s	298 tokens/s
Generation, 32 parallel streams (total)	1,050 tokens/s	1,297 tokens/s
Prompt processing (pp512)	2,732 tokens/s	6,988 tokens/s

Parallel streams: llama-batched-bench, 512 prompt + 128 generated tokens per stream. The ternary model needs 5.7× less memory (it fits a 24 GB GPU even with 32 parallel streams) and answers a single user about 2× faster. With many parallel streams and for long prompts the bf16 model is faster on a Hopper GPU, because its matrix multiplications run on the tensor cores while Q2_0 unpacks the ternary weights first.

Speed on a CPU

llama-bench on the CPU of an NVIDIA GH200 node (Grace, 64 threads), CPU-only llama.cpp build:

File	Prompt processing (512 tokens)	Generation
TQ2_0	67.7 tokens/s	20.7 tokens/s
TQ1_0	45.7 tokens/s	24.5 tokens/s

Files

File	Size	Notes
JiRackTernaryPrecision-36B-TQ2_0.gguf	9.91 GB	recommended for CPUs; tested (above)
JiRackTernaryPrecision-36B-Q2_0.gguf	10.71 GB	recommended for NVIDIA GPUs (CUDA); same ternary weights, stored exactly; tested (above)
JiRackTernaryPrecision-36B-TQ1_0.gguf	8.31 GB	same ternary weights, denser packing; not separately evaluated
jirack_gguf_proxy.py + jirack_openai_server.py		OpenAI-compatible endpoint with tool calls on top of llama-server
chat_template.jinja		chat template embedded in the GGUF (plain chat); for tool calls use the proxy
eval/		tool-calling test logs and per-case outputs (CPU and GPU), llama-bench / llama-batched-bench results, GPU peak memory
NOTICE.md		third-party licenses

Run with llama.cpp

Plain chat on a CPU:

llama-cli -m JiRackTernaryPrecision-36B-TQ2_0.gguf -c 8192 -cnv

On an NVIDIA GPU use the Q2_0 file and a recent llama.cpp built with CUDA (-DGGML_CUDA=ON, a build that knows the Q2_0 type), with all layers on the GPU:

llama-server -m JiRackTernaryPrecision-36B-Q2_0.gguf -ngl 99 --port 8080 -c 32768 --special

Tool calling through an OpenAI-compatible API. llama-server returns the model's tool calls as text (<|tool_call_start|> {"name": ..., "arguments": ...} <|tool_call_end|>); the proxy builds the exact fine-tuning prompt and turns the calls into standard OpenAI tool_calls, so any OpenAI client or agent framework can use the model:

llama-server -m JiRackTernaryPrecision-36B-TQ2_0.gguf --port 8080 -c 32768 --special   # GPU: Q2_0 file + -ngl 99
python3 jirack_gguf_proxy.py --llama http://127.0.0.1:8080 --port 8001   # needs jirack_openai_server.py next to it

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8001/v1", api_key="none")
tools = [{"type": "function", "function": {
    "name": "get_weather", "description": "Current weather for a city",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]
r = client.chat.completions.create(model="jirack-36b", tools=tools,
                                   messages=[{"role": "user", "content": "Weather in Austin?"}])
print(r.choices[0].message.tool_calls)

Return tool results as tool messages; several calls in one turn (parallel calls) are supported. Use greedy decoding (temperature 0) for tool calls.

Model details

Parameters	35.78B
Layers	64
Hidden size / FFN	5,120 / 30,720
Attention heads	40 query, 8 key-value (GQA), head dim 128
Vocabulary	152,064
Context	up to 32k tokens
Weights	ternary {-1, 0, +1}, per-tensor scale
Activations	8-bit, per token

How the model was made

• Base: DeepSeek-R1-Distill-Qwen-32B (weights from Qwen2.5-32B, Apache 2.0; tokenizer and chat format from DeepSeek-R1-Distill, MIT), FFN widened 27,648 → 30,720 (35.78B parameters).
• Continued pretraining in phases: bf16, gradual ternarization, ternary on C4 + Wikipedia, then ternary on code.
• SFT, ternary throughout: main SFT on the JiRack SFT mix (730 steps × 131k tokens), then the agentic stage on the JiRack agentic mix (tool calling and agent trajectories, 641 steps × 262k tokens, sequences up to 32k; final validation loss 0.8891).
• GGUF export: every ternary weight is stored exactly as the model computes it (clamp(round(W / mean|W|), -1, 1) * mean|W|, 448 matrices), then packed as TQ2_0 / TQ1_0 (CPU) and Q2_0 (GPU).

Intended use

Tool calling, function calling and agent pipelines; local, offline and on-premise deployment where GPU memory or hardware cost is limited.

Limitations

• The model calls a tool more often than it should: when no available tool fits the request (BFCL irrelevance 79.2%, live_irrelevance 63.5%) or when a required argument is missing (clarify 9–10/15 with the GGUF files). Validate calls before executing them, and give the agent an explicit way to answer without a tool.
• The model can make mistakes, invent facts and produce wrong tool arguments. Check its output before acting on it, especially in automated pipelines.
• Ternary model: world knowledge is below that of the bf16 base model; not for medical, legal or financial advice.

ROBOS HUMANOID Project

• https://github.com/cmsmanhattan/ROBOS

📧 Contact

For joint venture opportunities, hardware integration, or licensing inquiries:

• Email: grabko@cmsmanhattan.com
• Phone: +1 (516) 777-0945
• Location: New York, USA

License

© 2025-2026 Konstantin Vladimirovich Grabko, all rights reserved.