---
license: apache-2.0
language:
- en
- ru
pipeline_tag: text-generation
tags:
- gguf
- llama.cpp
- imatrix
- quantized
- 35b-a3b
- mixture-of-experts
- moe
- agent
- search-agent
- agentic-search
- search
- information-retrieval
- retrieval
- tool-use
- function-calling
- rag
base_model:
- t-tech/T-Search
base_model_relation: quantized
---

# T-Search-GGUF

**🚨 T-Search-GGUF is designed for use with the [official T-Search harness](https://github.com/turbo-llm/t-search-harness). The source T-Search checkpoint did not show noticeable differences relative to the Qwen3.6-35B-A3B base model on general-purpose benchmarks.**

**🚨 Users are advised to exercise caution and are responsible for any additional training and oversight required to ensure the model's responses meet acceptable ethical and safety standards. The responsibility for incorporating this model into industrial or commercial solutions lies entirely with those who choose to deploy it.**

## **Highlights**

We introduce **T-Search-GGUF**, a set of GGUF quantizations of the T-Search agentic retriever for difficult multi-step search in English and Russian. T-Search plans and executes searches across multiple rounds with a configurable search budget.

- **GGUF checkpoints:** calibrated Q4_K_M, Q5_K_M, Q6_K, and Q8_0 quantizations for deployment with `llama.cpp`.
- **Production-aligned calibration:** importance-matrix calibration includes a deterministic, balanced sample of English and Russian production SFT traces.
- **Embedded MTP:** each GGUF contains the model's MTP head and can use self-speculative decoding without a separate draft checkpoint.
- **Agentic retrieval:** issues queries, inspects results, tracks coverage, carries compact evidence and search state across rounds, and returns a ranked chunk set.
- **Retrieval quality:** outperforms the base model and larger open models on average across the evaluated retrieval benchmarks.
- **Retriever robustness:** trained and evaluated with multiple retrievers, maintaining strong retrieval quality across configurations.

![T-Search benchmark recall](https://huggingface.co/t-tech/T-Search/raw/main/assets/t-search-benchmarks.svg)

## Description

T-Search-GGUF is built from the BF16 [`t-tech/T-Search`](https://huggingface.co/t-tech/T-Search) checkpoint. T-Search is based on **Qwen3.6-35B-A3B** and trained on fully synthetic search tasks generated for the same harness used at inference time.

The model is responsible for collecting evidence. Given a question and access to a corpus search backend, it formulates queries, reads retrieved snippets, decides which chunks are worth preserving, and returns a ranked evidence set. A downstream generator, reranker, or full-text fetcher can then consume that ranking.

## ⛷️ T-Search Harness

The [official T-Search harness](https://github.com/turbo-llm/t-search-harness) implements the inference protocol used for training and evaluation. The retriever operates in rounds. Within each round, it follows a ReAct loop with an approximately 32K-token context budget and three tools:

- `search_corpus` searches the corpus and returns document chunks;
- `save_and_advance` preserves important chunks and starts a new round;
- `finalize_ranking` ends the search and returns a ranked evidence set.

The agent sees the original question, previously saved chunks, and the current coverage state: which parts of the question are supported by evidence and which remain open. It chooses search queries, inspects retrieved chunks, and decides whether to continue searching, preserve state for another round, or finalize.

At 75% context utilization, `search_corpus` is locked. The agent must either call `finalize_ranking` or use `save_and_advance` to build compact memory for the next round: preserved chunks with reasons, covered and open parts of the question, previous attempts, and a useful next step. Full tool history and intermediate noise do not cross the round boundary.

## 🧠 Training

Training proceeds in two stages: supervised fine-tuning on fully synthetic search trajectories, followed by reinforcement learning with GSPO and recall-based rewards. At each stage, separate English and Russian experts are trained on language-specific data and then merged into a single checkpoint.

### Quantization

The GGUF files were produced from the BF16 [`t-tech/T-Search`](https://huggingface.co/t-tech/T-Search) checkpoint using [`llama.cpp`](https://github.com/ggml-org/llama.cpp).

An importance matrix was computed from a production-aligned calibration corpus that includes a deterministic 50/50 sample of English and Russian SFT traces.

The model's MTP/NextN layer is embedded directly in every GGUF file. Q4_K_M, Q5_K_M, and Q6_K use Q4_0 for the MTP tensors, while the Q8_0 release keeps them in Q8_0. No separate MTP sidecar is required.

The release was converted, calibrated, quantized, and validated with [`llama.cpp` build b10068](https://github.com/ggml-org/llama.cpp/commit/571d0d540df04f25298d0e159e520d9fc62ed121), commit `571d0d540df04f25298d0e159e520d9fc62ed121`.

| Quantization | File | Size | Embedded MTP |
|---|---|---:|---|
| **Q4_K_M** | `T-Search-Q4_K_M.gguf` | 21.71 GB | Q4_0 |
| **Q5_K_M** | `T-Search-Q5_K_M.gguf` | 25.35 GB | Q4_0 |
| **Q6_K** | `T-Search-Q6_K.gguf` | 29.21 GB | Q4_0 |
| **Q8_0** | `T-Search-Q8_0.gguf` | 37.80 GB | Q8_0 |

### Synthetic task factory

Training tasks contain a question, a fixed index, annotated evidence chunks, and a full tool-use trajectory. Candidate tasks pass adversarial checks for trivial query leakage, answerability from model weights, single-document shortcuts, missing evidence, and weak distractors.

### Supervised fine-tuning

Long teacher trajectories are split into self-contained rounds. Invalid tool actions are masked from the loss, while useful recovery behavior after a tool error is retained. Productive rounds are selected by evidence gain rather than by final recall alone, preserving useful behavior from difficult tasks.

Training uses **11K SFT examples from 8K unique questions per language**; one quarter targets robustness to different retrievers. An additional non-overlapping pool of **2K questions per language** is reserved for RL.

### Reinforcement learning

The policy is optimized with GSPO. RL optimizes the complete search policy on full tool-use trajectories. The main reward is recall over gold `chunk_id` values; precision and F-score are tracked for diagnostics but are not used as the primary training signal. This discourages the agent from finalizing early with a small high-precision but incomplete evidence set.

**A detailed Russian-language training report will be available soon on Habr.**

## 📊 Benchmarks

The reference results below are reproduced from the BF16 T-Search evaluation. GGUF results measured with the same harness and fixed benchmark indices are reported separately.

### Evaluation datasets

The accompanying evaluation datasets are released as [**TRuST**](https://huggingface.co/datasets/t-tech/TRuST) and [**SynthComp**](https://huggingface.co/datasets/t-tech/SynthComp).

| Benchmark | Description |
|---|---|
| **TRuST** | 324 manually authored Russian multi-step search questions with fixed indices and annotated evidence. |
| **SynthComp-En / SynthComp-Ru** | English and Russian synthetic fixed-index benchmarks with 395 questions each, separated from the SFT and RL data by question and evidence overlap checks. |
| **ru-BrowseComp-Plus** | Adapted Russian-language version of BrowseComp-Plus. |
| **ru-SealQA** | Adapted Russian-language version of SealQA. |

We use **Recall@10** as the primary metric.

We report single-rollout results for all models and, for T-Search, an additional N=3 configuration that runs three independent rollouts in parallel and combines their rankings using reciprocal rank fusion (RRF).

| Model | BrowseComp-Plus | ru-BrowseComp-Plus | SealQA | ru-SealQA | SynthComp-En | SynthComp-Ru | TRuST | Avg |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| **T-Search (N=3)** | **72.65** | **62.93** | **66.08** | **61.98** | **58.52** | **58.00** | **49.12** | **61.33** |
| **T-Search (N=1)** | 65.35 | 55.95 | 61.16 | 57.72 | 54.52 | 53.13 | 43.92 | 55.96 |
| GLM-5.1 | 64.32 | 58.18 | 55.49 | 53.21 | 51.69 | 51.71 | 43.11 | 53.96 |
| GLM-5.2 | 63.01 | 52.54 | 55.30 | 54.69 | 52.29 | 49.37 | 37.07 | 52.04 |
| Kimi-K2.6 | 60.71 | 49.76 | 56.86 | 52.46 | 48.25 | 47.06 | 42.39 | 51.07 |
| DeepSeek-V4-Flash | 53.27 | 46.30 | 55.70 | 51.12 | 44.83 | 45.68 | 40.40 | 48.19 |
| Qwen3.6-27B | 56.69 | 46.81 | 54.01 | 48.95 | 46.82 | 46.99 | 38.12 | 48.34 |
| Qwen3.6-35B-A3B (baseline) | 43.66 | 38.58 | 46.07 | 43.26 | 41.82 | 43.88 | 33.53 | 41.54 |
| gemma-4-26B-A4B-it | 35.32 | 27.51 | 45.13 | 39.05 | 36.24 | 34.77 | 21.75 | 34.25 |
| Qwen3.5-397B-A17B | 53.48 | 44.06 | 51.68 | 48.33 | 47.38 | 46.91 | 38.57 | 47.20 |
| Qwen3.5-122B-A10B | 47.81 | 40.15 | 51.57 | 44.74 | 42.11 | 42.25 | 30.55 | 42.74 |

### GGUF evaluation

The results below use the same T-Search harness, fixed benchmark indices, retriever configuration, generation parameters, 65,536-token serving context, and single-rollout (`N=1`) setup for every evaluated checkpoint. The original BF16 `t-tech/T-Search` checkpoint is included as a reference.

| Model | BrowseComp-Plus | ru-BrowseComp-Plus | SealQA | ru-SealQA | SynthComp-En | SynthComp-Ru | TRuST | Avg |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| **T-Search BF16 (original)** | **65.35** | **55.95** | **61.16** | **57.72** | **54.52** | **53.13** | **43.92** | **55.96** |
| T-Search-GGUF Q8_0 | 64.82 | 54.95 | 60.86 | 55.12 | 55.26 | 52.97 | 43.79 | 55.40 |
| T-Search-GGUF Q6_K | 63.54 | 54.38 | 60.91 | 54.17 | 53.20 | 52.42 | 43.22 | 54.55 |
| T-Search-GGUF Q5_K_M | 64.48 | 54.09 | 60.33 | 53.93 | 53.32 | 53.19 | 43.51 | 54.69 |
| T-Search-GGUF Q4_K_M | 60.82 | 52.79 | 60.44 | 53.38 | 53.67 | 51.67 | 42.49 | 53.61 |

---

We also study the latency–quality trade-off by varying the maximum number of search rounds and the number of parallel agent runs. This separates the effect of deeper sequential search from broader parallel exploration.

![Latency-quality trajectories](https://huggingface.co/t-tech/T-Search/raw/main/assets/t-search-latency-quality.svg)

To evaluate retrieval robustness, we varied the search backend while keeping the model and agent configuration fixed.

| Retriever | BrowseComp-Plus | ru-BrowseComp-Plus | SealQA | ru-SealQA | SynthComp-En | SynthComp-Ru | TRuST | Avg |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| Qwen3-Embedding-8B | 65.35 | 55.95 | 61.16 | 57.72 | 54.52 | 53.13 | 43.92 | 55.96 |
| **Qwen3-Embedding-8B + LLM reranking** | **75.04** | **66.24** | **64.82** | **59.95** | 62.71 | 62.39 | 48.93 | **62.87** |
| Qwen3-Embedding-0.6B | 51.70 | 43.71 | 56.80 | 49.53 | 54.72 | 52.27 | 36.95 | 49.38 |
| jina-embeddings-v5-text-small-retrieval | 60.52 | 51.41 | 62.46 | 55.60 | 56.37 | 54.56 | 39.31 | 54.32 |
| BM25 | 39.49 | 31.97 | 55.33 | 50.00 | **66.87** | **65.15** | **49.18** | 51.14 |

## 👨‍💻 Usage

### Recommended generation parameters

```yaml
do_sample: true
temperature: 0.7
top_p: 1.0
```

Serve `t-tech/T-Search-GGUF` through an OpenAI-compatible endpoint. See the [harness README](https://github.com/turbo-llm/t-search-harness) for installation and search-backend integration.

### Reference llama.cpp serving setup

Use `llama.cpp` build `b10068` or newer. The following configuration starts an OpenAI-compatible server with the embedded chat template, structured tool calling, separated reasoning output, and the embedded MTP head:

```bash
llama-server \
  --model "/path/to/T-Search-Q4_K_M.gguf" \
  --alias "t-tech/T-Search-GGUF" \
  --host 0.0.0.0 \
  --port 8000 \
  --n-gpu-layers all \
  --flash-attn on \
  --fit off \
  --ctx-size 65536 \
  --parallel 1 \
  --jinja \
  --reasoning on \
  --reasoning-format deepseek \
  --spec-type draft-mtp \
  --spec-draft-n-max 4 \
  --spec-draft-p-min 0.0 \
  --spec-draft-ngl all
```

Here, `deepseek` selects the OpenAI-compatible reasoning serialization format; it does not refer to the model architecture. With `--jinja`, `llama.cpp` uses the embedded T-Search chat template and exposes structured tool calls to the harness.

The context size is shared across parallel slots. To preserve a 65,536-token context for each concurrent request, set:

```text
ctx-size = parallel × 65536
```

For example, eight concurrent requests require `--parallel 8 --ctx-size 524288`.

For a two-GPU layer split, add:

```bash
--split-mode layer \
--tensor-split 1,1
```

### Example

```python
from retriever_agent import (
    AgentConfig,
    HttpSearchClient,
    OpenAILLMClient,
    RetrieverAgent,
)

config = AgentConfig(model="t-tech/T-Search-GGUF")
llm = OpenAILLMClient(["http://<llama-host>:8000/v1"], config)
search = HttpSearchClient("http://<search-host>:8000")

agent = RetrieverAgent(config, llm, search)
result = agent.retrieve("your query")

for doc in result.documents:
    print(doc.rank, doc.doc_id, doc.score, doc.text)
```