ColGrep-Agent: Cutting Frontier Coding Agents’ Tokens by a Third

We are releasing ColGrep-Agent-2B, an open 2-billion-parameter code-search agent that runs locally and sits between your coding agent and the repository. Given a question, it searches with ColGrep, our multi-vector semantic code search, reads the promising files, follows references, and hands back file paths, line ranges and the code itself. We plugged it into Claude Code on 52 code-search questions across 13 repositories unseen during training: delegating search to the local agent cut API tokens by 34% and cache-aware API cost by 40% relative to grep, with no statistically significant change in accuracy.

Contents

1. Introduction
2. Getting started
   • 2.1 Install ColGrep and the model
   • 2.2 Index a repository
   • 2.3 Ask a question
   • 2.4 Connect it to your coding agent
3. What happens when Claude Code delegates its search
   • 3.1 How we measured it
   • 3.2 A third fewer tokens
   • 3.3 Example
   • 3.4 Why it saves tokens
   • 3.5 Accuracy and time
   • 3.6 Which setup should you use
4. How good is a 2B code-search agent?
   • 4.1 Comparison with larger models
   • 4.2 Comparison with other code search tools
5. Training ColGrep-Agent-2B
   • 5.1 Agent harness
   • 5.2 Training data
   • 5.3 Training
6. Small models, big savings

1. Introduction

Before a coding agent can fix a bug, it often has to spend several turns figuring out where to look. It greps for a name, opens a file, follows a reference, tries another search. Each step goes through the frontier model, and the growing conversation gets sent back on the next turn.

We built ColGrep-Agent-2B to handle that part locally. It's a 2-billion-parameter model that searches a repository with ColGrep, reads promising files, follows references, and returns the relevant file paths and line ranges along with the code.

We tested it with Claude Code running Sonnet 5.5 on 52 code-search questions across 13 repositories. Each question was run three times in each of four setups, for 624 sessions total. When Claude Code used ColGrep-Agent as its search tool (colgrep settings --hook-mode agent), it consumed 34% fewer API tokens than grep. These results are for code-search questions, not end-to-end coding tasks.

Figure 1. Claude Code sends a question to the local 2B agent. The agent searches with ColGrep and returns code locations, so Claude doesn't have to do all the exploration itself.

A quick distinction before we get into it: ColGrep is the semantic code search engine, ColGrep-Agent-2B is the model, and colgrep --agent is the command that runs the model with ColGrep and a read-only terminal.

2. Getting started

2.1. Install ColGrep and the model

brew install lightonai/tap/colgrep && colgrep --install-agent

The second command downloads the default colgrep-agent-2B-Q8_0.gguf model (about 2.5 GB), verifies that it loads, and prepares the prompt cache. There's no Python environment to set up, and no external inference server or API key.

Inference uses the GPU when available (Metal on Apple Silicon, Vulkan on NVIDIA, AMD and Intel GPUs), or falls back to the CPU. On an M3 Pro, a question takes around 20 s with Metal with 3–4 GB of peak memory usage.

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/lightonai/next-plaid/releases/latest/download/colgrep-installer.sh | sh && "${CARGO_HOME:-$HOME/.cargo}/bin/colgrep" --install-agent

irm https://github.com/lightonai/next-plaid/releases/latest/download/colgrep-installer.ps1 | iex
colgrep --install-agent

cargo install colgrep && colgrep --install-agent

2.2. Index a repository

cd /path/to/project
colgrep init

You only need to build the index once. Subsequent searches detect changed files and update the index automatically.

2.3. Ask a question

Here's a session with colgrep 1.8.5 on ColGrep's own source code, running on a Linux machine with an NVIDIA H100. We've left out the agent's reasoning and the individual tool results, and shortened the code of the second location.

    33  /// The llama.cpp backend, initialized once per process.
    34  ///
    35  /// On Metal, initialization compiles ggml's GPU kernels from their embedded source.
    36  /// macOS caches compiled shaders (keyed by source and device) in a small shared cache
    37  /// in `$(getconf DARWIN_USER_CACHE_DIR)com.apple.metal`: a hit takes ~50 ms, a miss
    38  /// ~18 s on an M3 Pro. Misses happen on the first run of a new llama.cpp version and
    39  /// whenever other apps' shader compiles have rotated our entry out of the cache.
    40  fn backend() -> Result<&'static LlamaBackend, LlmError> {
    41      static BACKEND: OnceLock<Result<LlamaBackend, String>> = OnceLock::new();
    42      BACKEND
    43          .get_or_init(|| {
    44              // Route llama.cpp/ggml logs to `tracing` (no subscriber: dropped) before the
    45              // backend initializes, so device-init chatter never reaches the terminal.
    46              llama_cpp_2::send_logs_to_tracing(llama_cpp_2::LogOptions::default());
    47              let t = Instant::now();
    48              let backend = LlamaBackend::init().map_err(|e| e.to_string());
    49              BACKEND_READY.store(true, Ordering::Release);
    50              // Printed later by `load`: stderr may be silenced while this runs in the
    51              // background (colgrep mutes it during the encoder load).
    52              BACKEND_INIT_US.store(t.elapsed().as_micros() as u64, Ordering::Release);
    53              backend
    54          })
    55          .as_ref()
    56          .map_err(|e| LlmError::Engine(format!("llama.cpp init: {e}")))
    57  }

    65          let (gpu, lib_dir, gpu_note) = if opts.gpu_layers > 0 {
    66              find_gpu(server, opts.progress)
    67          } else {
    68              (None, None, None)
    69          };
        …
    95              .args(["--n-gpu-layers", &opts.gpu_layers.to_string()])
        …

$ colgrep --agent "where does the agent decide which GPU backend llama.cpp uses"
◆ colgrep agent  · colgrep-agent-2B-Q8_0.gguf (llama.cpp b11476, GPU: NVIDIA H100 80GB HBM3)
❯ where does the agent decide which GPU backend llama.cpp uses
  ● colgrep   llama.cpp GPU backend
  ● terminal  cat -n agent/src/llm/llama.rs
  ● terminal  cat -n agent/src/llm/llama.rs | sed -n '80,200p'
  ● colgrep   device backend Metal CUDA Vulkan llama.cpp
  ● terminal  cat -n agent/src/llm/server.rs
  ● terminal  cat -n agent/src/llm/server.rs | sed -n '85,589p'
  ● colgrep   find_gpu
  ● terminal  cat -n agent/src/llm/server.rs | sed -n '160,589p'
  ● finish    agent/src/llm/llama.rs:40-57, agent/src/llm/server.rs:65-105
✓ 2 locations · 9 turns · 3 searches · 5 terminal commands · 5.5s

agent/src/llm/llama.rs:40-57  backend
    33  /// The llama.cpp backend, initialized once per process.
    34  ///
    35  /// On Metal, initialization compiles ggml's GPU kernels from their embedded source.
    36  /// macOS caches compiled shaders (keyed by source and device) in a small shared cache
    37  /// in `$(getconf DARWIN_USER_CACHE_DIR)com.apple.metal`: a hit takes ~50 ms, a miss
    38  /// ~18 s on an M3 Pro. Misses happen on the first run of a new llama.cpp version and
    39  /// whenever other apps' shader compiles have rotated our entry out of the cache.
    40  fn backend() -> Result<&'static LlamaBackend, LlmError> {
    41      static BACKEND: OnceLock<Result<LlamaBackend, String>> = OnceLock::new();
    42      BACKEND
    43          .get_or_init(|| {
    44              // Route llama.cpp/ggml logs to `tracing` (no subscriber: dropped) before the
    45              // backend initializes, so device-init chatter never reaches the terminal.
    46              llama_cpp_2::send_logs_to_tracing(llama_cpp_2::LogOptions::default());
    47              let t = Instant::now();
    48              let backend = LlamaBackend::init().map_err(|e| e.to_string());
    49              BACKEND_READY.store(true, Ordering::Release);
    50              // Printed later by `load`: stderr may be silenced while this runs in the
    51              // background (colgrep mutes it during the encoder load).
    52              BACKEND_INIT_US.store(t.elapsed().as_micros() as u64, Ordering::Release);
    53              backend
    54          })
    55          .as_ref()
    56          .map_err(|e| LlmError::Engine(format!("llama.cpp init: {e}")))
    57  }

agent/src/llm/server.rs:65-105  LlamaServer.start
    65          let (gpu, lib_dir, gpu_note) = if opts.gpu_layers > 0 {
    66              find_gpu(server, opts.progress)
    67          } else {
    68              (None, None, None)
    69          };
        …
    95              .args(["--n-gpu-layers", &opts.gpu_layers.to_string()])
        …

Each location starts with path/to/file:START-END and the name of the enclosing function, followed by its code, so a coding agent can often answer without opening the file. The format is also easy to feed into scripts. When you run the command in a terminal, progress goes to stderr; stdout is reserved for the results.

A couple more examples

Search within a directory and include the matching code lines:

colgrep --agent "crash when the config file is empty" ./backend -c

Get JSON output:

colgrep --agent --json "where are retries configured"

To see the ColGrep agent in action, explore the additional trajectories on SWE-bench Lite and Multi-SWE-bench Flash in the dedicated space below!

2.4. Connect it to your coding agent

colgrep --install-claude-code   # Claude Code
colgrep --install-codex         # Codex
colgrep --install-opencode      # OpenCode

With the integration installed, Claude Code can pass a question like "Where is this behavior implemented?" to colgrep --agent and get back locations and code instead of working through every search itself.

The Claude Code integration uses hooks that suggest a search tool in the agent's context; they don't force Claude to use it. You can change the suggestion with colgrep settings --hook-mode:

colgrep settings --hook-mode hybrid    # default: Balanced setup in section 3
colgrep settings --hook-mode agent     # suggest ColGrep-Agent: largest token savings
colgrep settings --hook-mode colgrep   # suggest plain ColGrep only

3. What happens when Claude Code delegates its search

3.1. How we measured it

Repositories. We chose 13 public repositories created in 2026 to reduce the chance that the models had seen them during training. We also asked Sonnet 5.5, without repository access, to name 10 source files from each one. It couldn't name any. That's a useful sanity check, though it doesn't prove the repositories were absent from training data. The repositories span Rust, Go, TypeScript and Python, and range from 17k to 1.8M lines of code.

Questions. We built 52 questions, four per repository and ran 3 times each question:

• 39 cross-folder questions: we find a real reference chain (a symbol defined in one place and used in one or two others) then ask Claude Opus 5.5 to write a debugging question that requires finding all those locations.
• 13 single-location questions: these describe a behavior without necessarily using the same words as the code.

Setups. We run Claude Code with Sonnet 5.5, keeping the question and repository fixed while changing the available search tools. Each setup gets its own isolated configuration:

Table 1. The four setups. colgrep --install-claude-code installs the hooks that suggest the available search tools.

We ran each question three times per setup: 624 sessions in total.

What we count. Token totals include Claude Code's API input and output tokens. They don't include tokens generated by the local 2B model, since those aren't billed. We compute 95% bootstrap confidence intervals over questions, averaging the three runs of each question first.

Grading. Claude Opus 5.5 grades the final answers against the reference locations as correct, partial or incorrect, without knowing which setup produced them. For cross-folder questions, an answer is correct only if it finds every required location. A separate judge compares the full traces: for each question and run, it sees the four traces in random order and ranks them by answer correctness, then by how directly they reached the answer.

3.2. A third fewer tokens

Figure 2. Token savings relative to Grep. Dots show point estimates; bars show 95% bootstrap confidence intervals over questions.

Table 2. Results across 52 questions, each run three times per setup.

With ColGrep-Agent, Claude Code used 34.3% fewer tokens than with Grep: about 49k fewer per question. It used fewer tokens on 88% of the questions, and the 95% confidence interval for savings was 28–40%.

ColGrep-Agent answered 80.8% of questions correctly, versus 83.3% for Grep (−2.6 percentage points; 95% CI −10.9 to +5.1). That difference isn't statistically significant, but the interval is wide enough that we can't rule out a meaningful drop. Balanced version matched Grep's observed 83.3% accuracy while saving 21% of tokens.

Figure 3. How often the trace judge prefers each setup over Grep.

The trace judge also favored ColGrep-Agent: in head-to-head comparisons with Grep, it ranked the agent's trace higher 75% of the time. A common pattern was one delegated search instead of several rounds of guessing symbol names.

3.3. Example

Here's a concrete example from a CRM repository. These are the first runs from each setup, with commands shortened for readability. Both answers were correct. This particular question saw 51% token savings, so it's a better-than-average case.

If I add a new property to the custom attributes that admins define on records (like contacts or deals), where is the shape of the list returned to the frontend declared, and where does the auto-generated typed API layer bring that shape in to type the endpoint that lists those attributes?

The tricky part is vocabulary: the code calls these things fields (fieldDefinition, fieldListOutput), not "custom attributes." With grep, Claude has to work out the naming first:

Grep: 6 turns, 5 tool calls, 160.6k tokens
  grep "customField\|attributeDefinition"    → a planning doc
  grep "attribute"                           → dozens of unrelated files
  grep "propertyDefinition\|fieldDefinition" → apps/api/src/fields/
  cat fields/*.router.ts; grep "field" generated/*.ts
                                             → fieldListOutput, server.ts
  grep -A6 "fieldListOutput" fields.contracts.ts
                                             → the schema, line 101
  Answer: fields.contracts.ts (serializedFieldOutput, fieldListOutput),
          generated/server.ts (fields router)                 ✓ correct

ColGrep-Agent: 3 turns, 2 tool calls, 79.2k tokens
  colgrep --agent "custom attribute definitions list endpoint output
    schema, and generated typed trpc API layer (AppRouter) that
    imports that shape"                      → fields.service.ts, with its code
  grep "fieldList" server.ts; grep "serializedField"
                                             → fields.contracts.ts:101, server.ts:26
  Answer: fields.contracts.ts (fieldListOutput),
          generated/server.ts (appRouter)                     ✓ correct

ColGrep's multi-vector search can match a description of "custom attributes" to code about fields (see section 5.3). The agent then follows up on the promising results and returns code locations. Claude still checks the answer, but it doesn't have to do all the initial digging.

3.4. Why it saves tokens

Figure 4. Average tool calls per question, by tool.

The main saving comes from cutting down Claude's search turns. Each new turn resends its accumulated context (about 23k tokens per turn in these runs). With Grep, Claude averaged 5.5 turns, 5.1 grep/find/ls commands and 2.0 file reads per question. With ColGrep-Agent, it averaged 3.9 turns, 2.0 grep commands and 0.9 file reads.

The larger the repository, the more search work there tends to be. We measured savings of 19% below 50k lines, 29% from 50k to 150k, and 40% from 150k to 600k. ColGrep-Agent saved tokens on all 13 repositories.

3.5. Time it

Figure 5. Wall-clock time per question, compared with Grep's 13.0 s.

The trade-off is latency. On our GPU server, a local ColGrep-Agent call took 7 s at the median and 22 s at the 90th percentile. End-to-end question time rose from 13.0 s with Grep to 21.6 s with ColGrep-Agent.

More results: best trace, turns, repository size and per-repository savings

Figure 6. Share of questions and runs where each setup had the top-ranked trace.

Figure 7. Average Claude Code turns per question.

Figure 8. Token savings by repository size. On the two repositories over a million lines, where Grep averaged nearly nine turns per question, ColGrep and ColGrep-Agent saved 37–46%.

Figure 9. Token savings by repository, sorted by lines of code. With four questions per repository, individual estimates are noisy.

Figure 10. The ten questions with the largest token savings under ColGrep-Agent. Six are cross-folder questions.

4. How good is a 2B code-search agent?

4.1. Comparison with larger models

Figure 11. Average file F1 (left) and line F1 (right) versus model size, across SWE-bench Lite and Multi-SWE-bench Flash with the ColGrep harness.

We also wanted to know how well the 2B model could locate code on its own, rather than just how much it helped Claude Code. We adapted SWE-bench Lite and Multi-SWE-bench Flash into code-localization benchmarks, following the method in our paper.

The task here is not to generate a patch or resolve the issue. The model must identify the files and lines relevant to the bug. That lets us measure search accuracy separately from code generation.

Table 3. File and line F1 (%) on two code-localization benchmarks, averaged over five seeds.

We evaluate several open-source models using our ColGrep harness. File and line F1 measure overlap with the reference files and lines. Scores are averaged over five seeds (300 instances per benchmark per seed), with strict predictions, a 10-turn limit and mandatory final submission. Comparison models run at temperature 0.6 without reasoning.

ColGrep-Agent-2B delivers competitive repository-localization performance against models more than 10× its size. On SWE-bench Lite, it achieves the highest line F1 (19.92) among all evaluated models, outperforming substantially larger general-purpose models. This strong performance extends across multiple programming languages on Multi-SWE-bench Flash, where it achieves 62.57 file F1 and 15.93 line F1, surpassing several larger models in line localization. These results demonstrate that task-specific training enables compact models to rival much larger general-purpose models across diverse codebases and programming languages.

4.2. Comparison with other code search tools

On the Semble code-search benchmark, ColGrep-Agent-2B returns fewer files than ranked search, and most of them are relevant.

Table 4. Results on the Semble code-search benchmark (1,251 questions, 63 repositories, 19 languages), scored at the file level with the benchmark's code. Precision@10 is the share of returned files that are relevant, over at most 10 files; F1 combines it with recall@10 for each question. Higher is better, except for files returned.

5. Training ColGrep-Agent-2B

We started from MiniCPM5-2B and trained in two stages: a short supervised fine-tuning (SFT) cold start, then reinforcement learning with a variant of Group Relative Policy Optimization (GRPO).

Figure 12. The two training stages, SFT cold start then GRPO, with the reward used in the second stage. The harness and turn budget are identical at training time and in `colgrep --agent`.

5.1. Agent harness

We built a dedicated, minimalist harness for code localization with ColGrep. Given an issue and a repository, the agent searches the code with ColGrep, inspects promising locations and submits file paths with line ranges. The harness exposes just three tools and controls the search loop and turn budget.

The agent searches with ColGrep, our semantic code search tool built on NextPlaid. ColGrep supports natural-language queries alongside regex and path filters. It proposes relevant code; the agent decides which results to inspect and when to search again.

Under the hood, ColGrep uses LateOn-Code-edge, a 17M-parameter multi-vector retrieval model we introduced in an earlier post. Instead of representing a whole function with one embedding, it keeps a vector for each token and compares query tokens against code tokens. You get some of the specificity of lexical retrieval, but with softer matching. For example, a query about "custom attributes" can surface fieldDefinition even though the words differ.

That matters because coding agents describe behavior differently from how a codebase names it. A useful first search can save several rounds of trying possible identifiers. The index is stored locally with NextPlaid and updated incrementally as files change; code doesn't need to leave the machine for retrieval or agent inference.

The agent has three tools:

Table 5. Tools available to ColGrep-Agent-2B.

It can search, open a promising file, follow a reference and search again, rather than betting everything on one query. It has a 10-turn budget, with the harness forcing a finish call on the final turn. The released colgrep --agent command uses the same system prompt, tool schemas, chat template and turn budget as training.

5.2. Training data

We built a multilingual code-localization dataset spanning several programming languages, using subsets of nebius/SWE-rebench and internlm/SWE-Fixer-Train-110K, alongside samples from additional scraped repositories. Each example contains a repository, an issue description, and the commit that fixes it. The agent searches the repository before the patch, while the patch provides reference file and line locations for training. We use these sources in both SFT and RL.

5.3. Training

For the cold start, we generate non-reasoning trajectories in our ColGrep harness using Qwen3.5-27B as the teacher, then filter them and perform SFT on MiniCPM5-2B. During teacher generation, we replace grep calls with corresponding ColGrep commands: ColGrep also supports grep-style lexical search using regex filters alone, without a natural-language query. This lets us retain useful trajectories that would otherwise use grep, while teaching the smaller model to use ColGrep through actual tool calls and results.

We follow this first stage with reinforcement learning using GRPO in the same code-localization harness. Inspired by DAPO, we use zero-advantage filtering to discard trajectories with no learning signal, including groups where every attempt receives the same reward. We also average the loss over trainable tokens across the batch and omit the KL-divergence penalty to the frozen reference model. Each trajectory's advantage is its reward minus the group's mean reward, without dividing by the group's standard deviation, as in Dr. GRPO.

The reward starts from the equal-weight average of file F1 and line F1, both on a 0–1 scale, with line F1 measuring exact file-and-line overlap. We apply a turn penalty after the second assistant turn to discourage unnecessary exploration. We also add a tiny penalty when the model invokes grep, and another penalty if it never searches or fails to submit locations through the finish tool.

We use asynchronous RL with a maximum of 4 off-policy steps, training for 500 steps and sampling 16 trajectories per issue with a training batch size of 128 trajectories. We optimize with AdamW at a constant learning rate of 1e-6, weight decay 0.01 and gradient norm clipping at 1.0. Rollouts use temperature 1.0, a 16,384-token context, and up to 1,024 generated tokens per turn, with reasoning disabled.

6. Small models, big savings

We don't think every step in a coding agent's workflow needs to run through a frontier model. Searching for the right files is a good example: it's important work, but much of it can be handed to a smaller model trained specifically for repository exploration.

That's what we tested with ColGrep-Agent-2B. In our code-search benchmark, giving Claude Code a local search agent reduced its API token usage by 34% and cache-aware API cost by 40%. Searches also took longer. For a lower-latency default, the Balanced setup saved 21% of tokens while matching Grep's observed accuracy.

The model and GGUF weights are available now.

Related resources

• LateOn-Code & ColGrep: LightOn unveils state-of-the-art code retrieval models and code search tooling: our earlier post on ColGrep and the LateOn-Code retrieval models.
• lightonai/LateOn-Code-edge and lightonai/LateOn-Code: the multi-vector retrieval models behind ColGrep.
• lightonai/colgrep-agent-2B and its GGUF weights.
• lightonai/next-plaid: ColGrep, colgrep --agent and the NextPlaid index.

Citation

@misc{colgrepagent_2026,
  title        = {ColGrep-Agent},
  author       = {Lasbordes, Maxence and Sourty, Raphael and Sinha, Aarush and Chatelain, Amélie},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/blog/lightonai/colgrep-agent/}}
}

Acknowledgements

ColGrep is developed in the open at lightonai/next-plaid. Thanks to its contributors: Chao-Chun (Joe) Hsu, Robert Wasilewski, Anton Chen, Vlad Lasky, Artem Dinaburg, DeeJayPee, George, Igor Carron, Omer Faruk Oruc, Abhishek Tripathi, Amir, graelo, Gray Olson, Jeff Coggshall, Jonathan Ellis, Mark Motliuk, NickSdot, Paolo Valletta, Raymond Weitekamp and Szymon Rączka.

We also relied on open models, datasets and tools: MiniCPM from OpenBMB, our base model; Qwen, whose Qwen3.5-27B we used as the SFT teacher; SWE-bench, Multi-SWE-bench, SWE-Fixer and SWE-rebench for training and evaluation; TRL, prime-rl and vLLM for training and inference; and llama.cpp, ONNX Runtime, tree-sitter and Hugging Face Tokenizers, which ColGrep uses.