README.md

---
license: apache-2.0
base_model: convaiinnovations/laya
base_model_relation: finetune
pipeline_tag: text-classification
language: [en]
tags: [laya, code-search, reranker, code-retrieval, calibrated, claude-code, laya-codex]
---

# laya-code

A code-relevance re-ranker fine-tuned from [Laya](https://huggingface.co/convaiinnovations/laya)
(ModernBERT-large encoder + typed-decision head, 421M parameters). Given a task description and a
source-code chunk, it answers one yes/no (`noul`) question with a **calibrated probability**:

```
question: Is this source code relevant to the software change: "{task}"?
state:    file: <path> (lines a-b)\n<code>        (truncated to 128 tokens in production)
```

It is the default re-ranker of [laya-codex](https://github.com/pilotspace/laya-codex), which feeds
Claude Code the most relevant code spans for a prompt (tree-sitter chunks, then Moon BM25
candidates, then laya-code re-ranking).

## Model details

| | |
|---|---|
| Base model | `convaiinnovations/laya`, root checkpoint (revision `1c5edc17a7acd8701df6fc341c0d179f1c62c982`), Apache-2.0 |
| Architecture | unchanged: same safetensors keys, shapes and dtypes as the base (205 F16 tensors + `temperature` F32), 842,609,210 bytes |
| What changed | `model.safetensors` (fine-tuned weights) and `rl_agent_config.json` (`model_name`, `noul:2` temperature **0.9410**, was 1.9834; `finetune` block). `encoder/config.json`, `tokenizer/*`, `rl_agent_api.py` and `rl_common.py` are byte-identical to the base. The `training` block of `rl_agent_config.json` is inherited from the base and describes the base's training, not this fine-tune. |
| Context | 512 tokens (`max_len`), 192 for the question head (`head_max_len`) |
| Runtime | Python: `rl_agent_api.RLAgent` from this repo (same as the base). Rust: `laya-model` crate of laya-codex (candle; Metal F16 on macOS, CPU F32 elsewhere), parity-tested against the Python reference |
| License | Apache-2.0 (see `LICENSE` and `NOTICE`) |

## Training data

Weak supervision from git history. Nothing was hand-labelled.

- **8 training repositories** (mixed Rust, Python, TypeScript and JavaScript): openai/codex,
  TinDang97/velos, MervinPraison/PraisonAI, badlogic/pi-mono, Portkey-AI/gateway (local
  `ai-guard` checkout), TinDang97/python-dependency-injector, Netflix/dispatch and
  pilotspace/hydroa (local `ai-proxy` checkout). Source: `finetune/repos.py`.
- **Held out** (never used for training or calibration): pilotspace/moon and pilot-space.
  Moon client codebases (helios, helios-mono, lunaris) were left out so that Moon vocabulary
  does not leak into the Moon eval. A root-commit check rejects forks and clones of each other
  and of the held-out repos.
- **Examples**: up to 600 non-merge commits per repo (the `build_data.py` default) that touch 1–4 source files and have an
  informative subject (at least 20 characters). The task is the subject plus a short first body
  line. The candidate list is the BM25 top 12 over other files plus the parent-revision windows
  of the touched files (40-line windows, stride 30). Labels: 1.0 for a window that overlaps a
  changed hunk, 0.7 for another window of a touched file, 0 for other files. Extra examples:
  hunks BM25 missed (label 1.0), a random same-file window (label 0.4) and random windows from
  other files (label 0).
- **Size**: 50,926 pairs, split by commit hash into 45,790 train and 5,136 validation pairs.
  Train pairs: 2,779 candidate positives, 5,609 candidate same-file, 24,768 candidate negatives,
  4,656 extra positives, 2,392 same-file, 5,586 random negatives. Per-repo counts (v2 data;
  the warm start used v1 data from the same repos):

  | repo | train commits | val commits | train pairs | val pairs |
  |---|---|---|---|---|
  | PraisonAI | 231 | 18 | 3,711 | 280 |
  | ai-guard | 460 | 40 | 7,404 | 665 |
  | ai-proxy | 200 | 23 | 3,293 | 380 |
  | codex | 446 | 54 | 7,676 | 932 |
  | dispatch | 447 | 53 | 7,347 | 863 |
  | pi-mono | 439 | 61 | 7,386 | 1,013 |
  | python-dependency-injector | 453 | 47 | 7,066 | 744 |
  | velos | 117 | 16 | 1,907 | 259 |
- **Training**: top 8 of 28 encoder layers, the final norm and the decision head. fp32 on an
  M4 Pro (MPS). AdamW; learning rate 2e-5 for the encoder and 1e-4 for the head. 32 sequences
  per update. Log loss against soft targets. 536 updates on v2 data, warm-started from 300
  updates on v1 data (about 2.3 h in total). Checkpoint chosen by lowest validation NLL. The
  `noul:2` temperature was then refitted on the validation split, using the exported F16
  weights.

## Evaluation

All numbers come from files in the laya-codex repository and are quoted as recorded. Gold labels
are file-level: the files the commit touched. Candidates are BM25 windows at HEAD.

### Re-ranker comparison, 128-token state (production setting)

`spike/results/compare_models.json` (`spike/compare_models.py`): the 40 most recent qualifying
moon commits, BM25 top 24, state truncated to 128 tokens, probabilities pooled over all
candidates (base rate 0.309).

| model | Laya-only MRR | Laya-only P@10 | RRF MRR | AUROC | ECE | mean P |
|---|---|---|---|---|---|---|
| BM25 alone | 0.480 (MRR) | 0.340 | – | – | – | – |
| laya-base (`convaiinnovations/laya`) | 0.479 | 0.348 | 0.505 | 0.586 | 0.362 | 0.671 |
| laya-typed-decisions | 0.441 | 0.288 | 0.456 | 0.539 | 0.239 | 0.545 |
| **laya-code** | **0.702** | **0.405** | **0.630** | **0.713** | **0.049** | 0.328 |

### Held-out evaluation, 256-token state (training protocol)

`spike/results/finetune_eval.json` (`finetune/eval.py`): 40 tasks per held-out repo, BM25 top 32,
state truncated to 256 tokens. Paired bootstrap over the tasks.

| repo | model | Laya-only MRR | RRF MRR | RRF P@10 | ECE (15 bins) | AUROC pooled |
|---|---|---|---|---|---|---|
| moon | laya-base | 0.497 | 0.586 | 0.343 | 0.461 | 0.584 |
| moon | laya-code | 0.526 | 0.519 | 0.398 | 0.060 | 0.677 |
| pilot-space | laya-base | 0.519 | 0.762 | 0.323 | 0.509 | 0.536 |
| pilot-space | laya-code | 0.646 | 0.714 | 0.393 | 0.016 | 0.709 |

Under this protocol, laya-code clearly improves calibration and discrimination. It puts more
gold-file spans in the top 10: RRF P@10 rose by +0.055 (95% CI [0.013, 0.098]) on moon and by
+0.070 ([0.033, 0.108]) on pilot-space. Its **RRF MRR did not beat laya-base's**: −0.066
([−0.195, 0.061]) on moon and −0.049 ([−0.172, 0.073]) on pilot-space. For that reason,
laya-codex fuses laya-code by score (`(1−w)·lexical + w·P`, w = 0.5) rather than by RRF.

End to end, the laya-codex paired Claude Code benchmark (20 moon tasks, `docs/RESULTS.md`)
measured −45% code-reading tokens and −25% wall-clock time for the whole pipeline, with no loss
of answer recall. The same document reports that the model's **marginal** contribution over
lexical-only ranking is within run-to-run noise at n = 20. Do not read the pipeline numbers as a
property of this model.

## Intended use

- Re-ranking lexical (BM25) candidates of source-code chunks for a natural-language
  software-change task, as a calibrated `P(relevant)`.
- Gating or fusing retrieval results by probability; P is calibrated to the training
  distribution (ECE ≤ 0.06 on held-out repos).

## Out of scope and limitations

- **Weak labels.** A commit touching a file does not make every window of it relevant, and the
  file-level gold is coarse.
- **Small evaluation.** Each held-out set has 40 tasks, and most CIs are wide. The two
  protocols (128 vs 256 tokens, top 24 vs 32) give different absolute numbers, as shown above.
- **Low probabilities.** P rarely exceeds 0.5 (max 0.49 on moon, 0.50 on pilot-space at 256
  tokens). Use rank or score fusion, or a threshold near 0.4, not "P ≥ 0.5 means relevant".
- **Under-trained.** Only about half an epoch of the v2 data was used, on a shared laptop.
  Validation AUROC was still rising when training stopped.
- **English prompts only.** Training covered Rust, Python, TypeScript and JavaScript; other
  languages are untested.
- **Other tasks untested.** It is not a general Laya replacement: `choice`/`score` questions
  (for example, task scope) were not trained, and zero-shot scope accuracy is poor
  (`spike/results/scope_eval.json`).
- **Too slow for interactive CPU use.** At 421M parameters, CPU re-ranking of 24 candidates is
  too slow for interactive use. laya-codex runs it on Metal, or falls back to lexical ranking.
- **Legal status of training data.** The model was trained on permissively licensed public
  code plus the author's own repositories. It is a classifier and cannot reproduce that code,
  but the legal status of weights trained on source code is not settled.

## License and attribution

Apache-2.0, like the base model. laya-code is a Derivative Work of
[convaiinnovations/laya](https://huggingface.co/convaiinnovations/laya) (Apache-2.0, © Convai
Innovations), which builds on
[answerdotai/ModernBERT-large](https://huggingface.co/answerdotai/ModernBERT-large) (Apache-2.0).
The modified files are `model.safetensors` and `rl_agent_config.json`; every other file is
unchanged from the base. See `NOTICE`.

## Files

See `MANIFEST.sha256` for the sha256 of every uploaded file.