README.md

---
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- information-retrieval
- code-search
- code-embedding
- dense-retrieval
- modernbert
- generated_from_trainer
- dataset_size:4073472
- loss:CachedMultipleNegativesRankingLoss
base_model: Shuu12121/NightJar
pipeline_tag: feature-extraction
library_name: sentence-transformers
license: apache-2.0
language:
- en
metrics:
- ndcg_at_10
- cosine_accuracy@1
- cosine_accuracy@10
- cosine_ndcg@10
- cosine_mrr@10
model-index:
- name: NightJar-CodeSearch-Embedding
  results:
  - task:
      type: information-retrieval
      name: CodeSearchNet Retrieval
    dataset:
      name: CodeSearchNet (6-language macro average)
      type: code-search-net
    metrics:
    - type: ndcg_at_10
      value: 0.9026
      name: nDCG@10 (6-language macro average)
  - task:
      type: information-retrieval
      name: Code Edit Search Retrieval
    dataset:
      name: CodeEditSearchRetrieval (13-language macro average)
      type: code-edit-search
    metrics:
    - type: ndcg_at_10
      value: 0.7381646153846154
      name: nDCG@10 (13-language macro average)
---

# NightJar-CodeSearch-Embedding

NightJar-CodeSearch-Embedding is a 768-dimensional dense embedding model for
natural-language-to-code retrieval and code-edit retrieval. It is based on
[Shuu12121/NightJar](https://huggingface.co/Shuu12121/NightJar) and fine-tuned
with hard negatives and large in-batch negatives.

The model embeds a text query and a code document into the same vector space.
Higher cosine similarity indicates a stronger match. It does not require query
or document prefixes.

## Model details

| Property | Value |
|---|---|
| Architecture | ModernBERT Sentence Transformer |
| Parameters | Same transformer size as `Shuu12121/NightJar` |
| Embedding dimensions | 768 |
| Maximum sequence length | 1,024 tokens |
| Pooling | CLS token |
| Similarity | Cosine similarity |
| Training objective | Cached Multiple Negatives Ranking Loss |
| License | Apache-2.0 |

## Usage

### Sentence Transformers

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Shuu12121/NightJar-CodeSearch-Embedding")

query = "parse a JSON string and return an error when it is invalid"
code = [
    "def parse_json(text):\n    return json.loads(text)",
    "def read_lines(path):\n    return Path(path).read_text().splitlines()",
]

query_embedding = model.encode(query, normalize_embeddings=True)
code_embeddings = model.encode(code, normalize_embeddings=True)
scores = code_embeddings @ query_embedding

for score, snippet in sorted(zip(scores, code), reverse=True):
    print(float(score), snippet)
```

No `query:`, `passage:`, or task-specific prefix is needed. Use the same model
and encoding settings for queries and code. Normalized embeddings make the dot
product equivalent to cosine similarity.

### Transformers

Loading through `SentenceTransformer` is recommended because the repository's
pooling configuration is part of the model. If the backbone is loaded directly
with Transformers, use the final hidden state of the CLS token and L2-normalize
the resulting vector.

## Intended uses

- Semantic code search from natural-language queries
- Retrieval of code relevant to a requested edit
- Candidate generation for reranking or retrieval-augmented generation
- Code clustering, deduplication, and nearest-neighbor exploration

The model is a retriever, not a code generator or correctness verifier. A high
similarity score does not guarantee that code is safe, correct, or appropriate
for execution.

## Supported programming languages

The code-search training mixture includes:

`Bash`, `C`, `C++`, `C#`, `Dart`, `Go`, `Java`, `JavaScript`, `Kotlin`, `Lua`,
`PHP`, `Python`, `Ruby`, `Rust`, `Scala`, `Swift`, and `TypeScript`.

Performance may transfer to related languages, but languages not represented in
training have not been systematically validated.

## Training

The model was fine-tuned on 4,073,472 examples using knowledge-distilled hard
negative datasets for code search and code-edit retrieval. Each example
contained one positive and up to 15 explicit hard negatives; other positives in
the 1,024-example logical batch also served as in-batch negatives.

The three training groups were sampled at a `1:8:8` batch ratio:

| Training group | Weight | Purpose |
|---|---:|---|
| Code-edit retrieval | 1 | Natural-language edit request to relevant code |
| Core-language code search | 8 | Natural-language query to code |
| Additional-language code search | 8 | Broader programming-language coverage |

Each 17-batch weighted cycle was shuffled deterministically, rather than always
presenting the groups in a fixed order. Languages within each group were
balanced, and rows within each batch were shuffled.

### Training data and decontamination

The training mixture uses the same decontaminated retrieval datasets as
NightOwl-CodeEmbedding: code-search and code-comment pairs, together with
commitpackft-derived code-edit pairs. All examples were constructed with one
positive and 15 hard negatives mined by `Qwen/Qwen3-Embedding-0.6B`.

Before training, overlaps were removed between:

- The code-search/code-comment data and the CodeSearchNet test splits
- The commitpackft-derived code-edit data and the CodeEditSearchRetrieval
  benchmark evaluation examples

Although MTEB names the CodeEditSearchRetrieval evaluation split `train`, those
evaluated examples were not included in this model's fine-tuning data.

### Main hyperparameters

| Hyperparameter | Value |
|---|---:|
| Epochs | 1 |
| Logical batch size | 1,024 |
| Cached-loss mini-batch size | 64 |
| Learning rate | 6e-5 |
| Warmup ratio | 0.0 |
| Scheduler | Cosine |
| Weight decay | 0.01 |
| MNRL scale | 100.0 |
| Precision | bfloat16 |
| Gradient accumulation | 1 |
| Hard negatives per example | 15 |

## Evaluation

### MTEB CodeSearchNetRetrieval

The model was evaluated with MTEB 2.5.1 on the `CodeSearchNetRetrieval` test
sets. The benchmark's `main_score` is nDCG@10. The reported average is an
unweighted macro average over all six languages.

| Go | Java | JavaScript | PHP | Python | Ruby | Average |
|---:|---:|---:|---:|---:|---:|---:|
| 0.9656 | 0.9278 | 0.8236 | 0.8915 | 0.9410 | 0.8663 | **0.9026** |

### MTEB CodeEditSearchRetrieval

Code-edit retrieval was evaluated with MTEB 2.5.1. The task's `main_score` is
nDCG@10. The macro average across the 13 language subsets is **0.7382**.

| Language | nDCG@10 | Language | nDCG@10 |
|---|---:|---|---:|
| Python | 0.7711 | JavaScript | 0.7386 |
| TypeScript | 0.7759 | Go | 0.7716 |
| Ruby | 0.7723 | Java | 0.7365 |
| PHP | 0.7162 | C | 0.6687 |
| C++ | 0.7105 | Rust | 0.6900 |
| Swift | 0.7470 | Scala | 0.7859 |
| Shell | 0.7119 | **Macro average** | **0.7382** |

`CodeEditSearchRetrieval` does not provide a standard test split in MTEB, so its
official `train` split is used for evaluation. The evaluated examples were
removed from the fine-tuning data and were not used to train this model. The
score therefore measures in-domain retrieval on held-out benchmark examples;
it is not training-set performance or a strictly zero-shot result.

### In-training validation

The following results use cosine retrieval on 1,000 validation examples per
CodeSearchNet language. Each query has one relevant document, so Accuracy@k and
Recall@k are equivalent in this setup. The macro average is an unweighted mean
over the six evaluated languages.

| Language | Accuracy@1 | Accuracy@10 | nDCG@10 | MRR@10 |
|---|---:|---:|---:|---:|
| Go | 0.864 | 0.978 | 0.9270 | 0.9101 |
| Java | 0.722 | 0.935 | 0.8389 | 0.8070 |
| JavaScript | 0.745 | 0.891 | 0.8187 | 0.7954 |
| PHP | 0.753 | 0.819 | 0.7921 | 0.7828 |
| Python | 0.857 | 0.983 | 0.9278 | 0.9092 |
| Ruby | 0.824 | 0.944 | 0.8896 | 0.8716 |
| **Macro average** | **0.7942** | **0.9250** | **0.8657** | **0.8460** |

These numbers come from an in-training validation setup and should not be
treated as directly comparable to results produced with a different corpus,
candidate pool, preprocessing pipeline, or benchmark implementation.

## License

The model weights and original code in this repository are released under the Apache License 2.0.

The author's code-search training dataset was constructed from repositories licensed under MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, Unlicense, CC0-1.0, and ISC.

The Apache-2.0 license for this model does not supersede any third-party rights or license obligations that may apply to source materials used during training. Users are responsible for complying with applicable licenses when redistributing or reusing original source code obtained independently of the model.


## Limitations

* Evaluation above covers six CodeSearchNet languages, not every training language.
* Inputs longer than 1,024 tokens are truncated.
* The training data may contain public-code biases, duplicated patterns, or insecure implementations inherited from its sources.