tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- information-retrieval
- code-search
- code-embedding
- dense-retrieval
- modernbert
- generated_from_trainer
- dataset_size:4073472
- loss:CachedMultipleNegativesRankingLoss
base_model: Shuu12121/NightJar
pipeline_tag: feature-extraction
library_name: sentence-transformers
license: apache-2.0
language:
- en
metrics:
- ndcg_at_10
- cosine_accuracy@1
- cosine_accuracy@10
- cosine_ndcg@10
- cosine_mrr@10
model-index:
- name: NightJar-CodeSearch-Embedding
results:
- task:
type: information-retrieval
name: CodeSearchNet Retrieval
dataset:
name: CodeSearchNet (6-language macro average)
type: code-search-net
metrics:
- type: ndcg_at_10
value: 0.9026
name: nDCG@10 (6-language macro average)
- task:
type: information-retrieval
name: Code Edit Search Retrieval
dataset:
name: CodeEditSearchRetrieval (13-language macro average)
type: code-edit-search
metrics:
- type: ndcg_at_10
value: 0.7381646153846154
name: nDCG@10 (13-language macro average)
NightJar-CodeSearch-Embedding
NightJar-CodeSearch-Embedding is a 768-dimensional dense embedding model for natural-language-to-code retrieval and code-edit retrieval. It is based on Shuu12121/NightJar and fine-tuned with hard negatives and large in-batch negatives.
The model embeds a text query and a code document into the same vector space. Higher cosine similarity indicates a stronger match. It does not require query or document prefixes.
Model details
| Property | Value |
|---|---|
| Architecture | ModernBERT Sentence Transformer |
| Parameters | Same transformer size as Shuu12121/NightJar |
| Embedding dimensions | 768 |
| Maximum sequence length | 1,024 tokens |
| Pooling | CLS token |
| Similarity | Cosine similarity |
| Training objective | Cached Multiple Negatives Ranking Loss |
| License | Apache-2.0 |
Usage
Sentence Transformers
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("Shuu12121/NightJar-CodeSearch-Embedding")
query = "parse a JSON string and return an error when it is invalid"
code = [
"def parse_json(text):\n return json.loads(text)",
"def read_lines(path):\n return Path(path).read_text().splitlines()",
]
query_embedding = model.encode(query, normalize_embeddings=True)
code_embeddings = model.encode(code, normalize_embeddings=True)
scores = code_embeddings @ query_embedding
for score, snippet in sorted(zip(scores, code), reverse=True):
print(float(score), snippet)
No query:, passage:, or task-specific prefix is needed. Use the same model
and encoding settings for queries and code. Normalized embeddings make the dot
product equivalent to cosine similarity.
Transformers
Loading through SentenceTransformer is recommended because the repository's
pooling configuration is part of the model. If the backbone is loaded directly
with Transformers, use the final hidden state of the CLS token and L2-normalize
the resulting vector.
Intended uses
- Semantic code search from natural-language queries
- Retrieval of code relevant to a requested edit
- Candidate generation for reranking or retrieval-augmented generation
- Code clustering, deduplication, and nearest-neighbor exploration
The model is a retriever, not a code generator or correctness verifier. A high similarity score does not guarantee that code is safe, correct, or appropriate for execution.
Supported programming languages
The code-search training mixture includes:
Bash, C, C++, C#, Dart, Go, Java, JavaScript, Kotlin, Lua,
PHP, Python, Ruby, Rust, Scala, Swift, and TypeScript.
Performance may transfer to related languages, but languages not represented in training have not been systematically validated.
Training
The model was fine-tuned on 4,073,472 examples using knowledge-distilled hard negative datasets for code search and code-edit retrieval. Each example contained one positive and up to 15 explicit hard negatives; other positives in the 1,024-example logical batch also served as in-batch negatives.
The three training groups were sampled at a 1:8:8 batch ratio:
| Training group | Weight | Purpose |
|---|---|---|
| Code-edit retrieval | 1 | Natural-language edit request to relevant code |
| Core-language code search | 8 | Natural-language query to code |
| Additional-language code search | 8 | Broader programming-language coverage |
Each 17-batch weighted cycle was shuffled deterministically, rather than always presenting the groups in a fixed order. Languages within each group were balanced, and rows within each batch were shuffled.
Training data and decontamination
The training mixture uses the same decontaminated retrieval datasets as
NightOwl-CodeEmbedding: code-search and code-comment pairs, together with
commitpackft-derived code-edit pairs. All examples were constructed with one
positive and 15 hard negatives mined by Qwen/Qwen3-Embedding-0.6B.
Before training, overlaps were removed between:
- The code-search/code-comment data and the CodeSearchNet test splits
- The commitpackft-derived code-edit data and the CodeEditSearchRetrieval benchmark evaluation examples
Although MTEB names the CodeEditSearchRetrieval evaluation split train, those
evaluated examples were not included in this model's fine-tuning data.
Main hyperparameters
| Hyperparameter | Value |
|---|---|
| Epochs | 1 |
| Logical batch size | 1,024 |
| Cached-loss mini-batch size | 64 |
| Learning rate | 6e-5 |
| Warmup ratio | 0.0 |
| Scheduler | Cosine |
| Weight decay | 0.01 |
| MNRL scale | 100.0 |
| Precision | bfloat16 |
| Gradient accumulation | 1 |
| Hard negatives per example | 15 |
Evaluation
MTEB CodeSearchNetRetrieval
The model was evaluated with MTEB 2.5.1 on the CodeSearchNetRetrieval test
sets. The benchmark's main_score is nDCG@10. The reported average is an
unweighted macro average over all six languages.
| Go | Java | JavaScript | PHP | Python | Ruby | Average |
|---|---|---|---|---|---|---|
| 0.9656 | 0.9278 | 0.8236 | 0.8915 | 0.9410 | 0.8663 | 0.9026 |
MTEB CodeEditSearchRetrieval
Code-edit retrieval was evaluated with MTEB 2.5.1. The task's main_score is
nDCG@10. The macro average across the 13 language subsets is 0.7382.
| Language | nDCG@10 | Language | nDCG@10 |
|---|---|---|---|
| Python | 0.7711 | JavaScript | 0.7386 |
| TypeScript | 0.7759 | Go | 0.7716 |
| Ruby | 0.7723 | Java | 0.7365 |
| PHP | 0.7162 | C | 0.6687 |
| C++ | 0.7105 | Rust | 0.6900 |
| Swift | 0.7470 | Scala | 0.7859 |
| Shell | 0.7119 | Macro average | 0.7382 |
CodeEditSearchRetrieval does not provide a standard test split in MTEB, so its
official train split is used for evaluation. The evaluated examples were
removed from the fine-tuning data and were not used to train this model. The
score therefore measures in-domain retrieval on held-out benchmark examples;
it is not training-set performance or a strictly zero-shot result.
In-training validation
The following results use cosine retrieval on 1,000 validation examples per CodeSearchNet language. Each query has one relevant document, so Accuracy@k and Recall@k are equivalent in this setup. The macro average is an unweighted mean over the six evaluated languages.
| Language | Accuracy@1 | Accuracy@10 | nDCG@10 | MRR@10 |
|---|---|---|---|---|
| Go | 0.864 | 0.978 | 0.9270 | 0.9101 |
| Java | 0.722 | 0.935 | 0.8389 | 0.8070 |
| JavaScript | 0.745 | 0.891 | 0.8187 | 0.7954 |
| PHP | 0.753 | 0.819 | 0.7921 | 0.7828 |
| Python | 0.857 | 0.983 | 0.9278 | 0.9092 |
| Ruby | 0.824 | 0.944 | 0.8896 | 0.8716 |
| Macro average | 0.7942 | 0.9250 | 0.8657 | 0.8460 |
These numbers come from an in-training validation setup and should not be treated as directly comparable to results produced with a different corpus, candidate pool, preprocessing pipeline, or benchmark implementation.
License
The model weights and original code in this repository are released under the Apache License 2.0.
The author's code-search training dataset was constructed from repositories licensed under MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, Unlicense, CC0-1.0, and ISC.
The Apache-2.0 license for this model does not supersede any third-party rights or license obligations that may apply to source materials used during training. Users are responsible for complying with applicable licenses when redistributing or reusing original source code obtained independently of the model.
Limitations
- Evaluation above covers six CodeSearchNet languages, not every training language.
- Inputs longer than 1,024 tokens are truncated.
- The training data may contain public-code biases, duplicated patterns, or insecure implementations inherited from its sources.