README.md

---
license: mit
base_model: nomic-ai/CodeRankEmbed
base_model_relation: quantized
tags:
- code
- embeddings
- onnx
- int8
- quantized
language:
- code
pipeline_tag: feature-extraction
---

# CodeRankEmbed — Dynamic INT8 Quantized (ONNX)

A dynamically quantized INT8 version of [nomic-ai/CodeRankEmbed](https://huggingface.co/nomic-ai/CodeRankEmbed), converted to ONNX by [jalipalo](https://huggingface.co/jalipalo/CodeRankEmbed-onnx) and quantized for fast CPU inference.

## What is this?

CodeRankEmbed is a 137M-parameter embedding model trained specifically for code search and retrieval. This repository provides a **dynamic INT8 weight-quantized** version that is significantly smaller and faster with negligible quality loss:

| | FP32 (original) | INT8 (this model) |
|---|---|---|
| **File size** | 522 MB | 132 MB (−75%) |
| **CPU inference** | 1.00× | ~2.09× faster |
| **Min cosine vs FP32** | 1.000 | 0.961 |
| **Calibration data needed** | — | None |

Quantization was done with ONNX Runtime's `quantize_dynamic` (weights only, `QInt8`, `per_channel=True`). Activations remain in FP32 at runtime — the recommended approach for transformer/embedding models per the [ONNX Runtime documentation](https://onnxruntime.ai/docs/performance/model-optimizations/quantization.html).

## Usage

### With `@huggingface/transformers` (JavaScript / Node.js)

```js
import { pipeline } from "@huggingface/transformers";

const extractor = await pipeline(
  "feature-extraction",
  "mrsladoje/CodeRankEmbed-onnx-int8",
  { quantized: true }   // loads onnx/model_quantized.onnx automatically
);

const output = await extractor("def hello(): return 42", {
  pooling: "mean",
  normalize: true,
});
console.log(output.data); // Float32Array of 768 dimensions
```

### With `optimum` (Python)

```python
from optimum.onnxruntime import ORTModelForFeatureExtraction
from transformers import AutoTokenizer

model = ORTModelForFeatureExtraction.from_pretrained(
    "mrsladoje/CodeRankEmbed-onnx-int8",
    file_name="onnx/model_quantized.onnx",
)
tokenizer = AutoTokenizer.from_pretrained("mrsladoje/CodeRankEmbed-onnx-int8")

inputs = tokenizer("def hello(): return 42", return_tensors="pt")
outputs = model(**inputs)
embeddings = outputs.last_hidden_state.mean(dim=1)
```

### With `onnxruntime` directly (Python)

```python
import onnxruntime as ort
import numpy as np
from tokenizers import Tokenizer

tokenizer = Tokenizer.from_pretrained("mrsladoje/CodeRankEmbed-onnx-int8")
tokenizer.enable_padding(length=128, pad_id=0)
tokenizer.enable_truncation(max_length=128)

session = ort.InferenceSession("onnx/model_quantized.onnx")

encoded = tokenizer.encode("def hello(): return 42")
input_ids = np.array([encoded.ids], dtype=np.int64)
attention_mask = np.array([encoded.attention_mask], dtype=np.int64)

outputs = session.run(None, {"input_ids": input_ids, "attention_mask": attention_mask})
embedding = outputs[1]  # sentence_embedding output, shape (1, 768)
```

## Quantization Details

| Parameter | Value |
|---|---|
| Method | `quantize_dynamic` (ONNX Runtime) |
| Weight type | `QInt8` (signed 8-bit integer) |
| Scope | Weights only — activations quantized dynamically at runtime |
| Per-channel | Yes |
| Calibration | None required |
| ORT version | 1.21.x |

**Why dynamic over static?** Static INT8 quantization requires calibration data to pre-compute activation ranges. For transformer embedding models, activation distributions vary widely with input content and sequence length, making static calibration brittle (we validated this — static QDQ produced cosine similarities as low as 0.09–0.26 with MinMax calibration). Dynamic quantization sidesteps this entirely: weights are quantized offline and activations are quantized at runtime, giving robust quality across all inputs.

## Quality Validation

Validated on 10 code snippets across Python, JavaScript, Go, Java, Rust, TypeScript, and SQL:

```
Model            Size      Speedup    Min cosine vs FP32    Quality
FP32 (baseline)  522.3 MB  1.00×      —                     baseline
Dynamic INT8     132.2 MB  2.09×      0.9610                 excellent
```

A cosine similarity ≥ 0.96 means the INT8 embeddings point in essentially the same direction as FP32. For retrieval tasks — especially with a reranker in the pipeline — this difference is undetectable in practice.

The ~2× CPU speedup is real compute acceleration (not just faster file loading), coming from ONNX Runtime's `MatMulIntegerToFloat` fused kernels operating on INT8 weights. VNNI-capable CPUs (Intel 10th gen+, AMD Zen4+) may see even larger gains.

## Attribution

- **Original model:** [nomic-ai/CodeRankEmbed](https://huggingface.co/nomic-ai/CodeRankEmbed) — MIT License
- **ONNX conversion:** [jalipalo/CodeRankEmbed-onnx](https://huggingface.co/jalipalo/CodeRankEmbed-onnx) — MIT License (inherited)
- **INT8 quantization:** this repository — MIT License

All work in this repository respects and complies with the MIT license of the original model.