Upload README.md with huggingface_hub

README.md

@@ -1,3 +1,124 @@
 ---
 license: mit
+base_model: nomic-ai/CodeRankEmbed
+base_model_relation: quantized
+tags:
+- code
+- embeddings
+- onnx
+- int8
+- quantized
+language:
+- code
+pipeline_tag: feature-extraction
 ---
+
+# CodeRankEmbed — Dynamic INT8 Quantized (ONNX)
+
+A dynamically quantized INT8 version of [nomic-ai/CodeRankEmbed](https://huggingface.co/nomic-ai/CodeRankEmbed), converted to ONNX by [jalipalo](https://huggingface.co/jalipalo/CodeRankEmbed-onnx) and quantized for fast CPU inference.
+
+## What is this?
+
+CodeRankEmbed is a 137M-parameter embedding model trained specifically for code search and retrieval. This repository provides a **dynamic INT8 weight-quantized** version that is significantly smaller and faster with negligible quality loss:
+
+| | FP32 (original) | INT8 (this model) |
+|---|---|---|
+| **File size** | 522 MB | 132 MB (−75%) |
+| **CPU inference** | 1.00× | ~2.09× faster |
+| **Min cosine vs FP32** | 1.000 | 0.961 |
+| **Calibration data needed** | — | None |
+
+Quantization was done with ONNX Runtime's `quantize_dynamic` (weights only, `QInt8`, `per_channel=True`). Activations remain in FP32 at runtime — the recommended approach for transformer/embedding models per the [ONNX Runtime documentation](https://onnxruntime.ai/docs/performance/model-optimizations/quantization.html).
+
+## Usage
+
+### With `@huggingface/transformers` (JavaScript / Node.js)
+
+```js
+import { pipeline } from "@huggingface/transformers";
+
+const extractor = await pipeline(
+  "feature-extraction",
+  "mrsladoje/CodeRankEmbed-onnx-int8",
+  { quantized: true }   // loads onnx/model_quantized.onnx automatically
+);
+
+const output = await extractor("def hello(): return 42", {
+  pooling: "mean",
+  normalize: true,
+});
+console.log(output.data); // Float32Array of 768 dimensions
+```
+
+### With `optimum` (Python)
+
+```python
+from optimum.onnxruntime import ORTModelForFeatureExtraction
+from transformers import AutoTokenizer
+
+model = ORTModelForFeatureExtraction.from_pretrained(
+    "mrsladoje/CodeRankEmbed-onnx-int8",
+    file_name="onnx/model_quantized.onnx",
+)
+tokenizer = AutoTokenizer.from_pretrained("mrsladoje/CodeRankEmbed-onnx-int8")
+
+inputs = tokenizer("def hello(): return 42", return_tensors="pt")
+outputs = model(**inputs)
+embeddings = outputs.last_hidden_state.mean(dim=1)
+```
+
+### With `onnxruntime` directly (Python)
+
+```python
+import onnxruntime as ort
+import numpy as np
+from tokenizers import Tokenizer
+
+tokenizer = Tokenizer.from_pretrained("mrsladoje/CodeRankEmbed-onnx-int8")
+tokenizer.enable_padding(length=128, pad_id=0)
+tokenizer.enable_truncation(max_length=128)
+
+session = ort.InferenceSession("onnx/model_quantized.onnx")
+
+encoded = tokenizer.encode("def hello(): return 42")
+input_ids = np.array([encoded.ids], dtype=np.int64)
+attention_mask = np.array([encoded.attention_mask], dtype=np.int64)
+
+outputs = session.run(None, {"input_ids": input_ids, "attention_mask": attention_mask})
+embedding = outputs[1]  # sentence_embedding output, shape (1, 768)
+```
+
+## Quantization Details
+
+| Parameter | Value |
+|---|---|
+| Method | `quantize_dynamic` (ONNX Runtime) |
+| Weight type | `QInt8` (signed 8-bit integer) |
+| Scope | Weights only — activations quantized dynamically at runtime |
+| Per-channel | Yes |
+| Calibration | None required |
+| ORT version | 1.21.x |
+
+**Why dynamic over static?** Static INT8 quantization requires calibration data to pre-compute activation ranges. For transformer embedding models, activation distributions vary widely with input content and sequence length, making static calibration brittle (we validated this — static QDQ produced cosine similarities as low as 0.09–0.26 with MinMax calibration). Dynamic quantization sidesteps this entirely: weights are quantized offline and activations are quantized at runtime, giving robust quality across all inputs.
+
+## Quality Validation
+
+Validated on 10 code snippets across Python, JavaScript, Go, Java, Rust, TypeScript, and SQL:
+
+```
+Model            Size      Speedup    Min cosine vs FP32    Quality
+FP32 (baseline)  522.3 MB  1.00×      —                     baseline
+Dynamic INT8     132.2 MB  2.09×      0.9610                 excellent
+```
+
+A cosine similarity ≥ 0.96 means the INT8 embeddings point in essentially the same direction as FP32. For retrieval tasks — especially with a reranker in the pipeline — this difference is undetectable in practice.
+
+The ~2× CPU speedup is real compute acceleration (not just faster file loading), coming from ONNX Runtime's `MatMulIntegerToFloat` fused kernels operating on INT8 weights. VNNI-capable CPUs (Intel 10th gen+, AMD Zen4+) may see even larger gains.
+
+## Attribution
+
+- **Original model:** [nomic-ai/CodeRankEmbed](https://huggingface.co/nomic-ai/CodeRankEmbed) — MIT License
+- **ONNX conversion:** [jalipalo/CodeRankEmbed-onnx](https://huggingface.co/jalipalo/CodeRankEmbed-onnx) — MIT License (inherited)
+- **INT8 quantization:** this repository — MIT License
+
+All work in this repository respects and complies with the MIT license of the original model.