HAKARI-Bench: A Lightweight Benchmark for Comparing Retrieval Architectures and Efficiency Settings under Unified Conditions

Evaluating and selecting the right retrieval configuration for RAG or semantic search has become notoriously difficult. While heavy-scale benchmarks like MTEB/MMTEB are comprehensive, they are too heavy to run repeatedly during iterative development. Furthermore, production-time settings—such as embedding dimensionality reduction, quantization, and reranking—are rarely reported under identical, model-agnostic conditions.

To bridge this gap, I built HAKARI-Bench—a lightweight evaluation infrastructure that reconstructs existing retrieval benchmarks into shrunken "Nano-sets" (spanning 35 benchmarks, 43 languages, and 551 tasks).

Despite its lightweight data footprint, it acts as a high-fidelity ranking proxy, reproducing the official MTEB retrieval v2, MMTEB v2 retrieval, and English BEIR (full) rankings at a Spearman rank correlation of > 0.97.

🛠️ Why it's a game-changer for practical IR/RAG development:

• 5 Retrieval Families & Realistic 2-Stage Shuffling: Evaluate BM25, Dense, Sparse (SPLADE), Late Interaction (ColBERT), and Rerankers under a unified format. To accurately simulate real-world production setups, Rerankers are evaluated over a fixed, reproducible hybrid candidate set fusing BM25 and Dense via Reciprocal Rank Fusion (RRF Top 100). This decouples candidate-generation failures from pure reranking accuracy.

• Mapping the Quality-Efficiency Pareto Frontier: We applied post-encoding efficiency transformations uniformly across 33 dense models. You can instantly see how much quality you sacrifice when compressing embeddings to save storage, memory, and latency:

• Int8 is nearly free: Scalar int8 quantization causes a minor, uniform drop (mean -1.95 points).

• Binary is model-specific, but fixable: Binary quantization severely collapses some models (e.g., the multilingual-E5 family dropping up to -35.8 points) but spares others (Jina v5, Gemma, Arctic, Qwen3). Crucially, adding a simple float rescore on the top-100 candidates almost completely restores the original rankings (reducing the binary drop to a mere -0.93 points).

• Sparse Pruning Knobs: For SPLADE-style models, document-side pruning is a "cheap knob" (lossless down to 256 active dimensions), whereas query-side pruning is an "expensive knob" that degrades quality sharply below 16 dimensions.

• Fully Open-Source with a DuckDB Analytics Backend: We provide a multi-axis leaderboard/viewer alongside the pipeline. Because the backend stores all runs in a DuckDB warehouse, rendering is lightning-fast, and you can easily slice the data via SQL to filter by language, domain, context length, or parameter counts.

HAKARI-Bench is not a replacement for full-scale evaluation; rather, it is designed to supercharge your model selection, regression testing, and production cost-optimization under a single unified harness.

• Spaces Leaderboard: https://huggingface.co/spaces/hakari-bench/leaderboard

• GitHub Repository: https://github.com/hakari-bench/hakari-bench