| # FLES-2 v32 — Sparse Lexical Embeddings via Two-Pass Distillation | |
| **NDCG@10: 0.3119 | MRR: 0.5291 | NNZ: 420 (at eval) | Zero loss spikes** | |
| A sparse retrieval encoder that transforms text into interpretable, indexable sparse vectors using BERT's MLM predictions. Trained with a novel two-pass distillation methodology: ranking-heavy first (α=0.7), vocabulary-heavy second (α=0.3, low LR). | |
| ## Model Description | |
| FLES-2 v32 produces sparse vectors over a 30,522-dimensional vocabulary space (BERT WordPiece). Each dimension corresponds to a vocabulary term, and the weight indicates how strongly that term is predicted for the input text. The result is a bag-of-expanded-terms representation that can be indexed with standard inverted indices. | |
| ### Architecture | |
| ``` | |
| Text → BERT (bert-base-uncased) → MLM Head → log(1 + ReLU(logits)) → Max Pool → Sparse Vector | |
| ``` | |
| ### Training Methodology | |
| Two-pass sparse self-distillation from a frozen teacher (mindoval/fles1-v12b): | |
| 1. **Pass 1 (f2-v15):** Student=v7, Teacher=v12b, α=0.7 (ranking-dominant), lr=2e-5, 200K×2ep | |
| 2. **Pass 2 (f2-v32):** Student=f2-v15, Teacher=v12b, α=0.3 (vocabulary-dominant), lr=5e-6, 200K×2ep | |
| Key innovations: | |
| - Teacher thresholding (t=0.3) prevents density transfer | |
| - L1 FLOPS regularization (constant gradient, no density explosions) | |
| - Epoch-level CLFR (Closed-Loop FLOPS Regulation) for sparsity control | |
| - Zero loss spikes across all training (perfect stability) | |
| ## Usage | |
| ```python | |
| from fles1_encoder import FLES1Encoder | |
| # Load model | |
| encoder = FLES1Encoder.from_pretrained("mindoval/fles2-v32") | |
| # Encode text to sparse vector | |
| sparse_vec = encoder.encode("What is machine learning?") | |
| # Returns: {"machine": 1.82, "learning": 1.65, "artificial": 0.94, "intelligence": 0.87, ...} | |
| # Batch encode | |
| vectors = encoder.encode_batch(["query 1", "query 2"], batch_size=32) | |
| ``` | |
| ## Evaluation Results (nfcorpus) | |
| | Metric | Score | | |
| |--------|-------| | |
| | NDCG@10 | 0.3119 | | |
| | MRR | 0.5291 | | |
| | Recall@100 | 0.2456 | | |
| | Avg NNZ (non-zero terms) | 420 | | |
| | Loss spikes during training | 0 | | |
| ## Training Details | |
| | Parameter | Value | | |
| |-----------|-------| | |
| | Base model | bert-base-uncased | | |
| | Teacher | mindoval/fles1-v12b (frozen, thresholded at 0.3) | | |
| | Student init (pass 2) | mindoval/fles2-v15 | | |
| | Distillation loss | Sparse KL divergence | | |
| | Alpha (pass 2) | 0.3 (70% distillation, 30% ranking) | | |
| | Learning rate (pass 2) | 5e-6 | | |
| | FLOPS regularization | L1, λ_d=0.00003 | | |
| | Training data | 200K MS MARCO passages, 2 epochs | | |
| | Total steps (pass 2) | 12,500 | | |
| | Hardware | NVIDIA H100 NVL 80GB | | |
| | Training time (pass 2) | ~3.9 hours | | |
| ## Comparison | |
| | Model | NDCG@10 | NNZ | Method | | |
| |-------|---------|-----|--------| | |
| | **FLES-2 v32** | **0.3119** | **420** | **Two-pass sparse self-distillation** | | |
| | FLES-2 v15 | 0.3102 | 458 | Single-pass (α=0.7) | | |
| | FLES-1 v14 | 0.3049 | 359 | L1 FLOPS + epoch CLFR | | |
| | BM25 (Pyserini) | 0.325 | — | Unsupervised | | |
| | SPLADE-cocondenser | 0.340 | 125 | L2 FLOPS + cross-encoder distillation | | |
| ## Limitations | |
| - Evaluated only on nfcorpus (medical domain). Performance on other BEIR datasets may vary. | |
| - Gap to BM25 (0.013) and SPLADE (0.028) remains. The teacher (v12b) is the ceiling. | |
| - Sparse vectors are larger than dense (420 non-zero terms vs 768-dim dense). | |
| ## Citation | |
| ```bibtex | |
| @misc{fles2v32, | |
| title={FLES-2: Two-Pass Sparse Self-Distillation for Learned Sparse Retrieval}, | |
| author={Tavarez, Golvis}, | |
| year={2026}, | |
| publisher={Mindoval, Inc.}, | |
| url={https://huggingface.co/mindoval/fles2-v32} | |
| } | |
| ``` | |
| ## Acknowledgments | |
| Built by Mindoval, Inc. Training compute provided by Microsoft Corporation (Azure ML, H100 GPUs). | |
| ## License | |
| Apache 2.0 | |