---
license: apache-2.0
library_name: pytorch
tags:
- sentence-transformers
- feature-extraction
- sentence-similarity
- mteb
- hcae
---

# HCAE-21M (Hybrid Convolutional-Attention Encoder)
**HCAE-21M** is a mid-scale (21 Million parameters) text embedding model combining Depthwise Separable Convolutions and Self-Attention layers. It achieves high performance on Semantic Textual Similarity and Retrieval tasks while remaining extremely memory-efficient.



<img src="https://cdn-uploads.huggingface.co/production/uploads/680c9127408ea47e6c1dd6e8/0KKsVpqg2Id01nxh8zRjO.png" width="400">


## Architecture Description
- **Size:** ~21M parameters (d_model=384)
- **Lower Layers:** 5 layers of Depthwise Separable Conv1d + FFN.
- **Upper Layers:** 3 layers of Multihead Self-Attention.
- **Pooling Strategy:** Global Mean Pooling.



## Benchmark Comparison (MTEB)

This table delineates the performance disparities between architectural iterations:

| Model Revision | STSBenchmark (Spearman) | SciFact (Recall@10) | Description |
|---|---|---|---|
| **HCAE-21M-Base** | `0.507` | `0.324` | Baseline configuration trained extensively on the MS MARCO dataset. |
| **HCAE-21M-Instruct** | `0.591` | `0.393` | Multi-stage tuning incorporating ArXiv, STS-B, and SQuAD instruction tuning paradigms. |


 <p align="center">
  <img src="https://cdn-uploads.huggingface.co/production/uploads/680c9127408ea47e6c1dd6e8/VuJ6ayS--Ot8i-715fT3a.png" width="800" style="border-radius: 10px; box-shadow: 0 4px 20px rgba(0,0,0,0.3);">
</p>



## Utilization Guidelines (Instruction Format)
For optimal retrieval performance, prepend the instruction mapping to the query text:
`Instruction: Retrieve the exact document that answers the following question. Query: [Your Query]`