Update README.md
Browse files
README.md
CHANGED
|
@@ -10,7 +10,6 @@ language:
|
|
| 10 |
|
| 11 |
  
|
| 12 |
|
| 13 |
-
|
| 14 |
## Model Details
|
| 15 |
|
| 16 |
Socrates-embedding is a lightweight, high-density text embedding model. Unlike contemporary models that rely on massive parameter counts to brute-force semantic understanding, Socrates-embedding leverages Low-Rank Decay (LoRD) to achieve high-quality vector representations with minimal computational overhead.
|
|
@@ -21,6 +20,53 @@ This model is part of the Chunjiang Intelligence edge-computing initiative, aimi
|
|
| 21 |
- **Model Type:** Dual-Encoder Transformer.
|
| 22 |
- **Language:** English.
|
| 23 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
## Model Architecture
|
| 25 |
|
| 26 |
The model utilizes a custom Transformer Encoder architecture optimized for inference latency on Apple MPS and NVIDIA TensorRT backends.
|
|
|
|
| 10 |
|
| 11 |
  
|
| 12 |
|
|
|
|
| 13 |
## Model Details
|
| 14 |
|
| 15 |
Socrates-embedding is a lightweight, high-density text embedding model. Unlike contemporary models that rely on massive parameter counts to brute-force semantic understanding, Socrates-embedding leverages Low-Rank Decay (LoRD) to achieve high-quality vector representations with minimal computational overhead.
|
|
|
|
| 20 |
- **Model Type:** Dual-Encoder Transformer.
|
| 21 |
- **Language:** English.
|
| 22 |
|
| 23 |
+
|
| 24 |
+
The model was evaluated on the `AmazonCounterfactualClassification` dataset across multiple languages.
|
| 25 |
+
|
| 26 |
+
| Language | Accuracy |
|
| 27 |
+
| :--- | :---: |
|
| 28 |
+
| Japanese (ja) | 54.83 |
|
| 29 |
+
| German (de) | 52.57 |
|
| 30 |
+
| English (en) | 49.70 |
|
| 31 |
+
| English-Ext (en-ext)| 49.15 |
|
| 32 |
+
|
| 33 |
+
To put the model's efficiency into perspective, we compare its single-task score on Japanese classification against the *overall MTEB average scores* of much larger models. (Our budget is insufficient to cover the bill for the GPU used for the ongoing tests.)
|
| 34 |
+
|
| 35 |
+
<br>
|
| 36 |
+
|
| 37 |
+
<p align="center">
|
| 38 |
+
<img src="model_efficiency_comparison.png" width="800">
|
| 39 |
+
<br>
|
| 40 |
+
<em>Figure 1: Our 83M model's score on a single challenging task rivals the average performance of models up to 85x larger.</em>
|
| 41 |
+
</p>
|
| 42 |
+
|
| 43 |
+
<br>
|
| 44 |
+
|
| 45 |
+
Clustering performance was evaluated using the V-measure score (multiplied by 100) on the `StackExchangeClustering` task.
|
| 46 |
+
|
| 47 |
+
We compared Socrates-embedding against other popular lightweight models (<110M params).
|
| 48 |
+
|
| 49 |
+
| Model | Parameters | Clustering Score (V-measure x 100) |
|
| 50 |
+
| :--- | :--- | :---: |
|
| 51 |
+
| **Socrates-embedding** | **83M** | **8.92** 🏆 |
|
| 52 |
+
| `snowflake-arctic-embed-m`| 109M | 7.25 |
|
| 53 |
+
| `KartonBERT-USE-base-v1` | 104M | 6.93 |
|
| 54 |
+
| `jina-embedding-s-en-v1`| 35M | 6.64 |
|
| 55 |
+
|
| 56 |
+
| `all-MiniLM-L6-v2` | 23M | 6.62 |
|
| 57 |
+
|
| 58 |
+
* Observation: Our model achieves the highest clustering score in its weight class, demonstrating a superior vector space structure compared to established baselines.
|
| 59 |
+
|
| 60 |
+
<br>
|
| 61 |
+
|
| 62 |
+
<p align="center">
|
| 63 |
+
<img src="model_clustering_comparison.png" width="800">
|
| 64 |
+
<br>
|
| 65 |
+
<em>Figure 2: Leading clustering performance among lightweight embedding models.</em>
|
| 66 |
+
</p>
|
| 67 |
+
|
| 68 |
+
<br>
|
| 69 |
+
|
| 70 |
## Model Architecture
|
| 71 |
|
| 72 |
The model utilizes a custom Transformer Encoder architecture optimized for inference latency on Apple MPS and NVIDIA TensorRT backends.
|