Feature Extraction
Transformers
English
retrieval
reasoning
embedding
BRIGHT
information-retrieval
Eval Results (legacy)
Instructions to use ForwardAILabs/MRE-T1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ForwardAILabs/MRE-T1 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="ForwardAILabs/MRE-T1")# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("ForwardAILabs/MRE-T1", dtype="auto") - Notebooks
- Google Colab
- Kaggle
fix: correct per-task scores to match no-instruction variant (avg 39.6/35.1)
Browse files
README.md
CHANGED
|
@@ -75,32 +75,32 @@ MRE-T1 achieves state-of-the-art single-model performance on the [BRIGHT benchma
|
|
| 75 |
|
| 76 |
| Task | MRE-T1 |
|
| 77 |
|------|--------|
|
| 78 |
-
| Biology |
|
| 79 |
-
| Earth Science |
|
| 80 |
-
| Economics |
|
| 81 |
-
| Psychology |
|
| 82 |
-
| Robotics |
|
| 83 |
-
| StackOverflow |
|
| 84 |
-
| Sustainable Living |
|
| 85 |
-
| LeetCode |
|
| 86 |
-
| Pony |
|
| 87 |
-
| AOPS |
|
| 88 |
-
| TheoremQA (Questions) |
|
| 89 |
-
| TheoremQA (Theorems) |
|
| 90 |
| **Average** | **39.6** |
|
| 91 |
|
| 92 |
### Long Document Retrieval (nDCG@10)
|
| 93 |
|
| 94 |
| Task | MRE-T1 |
|
| 95 |
|------|--------|
|
| 96 |
-
| Biology |
|
| 97 |
-
| Earth Science |
|
| 98 |
-
| Economics |
|
| 99 |
-
| Psychology |
|
| 100 |
-
| Robotics |
|
| 101 |
-
| StackOverflow |
|
| 102 |
-
| Sustainable Living |
|
| 103 |
-
| Pony |
|
| 104 |
| **Average** | **35.1** |
|
| 105 |
|
| 106 |
### Comparison with Other Models (Short, Single Model Only)
|
|
|
|
| 75 |
|
| 76 |
| Task | MRE-T1 |
|
| 77 |
|------|--------|
|
| 78 |
+
| Biology | 55.3 |
|
| 79 |
+
| Earth Science | 56.5 |
|
| 80 |
+
| Economics | 32.9 |
|
| 81 |
+
| Psychology | 48.2 |
|
| 82 |
+
| Robotics | 33.1 |
|
| 83 |
+
| StackOverflow | 34.2 |
|
| 84 |
+
| Sustainable Living | 37.3 |
|
| 85 |
+
| LeetCode | 35.0 |
|
| 86 |
+
| Pony | 35.5 |
|
| 87 |
+
| AOPS | 16.7 |
|
| 88 |
+
| TheoremQA (Questions) | 43.3 |
|
| 89 |
+
| TheoremQA (Theorems) | 46.9 |
|
| 90 |
| **Average** | **39.6** |
|
| 91 |
|
| 92 |
### Long Document Retrieval (nDCG@10)
|
| 93 |
|
| 94 |
| Task | MRE-T1 |
|
| 95 |
|------|--------|
|
| 96 |
+
| Biology | 74.2 |
|
| 97 |
+
| Earth Science | 72.2 |
|
| 98 |
+
| Economics | 57.3 |
|
| 99 |
+
| Psychology | 71.3 |
|
| 100 |
+
| Robotics | 51.6 |
|
| 101 |
+
| StackOverflow | 51.4 |
|
| 102 |
+
| Sustainable Living | 66.2 |
|
| 103 |
+
| Pony | 33.9 |
|
| 104 |
| **Average** | **35.1** |
|
| 105 |
|
| 106 |
### Comparison with Other Models (Short, Single Model Only)
|