File size: 5,491 Bytes
114e5b5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 |
---
license: apache-2.0
language:
- en
- ko
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- feature-extraction
- sentence-similarity
- transformers
- embedding
- gte
- text-embedding
- retrieval
- matryoshka
- academic-search
- scientific-search
library_name: transformers
base_model: Snowflake/snowflake-arctic-embed-m-v2.0
datasets:
- ms_marco
---
# LinerAI/snowflake-arctic-embed-m-v2.0-academic for Academic Search
This is a fine-tuned version of [Snowflake/snowflake-arctic-embed-m-v2.0](https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v2.0) optimized for **academic and scientific literature search**. The model has been trained using contrastive learning with hard negative mining, specifically curated for academic search scenarios.
## Highlights
- **Optimized for Academic Search**: Fine-tuned on datasets specifically designed for academic literature retrieval
- **Hard Negative Mining**: Trained with carefully mined hard negatives to improve discrimination between similar academic papers
- **Matryoshka Representation Learning (MRL)**: Supports flexible embedding dimensions (768, 512, 256, 128) for efficiency
- **Efficient & Fast**: Medium-sized model offering excellent speed-performance trade-off
- **Long Context**: Supports up to 4096 tokens
## Model Description
| Attribute | Value |
|-----------|-------|
| Base Model | Snowflake/snowflake-arctic-embed-m-v2.0 |
| Architecture | GTE |
| Embedding Dimension | 768 |
| MRL Dimensions | 768, 512, 256, 128 |
| Max Sequence Length | 4096 |
| Pooling | CLS token |
| Precision | float16 |
## Evaluation Results
| **Model** | Avg. | SciFact: Recall@10 | TRECCOVID: Recall@10 | NFCorpus: Recall@10 | SCIDOCS: Recall@10 | LitSearch: Recall@10 | QASA: Recall@10 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| snowflake-arctic-embed-m-v2.0-academic | 0.3729 | 0.8609 | 0.0219 | 0.177 | 0.2129 | 0.6435 | 0.321 |
| snowflake-arctic-embed-m-v2.0 | 0.3654 | 0.8353 | 0.0224 | 0.1669 | 0.2122 | 0.6508 | 0.3046 |
## Training Details
### Training Configuration
| Parameter | Value |
|-----------|-------|
| Learning Rate | 2e-5 |
| Batch Size | 8192 (effective) |
| Per-Device Batch Size | 32 |
| Warmup Steps | 100 |
| Weight Decay | 0.1 |
| Precision | fp16 |
| Max Length | 4096 |
| Loss Function | InfoNCE (Contrastive) |
| Temperature (τ) | 0.02 |
### Training Data
The model was trained on [LEAD (Liner Embedding Academic Dataset)](https://huggingface.co/datasets/LinerAI/LEAD), a combination of ~55,560 samples tailored for academic search:
- **MS MARCO** (49%): General passage retrieval dataset with hard negatives
- **Academic Search Dataset** (51%): Custom dataset built specifically for academic literature search, with two-stage hard negative mining
### Matryoshka Representation Learning (MRL)
This model supports [Matryoshka Representation Learning](https://arxiv.org/abs/2205.13147). You can truncate embeddings to smaller dimensions (512, 256, 128) for faster computation and reduced storage.
```python
# Full dimension (768)
full_embedding = embeddings[:, :768]
# MRL dimensions
embedding_512 = embeddings[:, :512]
embedding_256 = embeddings[:, :256]
embedding_128 = embeddings[:, :128]
```
## Usage
### Using Transformers
```python
import torch
from transformers import AutoModel, AutoTokenizer
model_path = "LinerAI/snowflake-arctic-embed-l-v2.0-academic"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, torch_dtype=torch.float16, trust_remote_code=True)
model.eval()
# For queries
def encode_query(text):
input_text = f"query: {text}"
inputs = tokenizer(input_text, return_tensors="pt", max_length=4096, truncation=True)
with torch.no_grad():
outputs = model(**inputs)
embeddings = outputs.last_hidden_state[:, 0] # CLS token
embeddings = torch.nn.functional.normalize(embeddings, p=2, dim=1)
return embeddings
# For passages
def encode_passage(text):
inputs = tokenizer(text, return_tensors="pt", max_length=4096, truncation=True)
with torch.no_grad():
outputs = model(**inputs)
embeddings = outputs.last_hidden_state[:, 0] # CLS token
embeddings = torch.nn.functional.normalize(embeddings, p=2, dim=1)
return embeddings
# Example: Academic search
query = "transformer models for protein structure prediction"
abstract = "We introduce AlphaFold, a deep learning system that predicts protein structures..."
query_emb = encode_query(query)
passage_emb = encode_passage(abstract)
similarity = torch.nn.functional.cosine_similarity(query_emb, passage_emb)
print(f"Similarity: {similarity.item():.4f}")
```
## Input Format
### Query Format
```
query: {your_query_text}
```
### Passage Format
```
{your_passage_text}
```
## Intended Use
- **Academic Paper Search**: Finding relevant research papers given a research query
- **Literature Review**: Discovering related work in academic literature
- **Scientific Document Retrieval**: Retrieving scientific documents, abstracts, and articles
- **Research Question Answering**: Finding papers that address specific research questions
## Limitations
- Maximum sequence length is 4096 tokens
- Best performance achieved when using the specific input formats described above
- The model uses asymmetric encoding (query prefix for queries, no prefix for passages)
- Requires `trust_remote_code=True` for loading
## License
This model is released under the Apache 2.0 license.
|