|
|
--- |
|
|
language: en |
|
|
license: mit |
|
|
library_name: transformers |
|
|
tags: |
|
|
- bert |
|
|
- scientific-text |
|
|
- embeddings |
|
|
- fine-tuned |
|
|
pipeline_tag: feature-extraction |
|
|
--- |
|
|
|
|
|
# scibert-citation-model |
|
|
|
|
|
This model is a fine-tuned version of SciBERT specifically optimized for generating embeddings from scientific papers. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- **Base Model**: SciBERT (Scientific BERT) |
|
|
- **Fine-tuning Task**: Scientific paper understanding and embedding generation |
|
|
- **Language**: English (Scientific/Academic) |
|
|
- **Vocabulary**: Scientific vocabulary |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
from transformers import AutoTokenizer, AutoModel |
|
|
import torch |
|
|
|
|
|
# Load model and tokenizer |
|
|
tokenizer = AutoTokenizer.from_pretrained("your-username/scibert-citation-model") |
|
|
model = AutoModel.from_pretrained("your-username/scibert-citation-model") |
|
|
|
|
|
# Generate embeddings |
|
|
text = "Your scientific text here" |
|
|
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512) |
|
|
with torch.no_grad(): |
|
|
outputs = model(**inputs) |
|
|
embeddings = outputs.last_hidden_state[:, 0, :] # [CLS] token embedding |
|
|
|
|
|
print(f"Embeddings shape: {embeddings.shape}") |
|
|
``` |
|
|
|
|
|
## Performance |
|
|
|
|
|
Fine-tuned SciBert |
|
|
|
|
|
## Training Details |
|
|
|
|
|
- **Training Framework**: PyTorch/Transformers |
|
|
- **Fine-tuning Objective**: Scientific text understanding |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you use this model in your research, please cite appropriately. |
|
|
|
