| | --- |
| | license: mit |
| | tags: |
| | - feature-extraction |
| | language: en |
| | --- |
| | |
| | # PubMedNCL |
| |
|
| | A pretrained language model for document representations of biomedical papers. |
| | PubMedNCL is based on [PubMedBERT](https://huggingface.co/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext), which is a BERT model pretrained on abstracts and full-texts from PubMedCentral, and fine-tuned via citation neighborhood contrastive learning, as introduced by [SciNCL](https://huggingface.co/malteos/scincl). |
| |
|
| | ## How to use the pretrained model |
| |
|
| |
|
| | ```python |
| | from transformers import AutoTokenizer, AutoModel |
| | |
| | # load model and tokenizer |
| | tokenizer = AutoTokenizer.from_pretrained('malteos/PubMedNCL') |
| | model = AutoModel.from_pretrained('malteos/PubMedNCL') |
| | |
| | papers = [{'title': 'BERT', 'abstract': 'We introduce a new language representation model called BERT'}, |
| | {'title': 'Attention is all you need', 'abstract': ' The dominant sequence transduction models are based on complex recurrent or convolutional neural networks'}] |
| | |
| | # concatenate title and abstract with [SEP] token |
| | title_abs = [d['title'] + tokenizer.sep_token + (d.get('abstract') or '') for d in papers] |
| | |
| | # preprocess the input |
| | inputs = tokenizer(title_abs, padding=True, truncation=True, return_tensors="pt", max_length=512) |
| | |
| | # inference |
| | result = model(**inputs) |
| | |
| | # take the first token ([CLS] token) in the batch as the embedding |
| | embeddings = result.last_hidden_state[:, 0, :] |
| | ``` |
| |
|
| | ## Citation |
| |
|
| | - [Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings (EMNLP 2022 paper)](https://arxiv.org/abs/2202.06671). |
| | - [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://arxiv.org/abs/2007.15779). |
| |
|
| | ## License |
| |
|
| | MIT |
| |
|
| |
|
