Instructions to use RichardErkhov/deepset_-_roberta-base-squad2-covid-8bits with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RichardErkhov/deepset_-_roberta-base-squad2-covid-8bits with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="RichardErkhov/deepset_-_roberta-base-squad2-covid-8bits")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("RichardErkhov/deepset_-_roberta-base-squad2-covid-8bits")
model = AutoModelForCausalLM.from_pretrained("RichardErkhov/deepset_-_roberta-base-squad2-covid-8bits", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RichardErkhov/deepset_-_roberta-base-squad2-covid-8bits with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RichardErkhov/deepset_-_roberta-base-squad2-covid-8bits"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RichardErkhov/deepset_-_roberta-base-squad2-covid-8bits",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/RichardErkhov/deepset_-_roberta-base-squad2-covid-8bits

SGLang

How to use RichardErkhov/deepset_-_roberta-base-squad2-covid-8bits with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RichardErkhov/deepset_-_roberta-base-squad2-covid-8bits" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RichardErkhov/deepset_-_roberta-base-squad2-covid-8bits",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RichardErkhov/deepset_-_roberta-base-squad2-covid-8bits" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RichardErkhov/deepset_-_roberta-base-squad2-covid-8bits",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use RichardErkhov/deepset_-_roberta-base-squad2-covid-8bits with Docker Model Runner:
```
docker model run hf.co/RichardErkhov/deepset_-_roberta-base-squad2-covid-8bits
```

deepset_-_roberta-base-squad2-covid-8bits / README.md

RichardErkhov

uploaded readme

0e5f350 verified about 2 years ago

preview code

Raw

History Blame Contribute Delete

4.86 kB

	Quantization made by Richard Erkhov.

	[Github](https://github.com/RichardErkhov)

	[Discord](https://discord.gg/pvy7H8DZMG)

	[Request more models](https://github.com/RichardErkhov/quant_request)


	roberta-base-squad2-covid - bnb 8bits
	- Model creator: https://huggingface.co/deepset/
	- Original model: https://huggingface.co/deepset/roberta-base-squad2-covid/




	Original model description:
	---
	language: en
	datasets:
	- squad_v2
	license: cc-by-4.0
	---

	# roberta-base-squad2 for QA on COVID-19

	## Overview
	Language model: deepset/roberta-base-squad2
	Language: English
	Downstream-task: Extractive QA
	Training data: [SQuAD-style CORD-19 annotations from 23rd April](https://github.com/deepset-ai/COVID-QA/blob/master/data/question-answering/200423_covidQA.json)
	Code: See [an example QA pipeline on Haystack](https://haystack.deepset.ai/tutorials/01_basic_qa_pipeline)
	Infrastructure: Tesla v100

	## Hyperparameters
	```
	batch_size = 24
	n_epochs = 3
	base_LM_model = "deepset/roberta-base-squad2"
	max_seq_len = 384
	learning_rate = 3e-5
	lr_schedule = LinearWarmup
	warmup_proportion = 0.1
	doc_stride = 128
	xval_folds = 5
	dev_split = 0
	no_ans_boost = -100
	```
	---
	license: cc-by-4.0
	---

	## Performance
	5-fold cross-validation on the data set led to the following results:

	Single EM-Scores: [0.222, 0.123, 0.234, 0.159, 0.158]
	Single F1-Scores: [0.476, 0.493, 0.599, 0.461, 0.465]
	Single top\\_3\\_recall Scores: [0.827, 0.776, 0.860, 0.771, 0.777]
	XVAL EM: 0.17890995260663506
	XVAL f1: 0.49925444207319924
	XVAL top\\_3\\_recall: 0.8021327014218009

	This model is the model obtained from the third fold of the cross-validation.

	## Usage

	### In Haystack
	For doing QA at scale (i.e. many docs instead of single paragraph), you can load the model also in [haystack](https://github.com/deepset-ai/haystack/):
	```python
	reader = FARMReader(model_name_or_path="deepset/roberta-base-squad2-covid")
	# or
	reader = TransformersReader(model="deepset/roberta-base-squad2",tokenizer="deepset/roberta-base-squad2-covid")
	```

	### In Transformers
	```python
	from transformers import AutoModelForQuestionAnswering, AutoTokenizer, pipeline


	model_name = "deepset/roberta-base-squad2-covid"

	# a) Get predictions
	nlp = pipeline('question-answering', model=model_name, tokenizer=model_name)
	QA_input = {
	'question': 'Why is model conversion important?',
	'context': 'The option to convert models between FARM and transformers gives freedom to the user and let people easily switch between frameworks.'
	}
	res = nlp(QA_input)

	# b) Load model & tokenizer
	model = AutoModelForQuestionAnswering.from_pretrained(model_name)
	tokenizer = AutoTokenizer.from_pretrained(model_name)
	```

	## Authors
	Branden Chan: branden.chan@deepset.ai
	Timo Möller: timo.moeller@deepset.ai
	Malte Pietsch: malte.pietsch@deepset.ai
	Tanay Soni: tanay.soni@deepset.ai
	Bogdan Kostić: bogdan.kostic@deepset.ai

	## About us
	<div class="grid lg:grid-cols-2 gap-x-4 gap-y-3">
	<div class="w-full h-40 object-cover mb-2 rounded-lg flex items-center justify-center">
	<img alt="" src="https://raw.githubusercontent.com/deepset-ai/.github/main/deepset-logo-colored.png" class="w-40"/>
	</div>
	<div class="w-full h-40 object-cover mb-2 rounded-lg flex items-center justify-center">
	<img alt="" src="https://raw.githubusercontent.com/deepset-ai/.github/main/haystack-logo-colored.png" class="w-40"/>
	</div>
	</div>

	[deepset](http://deepset.ai/) is the company behind the open-source NLP framework [Haystack](https://haystack.deepset.ai/) which is designed to help you build production ready NLP systems that use: Question answering, summarization, ranking etc.


	Some of our other work:
	- [Distilled roberta-base-squad2 (aka "tinyroberta-squad2")]([https://huggingface.co/deepset/tinyroberta-squad2)
	- [German BERT (aka "bert-base-german-cased")](https://deepset.ai/german-bert)
	- [GermanQuAD and GermanDPR datasets and models (aka "gelectra-base-germanquad", "gbert-base-germandpr")](https://deepset.ai/germanquad)

	## Get in touch and join the Haystack community

	<p>For more info on Haystack, visit our <strong><a href="https://github.com/deepset-ai/haystack">GitHub</a></strong> repo and <strong><a href="https://docs.haystack.deepset.ai">Documentation</a></strong>.

	We also have a <strong><a class="h-7" href="https://haystack.deepset.ai/community/join">Discord community open to everyone!</a></strong></p>

	[Twitter](https://twitter.com/deepset_ai) \| [LinkedIn](https://www.linkedin.com/company/deepset-ai/) \| [Discord](https://haystack.deepset.ai/community) \| [GitHub Discussions](https://github.com/deepset-ai/haystack/discussions) \| [Website](https://deepset.ai)

	By the way: [we're hiring!](http://www.deepset.ai/jobs)