How to use from
SGLangUse Docker images
docker run --gpus all \
--shm-size 32g \
-p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=<secret>" \
--ipc=host \
lmsysorg/sglang:latest \
python3 -m sglang.launch_server \
--model-path "fava-uw/fava-model" \
--host 0.0.0.0 \
--port 30000# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
-H "Content-Type: application/json" \
--data '{
"model": "fava-uw/fava-model",
"prompt": "Once upon a time,",
"max_tokens": 512,
"temperature": 0.5
}'Quick Links
FAVA, a verification model.
import torch
import vllm
from transformers import AutoTokenizer, AutoModelForSequenceClassification
model = vllm.LLM(model="fava-uw/fava-model")
sampling_params = vllm.SamplingParams(
temperature=0,
top_p=1.0,
max_tokens=1024,
)
INPUT = "Read the following references:\n{evidence}\nPlease identify all the errors in the following text using the information in the references provided and suggest edits if necessary:\n[Text] {output}\n[Edited] "
output = "" # add your passage to verify
evidence = "" # add a piece of evidence
prompts = [INPUT.format_map({"evidence": evidence, "output": output})]
outputs = model.generate(prompts, sampling_params)
outputs = [it.outputs[0].text for it in outputs]
print(outputs[0])
- Downloads last month
- 77
Install from pip and serve model
# Install SGLang from pip: pip install sglang# Start the SGLang server: python3 -m sglang.launch_server \ --model-path "fava-uw/fava-model" \ --host 0.0.0.0 \ --port 30000# Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "fava-uw/fava-model", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'