Instructions to use PatronusAI/glider with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use PatronusAI/glider with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="PatronusAI/glider", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("PatronusAI/glider", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("PatronusAI/glider", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use PatronusAI/glider with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "PatronusAI/glider"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PatronusAI/glider",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/PatronusAI/glider

SGLang

How to use PatronusAI/glider with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "PatronusAI/glider" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PatronusAI/glider",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "PatronusAI/glider" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PatronusAI/glider",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use PatronusAI/glider with Docker Model Runner:
```
docker model run hf.co/PatronusAI/glider
```

DarshanDeshpande commited on Dec 19, 2024

Commit

ee47592

verified ·

1 Parent(s): 5303f4b

Update README.md

Browse files

Files changed (1) hide show

README.md +16 -6

README.md CHANGED Viewed

@@ -17,8 +17,10 @@ pipeline_tag: text-generation
 # Patronus GLIDER
 GLIDER is a fine tuned phi-3.5-mini-instruct which can be used as a general purpose evaluation model to judge texts, conversations and RAG setups according to arbitrary, user defined criteria and rubric scale.
-This model was trained using a combination of synthetic and domain adapted data from popular datasets like Mocha, FinQA, Realtoxicity, etc.  The training data for this model covers over 183 metrics and 683+ domains including finance, medicine, and many more.
 The maximum sequence length is 8192 tokens but the model can support longer texts as well (tested upto 12,000 tokens).
@@ -27,14 +29,14 @@ The maximum sequence length is 8192 tokens but the model can support longer text
 - **Model Type:** GLIDER is a fine-tuned version of microsoft/Phi-3.5-mini-instruct model.
 - **Language:** Primarily English but supports Korean, Kazakh, Hindi, Bengali, Spanish, Indonesian, German, French, Arabic, Russian, Thai, Turkish, Ukraninan, Romainian and more.
 - **Developed by:** Patronus AI
-- **Paper:** [TBD]
 - **License:** [https://creativecommons.org/licenses/by-nc/4.0/](https://creativecommons.org/licenses/by-nc/4.0/)
 ### Model Sources
 <!-- Provide the basic links for the model. -->
-- **Repository:** [https://github.com/patronus-ai/slm-evaluator](https://github.com/patronus-ai/slm-evaluator)
 ## How to Get Started with the Model
@@ -73,6 +75,7 @@ Your output must in the following format:
 <score>
 [The final integer score assigned based on the scoring rubric]
 </score>
 ```
 Since the model supports arbitrary number of inputs and outputs, the data can be structured in any one of the following ways:
@@ -134,7 +137,6 @@ Note that these XML tags can be changed according to your convenience and task
 To run inference, you can use HF pipeline:
 ```
 model_name = 'PatronusAI/glider'
 pipe = pipeline(
           "text-generation",
@@ -159,13 +161,21 @@ Since the model is trained in chat format, ensure that you pass the prompt as a
 The model was evaluated on several popular datasets:
-<img src="https://i.imgur.com/PuW1DzP.png" alt="Results" width="70%"/>
 ## Citation
 If you are using the model, cite using
 ```
-[Paper citation]
 ```
 ## Model Card Contact

 # Patronus GLIDER
+<img src="https://i.imgur.com/1AbgTJa.png" alt="GLIDER" width="100%"/>
 GLIDER is a fine tuned phi-3.5-mini-instruct which can be used as a general purpose evaluation model to judge texts, conversations and RAG setups according to arbitrary, user defined criteria and rubric scale.
+This model was trained using a combination of synthetic and domain adapted data from popular datasets like Mocha, FinQA, Realtoxicity, etc.  The training data for this model covers over 183 metrics and 685 domains including finance, medicine, and many more.
 The maximum sequence length is 8192 tokens but the model can support longer texts as well (tested upto 12,000 tokens).
 - **Model Type:** GLIDER is a fine-tuned version of microsoft/Phi-3.5-mini-instruct model.
 - **Language:** Primarily English but supports Korean, Kazakh, Hindi, Bengali, Spanish, Indonesian, German, French, Arabic, Russian, Thai, Turkish, Ukraninan, Romainian and more.
 - **Developed by:** Patronus AI
+- **Paper:** [https://arxiv.org/abs/2412.14140](https://arxiv.org/abs/2412.14140)
 - **License:** [https://creativecommons.org/licenses/by-nc/4.0/](https://creativecommons.org/licenses/by-nc/4.0/)
 ### Model Sources
 <!-- Provide the basic links for the model. -->
+- **Repository:** [https://github.com/patronus-ai/glider](https://github.com/patronus-ai/glider)
 ## How to Get Started with the Model
 <score>
 [The final integer score assigned based on the scoring rubric]
 </score>
+"""
 ```
 Since the model supports arbitrary number of inputs and outputs, the data can be structured in any one of the following ways:
 To run inference, you can use HF pipeline:
 ```
 model_name = 'PatronusAI/glider'
 pipe = pipeline(
           "text-generation",
 The model was evaluated on several popular datasets:
+<img src="https://i.imgur.com/77lhcwf.png" alt="Results" width="100%"/>
 ## Citation
 If you are using the model, cite using
 ```
+@misc{deshpande2024glider,
+      title={GLIDER: Grading LLM Interactions and Decisions using Explainable Ranking},
+      author={Darshan Deshpande and Selvan Sunitha Ravi and Sky CH-Wang and Bartosz Mielczarek and Anand Kannappan and Rebecca Qian},
+      year={2024},
+      eprint={2412.14140},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2412.14140},
+}
 ```
 ## Model Card Contact