Instructions to use ncauchi1/general_questions_model_v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ncauchi1/general_questions_model_v1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="ncauchi1/general_questions_model_v1")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("ncauchi1/general_questions_model_v1")
model = AutoModelForMultimodalLM.from_pretrained("ncauchi1/general_questions_model_v1")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ncauchi1/general_questions_model_v1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ncauchi1/general_questions_model_v1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ncauchi1/general_questions_model_v1",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/ncauchi1/general_questions_model_v1

SGLang

How to use ncauchi1/general_questions_model_v1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ncauchi1/general_questions_model_v1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ncauchi1/general_questions_model_v1",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ncauchi1/general_questions_model_v1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ncauchi1/general_questions_model_v1",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use ncauchi1/general_questions_model_v1 with Docker Model Runner:
```
docker model run hf.co/ncauchi1/general_questions_model_v1
```

Model Card for Model ID

Second version of VLLM fine tuned to answer general questions about cyclic voltammographs. Evaluated on bxw315-umd/general-cv-questions

Uses

Used to answer general multiple choice questions about cyclic voltammogram graphs

Training Details

Trained on ncauchi1/general_questions_dataset with 12k samples. Logs found here: [https://wandb.ai/ncauchi-university-of-maryland/huggingface/runs/i3455num/logs]

Training dataset consists of a mix of data:

10k Pointing samples

Given graphs with a range 2-4 CVs and points out all peaks in voltage/currents
graphs do not have legends

2k Question samples

Given question, graph, and four options
gives reasononing and correct answer
graphs are catagorized into templates and questions/reasoning are generated based on graph template
Graphs are generated from raw data gathered by me, consisting of CV's of Ferrocene and Tryptophan in PBS with concentrations of 0uM, 100uM and 200uM.

Evaluation

35% ± 5 chance to answer correct Evaluation done on bxw315-umd/general-cv-questions, with an 15% increase in performance over base model (35% chance to answer correct)

Error analysis reveals trying to identify features and connecting them to reasoning. Error analysis shows model still has trouble percieving graphs correctly