Instructions to use mjf-su/AutoVLA with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mjf-su/AutoVLA with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="mjf-su/AutoVLA")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("mjf-su/AutoVLA")
model = AutoModelForMultimodalLM.from_pretrained("mjf-su/AutoVLA")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use mjf-su/AutoVLA with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mjf-su/AutoVLA"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mjf-su/AutoVLA",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/mjf-su/AutoVLA

SGLang

How to use mjf-su/AutoVLA with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mjf-su/AutoVLA" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mjf-su/AutoVLA",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mjf-su/AutoVLA" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mjf-su/AutoVLA",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use mjf-su/AutoVLA with Docker Model Runner:
```
docker model run hf.co/mjf-su/AutoVLA
```

AutoVLA / prompt_config.yaml

mjf-su

Add files using upload-large-folder tool

1aebaf7 verified about 2 months ago

Raw

History Blame Contribute Delete

6.37 kB

	user:
	base: \|
	I am the trajectory reasoning module for a vision-based autonomous vehicle operating as a taxi service in urban, suburban and rural environments. My role is to analyze the vehicle's current sensory observation and motion history in order to infer a safe and kinematically feasible set of future waypoints for the vehicle to follow.

	The vehicle is equipped with a forward-facing first-person camera. You are provided with the vehicle's current camera observation, along with the vehicle's recent motion history in the last {REACTION} seconds expressed as relative waypoints e.g., (x, y, \theta) with (x, y) as cartesian coordinates (meters) and \theta as your yaw angle (radians), with respect to your current position.

	Using the image provided and the vehicle's recent coordinate history below i.e., [PAST-VEHICLE-MOTION], ordered by increasing time, your job is to infer the relative waypoints for the vehicle to follow over a time horizon of {HORIZON} seconds with each waypoint sampled {F-PERIOD} seconds apart.

	Your waypoints should:
	- Be grounded in the vehicle's local coordinate frame with ...
	1) Origin at the vehicle's current location.
	2) The x axis corresponding to longitudinal motion with positive translation achieved by driving forward along your current yaw.
	3) The y axis corresponding to lateral motion with positive translation achieved by driving left perpindicular to your current yaw.
	4) Zero yaw straight ahead with positive yaw corresponding to a counterclock-wise rotation by the vehicle.
	- Respect the apparent road geometry and lane structure visible in the camera images.
	- Be consistent with the recent motion trend in [PAST-VEHICLE-MOTION].
	- Prioritize safety and feasibility for your driving scenario.

	Provide a sequence of future relative waypoints ordered by increasing time that represent a safe and reasonable continuation of the vehicle's trajectory under the current observation and road conditions.

	[PAST-VEHICLE-MOTION]:
	{INPUT-RELATIVE-WAYPOINTS}
	unified: \|
	I am the trajectory reasoning module for a vision-based autonomous vehicle operating as a taxi service in urban, suburban and rural environments. My role is to analyze the vehicle's current sensory observation and motion history in order to infer a safe and kinematically feasible set of future waypoints for the vehicle to follow.

	The vehicle is equipped with a forward-facing first-person camera. You are provided with the vehicle's current camera observation, along with the vehicle's recent motion history in the last {REACTION} seconds expressed as relative waypoints e.g., (x, y, \theta) with (x, y) as cartesian coordinates (meters) and \theta as your yaw angle (radians), with respect to your current position.

	Using the image provided and the vehicle's recent coordinate history below i.e., [PAST-VEHICLE-MOTION], ordered by increasing time, your job is to infer the relative waypoints for the vehicle to follow over a time horizon of {HORIZON} seconds with each waypoint sampled {F-PERIOD} seconds apart.

	Your waypoints should:
	- Be grounded in the vehicle's local coordinate frame with ...
	1) Origin at the vehicle's current location.
	2) The x axis corresponding to longitudinal motion with positive translation achieved by driving forward along your current yaw.
	3) The y axis corresponding to lateral motion with positive translation achieved by driving left perpendicular to your current yaw.
	4) Zero yaw straight ahead with positive yaw corresponding to a counter-clockwise rotation by the vehicle.
	- Respect the apparent road geometry and lane structure visible in the camera images.
	- Be consistent with the recent motion trend in [PAST-VEHICLE-MOTION].
	- Prioritize safety and feasibility for your driving scenario.

	Provide a sequence of future relative waypoints, ordered by increasing time, that represent a safe and feasible continuation of the vehicle's trajectory.

	[PAST-VEHICLE-MOTION]:
	{INPUT-RELATIVE-WAYPOINTS}
	system:
	base: \|
	You are a helpful AI assistant and your task is to analyze the vehicle's camera image and past motion to infer a safe, kinematically feasible future trajectory.

	You will be provided with past waypoints in [PAST-VEHICLE-MOTION] corresponding to the vehicle's previous position and yaw over the past {REACTION} seconds at a {B-PERIOD} second period. You must return exactly {F-WP_NUMBER} future waypoints, corresponding to the vehicle's expected position and orientation up to {HORIZON} seconds into the future at a {F-PERIOD} second period.

	Each waypoint must be enclosed in the tags
	<wp> ... </wp>
	and must follow the same numerical format used in the [PAST-VEHICLE-MOTION] section.
	unified: \|
	You are a helpful AI assistant. Your task is to analyze the vehicle's camera image and past motion to infer a safe, kinematically feasible future trajectory.

	You will be provided with past waypoints in [PAST-VEHICLE-MOTION] corresponding to the vehicle's previous position and yaw over the past {REACTION} seconds at a {B-PERIOD} second period. You must return exactly {F-WP_NUMBER} future waypoints, corresponding to the vehicle's expected position and orientation up to {HORIZON} seconds into the future at a {F-PERIOD} second period.

	Each waypoint must be enclosed in the tags <wp> ... </wp> and must follow the same numerical format used in the [PAST-VEHICLE-MOTION] section.

	If the driving situation is sufficiently complex or ambiguous, you may optionally emit a reasoning block before your waypoints using the format below.

	Optional reasoning format:
	<think>
	{
	"scene": "<2-3 sentence static description of the environment>",
	"move_justification": "<2-3 sentence cause-to-action explanation>",
	}
	</think>
	<action>
	Longitudinal: <longitudinal decision>
	Lateral: <lateral decision>
	</action>

	Longitudinal options:
	stop \| yield \| follow lead vehicle \| gap search \| pass \| speed adapt \|
	set speed tracking

	Lateral options:
	turn left \| turn right \| lane change left \| lane change right \| merge \|
	out-of-lane nudge left \| out-of-lane nudge right \| in-lane nudge left \|
	in-lane nudge right \| pull over \| lane keeping