Image-Text-to-Text
Transformers
Safetensors
qwen3_5
vllm
video
multimodal
reinforcement-learning
temporal-grounding
object-tracking
video-segmentation
visual-question-answering
spatial-reasoning
qwen3.5
conversational
Instructions to use OraRL/Video-ORA-4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use OraRL/Video-ORA-4B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="OraRL/Video-ORA-4B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("OraRL/Video-ORA-4B") model = AutoModelForMultimodalLM.from_pretrained("OraRL/Video-ORA-4B", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use OraRL/Video-ORA-4B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "OraRL/Video-ORA-4B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "OraRL/Video-ORA-4B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/OraRL/Video-ORA-4B
- SGLang
How to use OraRL/Video-ORA-4B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "OraRL/Video-ORA-4B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "OraRL/Video-ORA-4B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "OraRL/Video-ORA-4B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "OraRL/Video-ORA-4B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use OraRL/Video-ORA-4B with Docker Model Runner:
docker model run hf.co/OraRL/Video-ORA-4B
| set -euo pipefail | |
| SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" | |
| ORARL_ROOT="$(cd -- "${SCRIPT_DIR}/.." && pwd)" | |
| ENV_NAME="${ORARL_CONDA_ENV:-orarl}" | |
| PYTORCH_INDEX="https://download.pytorch.org/whl/cu129" | |
| EVALUATION_ONLY=0 | |
| usage() { | |
| cat <<'EOF' | |
| Usage: bash scripts/create_conda_env.sh [options] | |
| Create the pinned OraRL CUDA 12.9 environment. This checkout ships both the | |
| training runtime and the evaluators, so no external runtime is required. Use | |
| --evaluation-only to skip the training-side environment validation. | |
| Options: | |
| --name ENV_NAME Conda environment name (default: orarl). | |
| --evaluation-only Validate the environment for evaluation only. | |
| EOF | |
| } | |
| while [[ $# -gt 0 ]]; do | |
| case "$1" in | |
| --name) | |
| if [[ $# -lt 2 || -z "$2" ]]; then | |
| echo "ERROR: --name requires a non-empty environment name." >&2 | |
| exit 2 | |
| fi | |
| ENV_NAME="$2" | |
| shift 2 | |
| ;; | |
| --evaluation-only) | |
| EVALUATION_ONLY=1 | |
| shift | |
| ;; | |
| -h|--help) | |
| usage | |
| exit 0 | |
| ;; | |
| *) | |
| echo "ERROR: unknown argument: $1" >&2 | |
| usage >&2 | |
| exit 2 | |
| ;; | |
| esac | |
| done | |
| if ! command -v conda >/dev/null 2>&1; then | |
| echo "ERROR: conda is not available in PATH." >&2 | |
| exit 1 | |
| fi | |
| if [[ ! -d "${ORARL_ROOT}/verl" ]]; then | |
| echo "ERROR: the bundled training runtime is missing at ${ORARL_ROOT}/verl." >&2 | |
| echo "Clone the full repository instead of copying individual directories." >&2 | |
| exit 1 | |
| fi | |
| if command -v nvidia-smi >/dev/null 2>&1; then | |
| echo "Detected cluster GPU(s):" | |
| nvidia-smi --query-gpu=name,driver_version --format=csv,noheader || true | |
| echo "CUDA 12.9 GA officially requires Linux driver 575.51.03 or newer." | |
| else | |
| echo "WARNING: nvidia-smi is unavailable; run the GPU check on an allocated H20 node." >&2 | |
| fi | |
| if conda run -n "${ENV_NAME}" python -c "pass" >/dev/null 2>&1; then | |
| echo "ERROR: conda environment '${ENV_NAME}' already exists." >&2 | |
| echo "Choose another name with --name or remove/update it explicitly." >&2 | |
| exit 1 | |
| fi | |
| conda env create \ | |
| --name "${ENV_NAME}" \ | |
| --file "${ORARL_ROOT}/environment.yml" | |
| run_in_env() { | |
| conda run --no-capture-output -n "${ENV_NAME}" "$@" | |
| } | |
| run_in_env python -m pip install --upgrade \ | |
| "pip==26.0.1" \ | |
| "setuptools==82.0.1" \ | |
| "wheel==0.46.3" | |
| # Install the CUDA build explicitly. Generic PyPI resolves PyTorch 2.10 to the | |
| # CUDA 12.8 wheel, which is not the stack validated with vLLM 0.19.1 here. | |
| run_in_env python -m pip install \ | |
| "torch==2.10.0+cu129" \ | |
| "torchvision==0.25.0+cu129" \ | |
| "torchaudio==2.10.0+cu129" \ | |
| --index-url "${PYTORCH_INDEX}" | |
| # PyTorch must be importable before building FlashAttention. | |
| run_in_env python -m pip install \ | |
| "flash-attn==2.8.3" \ | |
| --no-build-isolation | |
| run_in_env python -m pip install \ | |
| "vllm==0.19.1" \ | |
| --extra-index-url "${PYTORCH_INDEX}" | |
| run_in_env python -m pip install \ | |
| --requirement "${ORARL_ROOT}/requirements-cu129.txt" | |
| # One editable install covers the CLIs, the trainer runtime, and the evaluators. | |
| # --no-deps keeps it from replacing the GPU stack pinned above. | |
| run_in_env python -m pip install --no-deps --editable "${ORARL_ROOT}" | |
| run_in_env python -m pip install "pytest" "ruff" | |
| run_in_env bash "${ORARL_ROOT}/scripts/install_conda_runtime_hook.sh" | |
| CHECK_ARGUMENTS=() | |
| if [[ "${EVALUATION_ONLY}" -eq 1 ]]; then | |
| CHECK_ARGUMENTS+=(--evaluation-only) | |
| fi | |
| run_in_env bash -c \ | |
| 'source "${CONDA_PREFIX}/etc/conda/activate.d/orarl-runtime.sh"; shift; exec python "$@"' \ | |
| _ "${ORARL_ROOT}/scripts/check_environment.py" "${CHECK_ARGUMENTS[@]}" | |
| cat <<EOF | |
| OraRL environment created successfully. | |
| conda activate ${ENV_NAME} | |
| python ${ORARL_ROOT}/scripts/check_environment.py --require-gpu | |
| EOF | |