docs/local-vlm.md

# Local VLM Selection

## Chosen Model

Use `Qwen/Qwen2.5-VL-32B-Instruct-AWQ` as the primary 30B-class VLM for sentence mode and Hugging Face Space experiments.

Why this model fits BridgeLink ASL:

- it is the practical 32B AWQ quantized target from the Qwen2.5-VL collection
- it supports visual and video-style inputs
- it is designed for long-video/event understanding, which matters for ASL sentence windows
- it can return structured text that we can parse into sentence events
- it can consume sampled keyframes plus the current word-level token trace

## Hardware Reality And Fallbacks

The 32B AWQ model is still large. It is more realistic than 72B, but it may still require paid Hugging Face Space hardware, quantization-friendly inference, or a strong local GPU setup.

Fallback order:

- `Qwen/Qwen2.5-VL-32B-Instruct-AWQ` as the current default
- `Qwen/Qwen2.5-VL-7B-Instruct` if the team needs a laptop-friendly emergency path
- `Qwen/Qwen2.5-VL-72B-Instruct` as a high-hardware stretch comparison if available
- `MockSentenceInterpreter` for tests and no-network/no-model demos

## Integration Plan

Start with a provider interface:

```text
SentenceInterpreter
-> MockSentenceInterpreter
-> LocalQwen25VlmInterpreter
```

The mock interpreter should always work without model downloads. The local Qwen2.5-VL interpreter should be enabled only when the model is installed and configured.

Install the optional local VLM stack with:

```bash
pip install -e ".[vlm]"
```

The VLM path is now compared directly against the CNN baseline. It should consume the same clip manifest records as the CNN branch, then produce sentence-level JSON that the comparison wrapper can score.

## Prompting Rule

The local VLM should receive both:

- sampled frames from a short gesture window
- the current classifier token trace, including labels, confidence, and timestamps

The model should return strict JSON:

```json
{
  "gloss": ["HELLO", "WANT", "DRINK"],
  "sentence": "Hello, I want a drink.",
  "confidence": 0.82,
  "needs_clarification": false
}
```

If the VLM output is invalid or low confidence, the pipeline should fall back to a simple gloss sentence from the token trace.

For isolated-sign hybrid evaluation rows, the wrapper also accepts the
`video_id` / `true_label` / `cnn_top5` JSONL format already produced by the
Colab notebook. In that mode, the local VLM chooses one label from the CNN's
candidate list while the CNN baseline remains the top-1 candidate.

## Practical Demo Note

Download the model before presentation day. The final demo should not depend on internet access. Also keep a recorded-clip fallback because 32B local inference can still be too slow for a live webcam demo on weak hardware.