README.md

---
license: cc-by-4.0
language:
- en
library_name: transformers
pipeline_tag: video-text-to-text
tags:
- video-captioning
- multimodal
- gemma
- parakeet
---

# Sulphur Caption

This is a captioning model I made purely to make things easier and cheaper for myself. If you don't like how the model works or it doesn't work on your favorite runtime, I'm sorry, but I'm not going to put any further effort into this, as it works how I want it to. If someone wants to figure out gguf or anything like that, you're welcome to try, you can dm me on discord under the username `kiwv`. Also it's worth noting this model will not be used in sulphur 3's dataset creation.

You're going to need ~70gb vram to run this model. The fp8 variant will require ~38gb.

Note the fp8 variant is ~1.6x faster than the bf16 variant.

Also this model isn't SUPER great, it's just solid. I wouldn't rely on this model solely, if you're using it to produce captions for a lora, look at the captions, or pass in tags, or something.

I was planning to release this model at a better time, but it's been sitting here so long I just decided to release it. I would probably change some stuff if I redid this model now but whatever.

## Installation

You need `ffmpeg`, `ffprobe`, and `unzip` available on `PATH`.

```bash
cd /wherever/you/downloaded/this/thing
unzip -q vllm.zip
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip setuptools wheel
pip install -r requirements.txt
```

Or run the helper:

```bash
cd /wherever/you/downloaded/this/thing
./setup_vllm.sh
source .venv/bin/activate
```

## Single Video Inference

Edit `VIDEO_PATH` in `example_run.py`:

```python
VIDEO_PATH = "/path/to/video.mp4"
```

Then run:

```bash
cd /path/to/this/repo
source .venv/bin/activate
python example_run.py
```

The script prints the raw model output.

## Folder Captioning

Edit these variables in `caption_folder.py`:

```python
INPUT_FOLDER = "/path/to/videos"
OUTPUT_FOLDER = "/path/to/captions"
```

Then run:

```bash
cd /path/to/this/repo
source .venv/bin/activate
python caption_folder.py
```

The script mirrors the input folder structure into `OUTPUT_FOLDER` and writes one `.txt` file per video. It loads the model once and batches videos using `BATCH_SIZE`.

## Prompt Settings

The model was trained with some settings, grouped under `Prompt Settings`. Here are the recommended values:

- `HAS_THINKING` | True
- `CAPTION_LENGTH` | "very large"
- `INCLUDE_WATERMARK_INFO` | True
- `VULGARITY` | "low"
- `UNCERTAINTY` | "low"
- `CHARACTER_NAMES` | "none"
- `FLUFF` | "none"
- `HAS_REPETITION` | False
- `SPECULATION` | "low"
- `TEMPORAL_DETAIL` | "medium"
- `VISUAL_SPECIFICITY` | "moderate"
- `CAMERA_DETAIL` | "medium"
- `CAPTION_STYLE` | "plain"

### Possible Options

- `HAS_THINKING` | True, False
- `CAPTION_LENGTH` | "very small", "small", "medium", "large", "very large"
- `INCLUDE_WATERMARK_INFO` | True, False
- `VULGARITY` | "none", "low", "medium", "high"
- `UNCERTAINTY` | "none", "low", "medium", "high"
- `CHARACTER_NAMES` | "none", "ambiguous", "single", "multiple"
- `FLUFF` | "none", "low", "medium", "high"
- `HAS_REPETITION` | True, False
- `SPECULATION` | "none", "low", "medium", "high"
- `TEMPORAL_DETAIL` | "static", "low", "medium", "high"
- `VISUAL_SPECIFICITY` | "generic", "moderate", "detailed", "excessive"
- `CAMERA_DETAIL` | "none", "low", "medium", "high"
- `CAPTION_STYLE` | "plain", "verbose", "ornate", "robotic"

Generation settings are grouped under `vLLM / Generation Hyperparameters`:

- `NUM_FRAMES` | (was trained between 4 and 20)
- `MAX_MODEL_LEN`
- `MAX_NUM_SEQS`
- `BATCH_SIZE` in `caption_folder.py`
- `GPU_MEMORY_UTILIZATION`
- `MAX_TOKENS`
- `TEMPERATURE` | (recommend a lower value)
- `TOP_P`
- `REPETITION_PENALTY`

## Performance

This graph shows measured vLLM decode throughput on an RTX PRO 6000 using 12 video frames, `max_tokens = 96`, `max_model_len = 4096`, prefix cache disabled, and unique requests:

### BF16

![Decode Throughput BF16](assets/decode_throughput.png)

### FP8

![Decode Throughput FP8](assets/fp8_decode_throughput.png)

## Limitations

- This model still makes plenty of mistakes on very complex videos.
- It requires the bundled patched vLLM runtime for the vLLM examples.
- Audio captioning depends on the Parakeet bridge and may still miss unclear, quiet, overlapping, or non-speech sounds.

## Credits
- Gemma 4 31b for main model
- Parakeet 0.6b TDT v3 for audio