File size: 4,447 Bytes
88a8eb1 a5a8b9b 7011765 1ccf7b6 7011765 1ccf7b6 7011765 1ccf7b6 88a8eb1 7011765 89f739a 7011765 23c5a52 18e3867 369bd2b 7011765 260fb83 f7d15f7 49b402b 80a7ad8 7011765 e2712fc 7011765 e2712fc 7011765 e2712fc 7011765 9c753fd 7011765 baebe31 54f1de5 7011765 54f1de5 7011765 54f1de5 7011765 e91b871 7011765 54f1de5 7011765 fb7d5bd | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 | ---
license: cc-by-4.0
language:
- en
library_name: transformers
pipeline_tag: video-text-to-text
tags:
- video-captioning
- multimodal
- gemma
- parakeet
---
# Sulphur Caption
This is a captioning model I made purely to make things easier and cheaper for myself. If you don't like how the model works or it doesn't work on your favorite runtime, I'm sorry, but I'm not going to put any further effort into this, as it works how I want it to. If someone wants to figure out gguf or anything like that, you're welcome to try, you can dm me on discord under the username `kiwv`. Also it's worth noting this model will not be used in sulphur 3's dataset creation.
You're going to need ~70gb vram to run this model. The fp8 variant will require ~38gb.
Note the fp8 variant is ~1.6x faster than the bf16 variant.
Also this model isn't SUPER great, it's just solid. I wouldn't rely on this model solely, if you're using it to produce captions for a lora, look at the captions, or pass in tags, or something.
I was planning to release this model at a better time, but it's been sitting here so long I just decided to release it. I would probably change some stuff if I redid this model now but whatever.
## Installation
You need `ffmpeg`, `ffprobe`, and `unzip` available on `PATH`.
```bash
cd /wherever/you/downloaded/this/thing
unzip -q vllm.zip
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip setuptools wheel
pip install -r requirements.txt
```
Or run the helper:
```bash
cd /wherever/you/downloaded/this/thing
./setup_vllm.sh
source .venv/bin/activate
```
## Single Video Inference
Edit `VIDEO_PATH` in `example_run.py`:
```python
VIDEO_PATH = "/path/to/video.mp4"
```
Then run:
```bash
cd /path/to/this/repo
source .venv/bin/activate
python example_run.py
```
The script prints the raw model output.
## Folder Captioning
Edit these variables in `caption_folder.py`:
```python
INPUT_FOLDER = "/path/to/videos"
OUTPUT_FOLDER = "/path/to/captions"
```
Then run:
```bash
cd /path/to/this/repo
source .venv/bin/activate
python caption_folder.py
```
The script mirrors the input folder structure into `OUTPUT_FOLDER` and writes one `.txt` file per video. It loads the model once and batches videos using `BATCH_SIZE`.
## Prompt Settings
The model was trained with some settings, grouped under `Prompt Settings`. Here are the recommended values:
- `HAS_THINKING` | True
- `CAPTION_LENGTH` | "very large"
- `INCLUDE_WATERMARK_INFO` | True
- `VULGARITY` | "low"
- `UNCERTAINTY` | "low"
- `CHARACTER_NAMES` | "none"
- `FLUFF` | "none"
- `HAS_REPETITION` | False
- `SPECULATION` | "low"
- `TEMPORAL_DETAIL` | "medium"
- `VISUAL_SPECIFICITY` | "moderate"
- `CAMERA_DETAIL` | "medium"
- `CAPTION_STYLE` | "plain"
### Possible Options
- `HAS_THINKING` | True, False
- `CAPTION_LENGTH` | "very small", "small", "medium", "large", "very large"
- `INCLUDE_WATERMARK_INFO` | True, False
- `VULGARITY` | "none", "low", "medium", "high"
- `UNCERTAINTY` | "none", "low", "medium", "high"
- `CHARACTER_NAMES` | "none", "ambiguous", "single", "multiple"
- `FLUFF` | "none", "low", "medium", "high"
- `HAS_REPETITION` | True, False
- `SPECULATION` | "none", "low", "medium", "high"
- `TEMPORAL_DETAIL` | "static", "low", "medium", "high"
- `VISUAL_SPECIFICITY` | "generic", "moderate", "detailed", "excessive"
- `CAMERA_DETAIL` | "none", "low", "medium", "high"
- `CAPTION_STYLE` | "plain", "verbose", "ornate", "robotic"
Generation settings are grouped under `vLLM / Generation Hyperparameters`:
- `NUM_FRAMES` | (was trained between 4 and 20)
- `MAX_MODEL_LEN`
- `MAX_NUM_SEQS`
- `BATCH_SIZE` in `caption_folder.py`
- `GPU_MEMORY_UTILIZATION`
- `MAX_TOKENS`
- `TEMPERATURE` | (recommend a lower value)
- `TOP_P`
- `REPETITION_PENALTY`
## Performance
This graph shows measured vLLM decode throughput on an RTX PRO 6000 using 12 video frames, `max_tokens = 96`, `max_model_len = 4096`, prefix cache disabled, and unique requests:
### BF16
### FP8
## Limitations
- This model still makes plenty of mistakes on very complex videos.
- It requires the bundled patched vLLM runtime for the vLLM examples.
- Audio captioning depends on the Parakeet bridge and may still miss unclear, quiet, overlapping, or non-speech sounds.
## Credits
- Gemma 4 31b for main model
- Parakeet 0.6b TDT v3 for audio |