license: cc-by-nc-4.0
library_name: pytorch
pipeline_tag: image-text-to-text
tags:
- optical-gas-imaging
- thermal-imaging
- methane
- carbon-dioxide
- vision-language-model
- referring-expression-segmentation
- visual-question-answering
base_model:
- facebook/vjepa2-vitl-fpc64-256
- HuggingFaceTB/SmolLM2-360M-Instruct
datasets:
- toqi/GasGround
GasVLM
Trained weights for GasVLM, a sub-billion-parameter grounded vision-language model
for optical gas imaging (OGI). Given a thermal frame and a question, the model generates
a text answer that may contain [SEG] tokens; each [SEG] drives a mask decoder to
produce a binary segmentation. Gas concentrations and the CH4/CO2 ratio class appear
inside the answer text.
Code: https://github.com/toqitahamid/GasVLM
Model
Frozen V-JEPA 2 ViT-L with a
single low-rank LoRA adapter on the attention projections, a pixel-shuffle projector, and
SmolLM2-360M-Instruct with
q/k/v/o LoRA and trainable token embeddings. A multi-scale [SEG]-conditioned mask
decoder reads the encoder features to produce one mask per [SEG].
Roughly 700M parameters, of which about 58M (8.3%) are trainable. Trained on GasGround, an OGI set covering CH4 and CO2 plume video across three thermal cameras.
Usage
git clone https://github.com/toqitahamid/GasVLM && cd GasVLM && pip install -e .
hf auth login
python tools/download_weights.py --repo-id toqi/GasVLM --filename gasvlm.pt
The checkpoint lands in checkpoints/. To reproduce the numbers below you also need
the GasGround test split:
python tools/unpack_gasground.py --repo-id toqi/GasGround --out GasGround --splits test
Point the data: paths in configs/gasvlm_headline.yaml at the unpacked JSONL, then:
bash dist_eval.sh eval.py 1 --config configs/gasvlm_headline.yaml \
--checkpoint checkpoints/gasvlm.pt --output-dir runs/eval --split test --levels 1
gasvlm.pt is a plain state_dict and loads into GasVLM with strict=True.
Results
GasGround test split.
| Level | Metric | Value |
|---|---|---|
| RES | gIoU / cIoU | 0.705 / 0.701 |
| RES | gas IoU / source IoU | 0.691 / 0.834 |
| VQA | ratio-class accuracy / within-1 | 0.713 / 0.963 |
| VQA | CH4 / CO2 MAE | 6.45 / 6.45 |
| GCG | mIoU / mask AP50 | 0.679 / 0.611 |
License
The weights are released under CC BY-NC 4.0. Commercial use is not permitted. The code at https://github.com/toqitahamid/GasVLM is under the MIT license.