| --- |
| license: cc-by-nc-4.0 |
| library_name: pytorch |
| pipeline_tag: image-text-to-text |
| tags: |
| - optical-gas-imaging |
| - thermal-imaging |
| - methane |
| - carbon-dioxide |
| - vision-language-model |
| - referring-expression-segmentation |
| - visual-question-answering |
| base_model: |
| - facebook/vjepa2-vitl-fpc64-256 |
| - HuggingFaceTB/SmolLM2-360M-Instruct |
| datasets: |
| - toqi/GasGround |
| --- |
| |
| # GasVLM |
|
|
| Trained weights for **GasVLM**, a sub-billion-parameter grounded vision-language model |
| for optical gas imaging (OGI). Given a thermal frame and a question, the model generates |
| a text answer that may contain `[SEG]` tokens; each `[SEG]` drives a mask decoder to |
| produce a binary segmentation. Gas concentrations and the CH4/CO2 ratio class appear |
| inside the answer text. |
|
|
| Code: https://github.com/toqitahamid/GasVLM |
|
|
| ## Model |
|
|
| Frozen [V-JEPA 2](https://huggingface.co/facebook/vjepa2-vitl-fpc64-256) ViT-L with a |
| single low-rank LoRA adapter on the attention projections, a pixel-shuffle projector, and |
| [SmolLM2-360M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct) with |
| q/k/v/o LoRA and trainable token embeddings. A multi-scale `[SEG]`-conditioned mask |
| decoder reads the encoder features to produce one mask per `[SEG]`. |
|
|
| Roughly 700M parameters, of which about 58M (8.3%) are trainable. Trained on GasGround, |
| an OGI set covering CH4 and CO2 plume video across three thermal cameras. |
|
|
| ## Usage |
|
|
| ```bash |
| git clone https://github.com/toqitahamid/GasVLM && cd GasVLM && pip install -e . |
| hf auth login |
| python tools/download_weights.py --repo-id toqi/GasVLM --filename gasvlm.pt |
| ``` |
|
|
| The checkpoint lands in `checkpoints/`. To reproduce the numbers below you also need |
| the [GasGround](https://huggingface.co/datasets/toqi/GasGround) test split: |
|
|
| ```bash |
| python tools/unpack_gasground.py --repo-id toqi/GasGround --out GasGround --splits test |
| ``` |
|
|
| Point the `data:` paths in `configs/gasvlm_headline.yaml` at the unpacked JSONL, then: |
|
|
| ```bash |
| bash dist_eval.sh eval.py 1 --config configs/gasvlm_headline.yaml \ |
| --checkpoint checkpoints/gasvlm.pt --output-dir runs/eval --split test --levels 1 |
| ``` |
|
|
| `gasvlm.pt` is a plain `state_dict` and loads into `GasVLM` with `strict=True`. |
|
|
| ## Results |
|
|
| GasGround test split. |
|
|
| | Level | Metric | Value | |
| | --- | --- | --- | |
| | RES | gIoU / cIoU | 0.705 / 0.701 | |
| | RES | gas IoU / source IoU | 0.691 / 0.834 | |
| | VQA | ratio-class accuracy / within-1 | 0.713 / 0.963 | |
| | VQA | CH4 / CO2 MAE | 6.45 / 6.45 | |
| | GCG | mIoU / mask AP50 | 0.679 / 0.611 | |
|
|
| ## License |
|
|
| The weights are released under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/). Commercial use is not permitted. The code at https://github.com/toqitahamid/GasVLM is under the MIT license. |
|
|
