Upload folder using huggingface_hub

.gitignore

@@ -0,0 +1,4 @@
+*__pycache__
+*.idea
+*.DS_Store
+*data/

README.md

@@ -0,0 +1,127 @@
+---
+language: 
+  - en
+tags:
+- classifier
+- multimodal
+- olfaction-vision-language
+- olfaction
+- olfactory
+- scentience
+- neural-network
+- graph-neural-network
+- gnn
+- vision-language
+- vision
+- language
+- robotics
+- multimodal
+- smell
+license: mit
+datasets:
+- kordelfrance/olfaction-vision-language-dataset
+- detection-datasets/coco
+- seyonec/goodscents_leffingwell
+base_model: Scentience-OVL-Classifiers-Base
+---
+
+# Olfaction-Vision-Language Classifiers
+
+
+[![MIT license](https://img.shields.io/badge/License-MIT-blue.svg)](#license)
+[![Colab](https://img.shields.io/badge/Run%20in-Colab-yellow?logo=google-colab)](https://colab.research.google.com/drive/1H5OSeO43YfhAT9MqcJKaaSknFYhjimvg?usp=sharing)
+[![Paper](https://img.shields.io/badge/Research-Paper-red)](https://arxiv.org/abs/2506.00398)
+[![Open in Spaces](https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-sm.svg)](https://huggingface.co/kordelfrance/Olfaction-Vision-Language-Embeddings)
+
+</div>
+
+---
+
+## Description
+
+This repository is a foundational series of multimodal joint classifier models trained on olfaction, vision, and language data.
+It is meant as a quick start on loading the olfaction-vision-language models and getting the probability/logits of the presence of observed chemical compounds in a visual scene given a set of aroma descriptors.
+For example, given an input image and a set of observed aromas (fruity, musky, etc), what is the probability that acetone is present?
+
+Based on the original series of [embeddings models here](https://huggingface.co/kordelfrance/Olfaction-Vision-Language-Embeddings), these models are built specifically for prototyping and exploratory tasks within AR/VR, robotics, and embodied artificial intelligence.
+Analogous to how CLIP and SigLIP embeddings give vision-language relationships, our embeddings models here give olfaction-vision-language (OVL) relationships.
+
+Whether these models are used for better vision-scent navigation with drones, triangulating the source of an odor in an image, extracting aromas from a scene, or augmenting a VR experience with scent, we hope their release will catalyze further research in olfaction, especially olfactory robotics.
+We especially hope these models encourage the community to contribute to building standardized datasets and evaluation protocols for olfaction-vision-language learning.
+
+## Models
+We offer four olfaction-vision-language (OVL) classifier models with this repository:
+ - (1) `ovlc-gat`: The OVL base model built around a graph-attention network. This model is optimal for online tasks where accuracy is paramount and inference time is not as critical.
+ - (2) `ovlc-base`: The original OVL base model optimized for faster inference and edge-based robotics. This model is optimized for export to common frameworks that run on Android, iOS, Rust, and others.
+
+## Training Data
+A sample dataset is included, but the full datasets are linked in the `Datasets` pane of this repo.
+Training code for replicating full construction of all models will be released soon.
+
+Please refer to original series of [embeddings models here](https://huggingface.co/kordelfrance/Olfaction-Vision-Language-Embeddings) for more information.
+
+
+## Directory Structure
+
+```text
+Olfaction-Vision-Language-Classifier-Models/
+├── data/                     # Sample training dataset
+├── requirements.txt          # Python dependencies
+├── model/                    # Classifier models
+├── model_cards/              # Specifications for each embedding model
+├── notebooks/                # Notebooks for loading the models for inference
+├── src/                      # Source code for inference, model loading, utils
+└── README.md                 # Overview of repository contributions and usage
+```
+
+---
+
+## Citation
+If you use any of these models, please cite:
+```
+    @misc{france2025ovlembeddings,
+        title = {Scentience-OVLE-v1: Joint Olfaction-Vision-Language Embeddings},
+        author = {Kordel Kade France},
+        year = {2025},
+        howpublished = {Hugging Face},
+        url = {https://huggingface.co/kordelfrance/Olfaction-Vision-Language-Embeddings}
+    }
+```
+
+```
+    @misc{france2025olfactionstandards,
+          title={Position: Olfaction Standardization is Essential for the Advancement of Embodied Artificial Intelligence}, 
+          author={Kordel K. France and Rohith Peddi and Nik Dennler and Ovidiu Daescu},
+          year={2025},
+          eprint={2506.00398},
+          archivePrefix={arXiv},
+          primaryClass={cs.AI},
+          url={https://arxiv.org/abs/2506.00398}, 
+    }
+```
+
+
+If you leverage the CLIP or SigLIP models, please cite:
+```
+    @misc{radford2021clip,
+        title        = {Learning Transferable Visual Models From Natural Language Supervision},
+        author       = {Alec Radford and Jong Wook Kim and Chris Hallacy and Aditya Ramesh and Gabriel Goh and Sandhini Agarwal and Girish Sastry and Amanda Askell and Pamela Mishkin and Jack Clark and Gretchen Krueger and Ilya Sutskever},
+        year         = 2021,
+        url          = {https://arxiv.org/abs/2103.00020},
+        eprint       = {2103.00020},
+        archiveprefix = {arXiv},
+        primaryclass = {cs.CV}
+    }
+```
+
+```
+    @misc{zhai2023siglip,
+          title={Sigmoid Loss for Language Image Pre-Training}, 
+          author={Xiaohua Zhai and Basil Mustafa and Alexander Kolesnikov and Lucas Beyer},
+          year={2023},
+          eprint={2303.15343},
+          archivePrefix={arXiv},
+          primaryClass={cs.CV},
+          url={https://arxiv.org/abs/2303.15343}, 
+}
+```

model/ovlc/base/gnn_512_c.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eb4b6457aabc3b952367066bf1dc651d8ff04612b7e921a6c1110b5d880ccce1
+size 6316648

model/ovlc/base/gnn_512_c.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:413f346ff9acfec2e4070a66ac54dd80a2486a81ff2e253a3c0a5a68ee950310
+size 6304684

model/ovlc/base/gnn_state_dict_512_c.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:47203e8bc45a421252e1f2224d68ad11a58952b5c5cf321f36ca924864e601b9
+size 6308841

model/ovlc/base/olf_encoder_512_c.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5abeab85ebef72e56b1554cba0e03eb37e99e67adf0093ef67c9d46fcca23db8
+size 1284568

model/ovlc/base/olf_encoder_512_c.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c01e4857b307039fa54657c6840783788dd24c09da6bce909cf084925ac6f966
+size 1269944

model/ovlc/base/olf_encoder_state_dict_512_c.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:00d8b4b52c8a49d21bca5e6c3610f9c7b567a39f1838ad86d7363eddfb109627
+size 1277657

model/ovlc/gat/gat_gnn_512_c.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d19648560bf0d48a3e63daf9e3c610f29d0471cfdab0d3e70098a2c34d16fb8
+size 34801880

model/ovlc/gat/gat_gnn_512_c.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5533a3a457bfa0e95f3a7cffe122343dc25cbdfedabb31cc381ad65a12fb6c24
+size 34712556

model/ovlc/gat/gat_gnn_state_dict_512_c.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e0878cd5f89dee08aa87860a724a1c13420994b2c7fee651b053764aaed026d
+size 34716913

model/ovlc/gat/gat_olf_encoder_512_c.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1fdef4dd630e6ec48f34ea74a3b9a4d003a93f884d6ce707fe6a3ae81ca69d11
+size 1269944

model/ovlc/gat/gat_olf_encoder_state_dict_512_c.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b0bd47e1c2e63a17c9c84ce7eeb5956f004857c012e46f2bb365b6496a332c54
+size 1278625

model/ovlc/gat/olf_encoder_gat_512_c.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9419610adcb252387f080625c23e5aff62d0d4218ca7caae9f7aac8c5077f28a
+size 1284920

model_cards/ovl_classifier.md

@@ -0,0 +1,110 @@
+# Model Card: Scentience-OVLE-Large-v1
+
+
+## Model Details
+- **Model Name:** `Scentience OVLE Large v1`
+- **Developed by:** Kordel K. France
+- **Date:** September 2025
+- **Architecture:**
+  - **Olfaction encoder:** 138-sensor embedding
+  - **Vision encoder:** CLIP-based
+  - **Language encoder:** CLIP-based
+  - **Fusion strategy:** Joint embedding space via multimodal contrastive training
+  - **Parameter Count (Base):** 29.9M (without CLIP), 181.2M (with CLIP)
+  - **Parameter Count (GAT):** 143.2M (without CLIP), 294.5M (with CLIP)
+  - **Embedding Dimension:** 2048
+- **License:** MIT
+- **Contact:** kordel@scentience.ai, kordel.france@utdallas.edu
+
+---
+
+## Intended Use
+- **Primary purpose:** Research in multimodal machine learning involving olfaction, vision, and language.  
+- **Example applications:**
+  - Cross-modal retrieval (odor → image, odor → text, etc.)
+  - Robotics and UAV navigation guided by chemical cues
+  - Chemical dataset exploration and visualization
+- **Intended users:** Researchers, developers, and educators working in ML, robotics, chemistry, and HCI.
+- **Out of scope:** Not intended for safety-critical tasks (e.g., gas leak detection, medical diagnosis, or regulatory use).
+
+---
+
+## Training Data
+- **Olfaction data:** Language-aligned olfactory data curated from GoodScents and LeffingWell datasets.
+- **Vision data:** COCO dataset.
+- **Language data:** Smell descriptors and text annotations curated from literature.
+
+For more information on how the training data was accumulated, please see the [HuggingFace dataset URL here](https://huggingface.co/datasets/kordelfrance/olfaction-vision-language-dataset)
+
+---
+
+## Evaluation
+- Retrieval tasks: odor→image (Top-5 recall = 62%)
+- Odor descriptor classification accuracy = 71%
+- Cross-modal embedding alignment qualitatively verified on 200 sample triplets.
+
+---
+
+## Limitations of Evaluation
+To the best of our knowledge, there are currently no open-source datasets that provide aligned olfactory, visual, and linguistic annotations. A “true” multimodal evaluation would require measuring the chemical composition of scenes (e.g., using gas chromatography mass spectrometry) while simultaneously capturing images and collecting perceptual descriptors from human olfactory judges. Such a benchmark would demand substantial new data collection efforts and instrumentation.
+Consequently, we evaluate our models indirectly, using surrogate metrics (e.g., cross-modal retrieval performance, odor descriptor classification accuracy, clustering quality). While these evaluations do not provide ground-truth verification of odor presence in images, they offer a first step toward demonstrating alignment between modalities.
+We draw analogy from past successes in ML datasets such as precursors to CLIP that lacked large paired datasets and were evaluated on retrieval-like tasks.
+As a result, we release this model to catalyze further research and encourage the community to contribute to building standardized datasets and evaluation protocols for olfaction-vision-language learning.
+
+---
+
+## Limitations
+- Limited odor diversity (approx. 5000 unique compounds).
+- Embeddings depend on sensor calibration; not guaranteed across devices.
+- Cultural subjectivity in smell annotations may bias embeddings.
+
+---
+
+## Ethical Considerations
+- Not to be used for covert detection of substances or surveillance.
+- Unreliable in safety-critical contexts (e.g., gas leak detection).
+- Recognizes cultural sensitivity in smell perception.
+
+---
+
+## Environmental Impact
+- Trained on 4×A100 GPUs for 48 hours (~200 kg CO2eq).
+- Sensor dataset collection required ~500 lab hours.
+
+---
+
+## Citation
+If you use this model, please cite:
+```
+    @misc{france2025ovlembeddings,
+        title = {Scentience-OVLE-Base-v1: Joint Olfaction-Vision-Language Embeddings},
+        author = {Kordel Kade France},
+        year = {2025},
+        howpublished = {Hugging Face},
+        url = {https://huggingface.co/kordelfrance/Olfaction-Vision-Language-Embeddings}
+    }
+```
+
+```
+    @misc{radford2021clip,
+        title        = {Learning Transferable Visual Models From Natural Language Supervision},
+        author       = {Alec Radford and Jong Wook Kim and Chris Hallacy and Aditya Ramesh and Gabriel Goh and Sandhini Agarwal and Girish Sastry and Amanda Askell and Pamela Mishkin and Jack Clark and Gretchen Krueger and Ilya Sutskever},
+        year         = 2021,
+        url          = {https://arxiv.org/abs/2103.00020},
+        eprint       = {2103.00020},
+        archiveprefix = {arXiv},
+        primaryclass = {cs.CV}
+    }
+```
+
+```
+    @misc{zhai2023siglip,
+          title={Sigmoid Loss for Language Image Pre-Training}, 
+          author={Xiaohua Zhai and Basil Mustafa and Alexander Kolesnikov and Lucas Beyer},
+          year={2023},
+          eprint={2303.15343},
+          archivePrefix={arXiv},
+          primaryClass={cs.CV},
+          url={https://arxiv.org/abs/2303.15343}, 
+}
+```

notebooks/olfaction_vision_language_classifier_inference.ipynb

@@ -0,0 +1,235 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "gpuType": "A100"
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    },
+    "accelerator": "GPU"
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Olfaction-Vision-Language-Classifier"
+      ],
+      "metadata": {
+        "id": "SHn5L_NdNL-V"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "This is a quick start on loading the olfaction-vision-language models and getting the probability/logits of the presence of observed chemical compounds in a visual scene given a set of aroma descriptors."
+      ],
+      "metadata": {
+        "id": "_40H8e0QNCj_"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Install Libraries"
+      ],
+      "metadata": {
+        "id": "g0qE7ci4M_V6"
+      }
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 1,
+      "metadata": {
+        "id": "cxabQxw9LSzM",
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "outputId": "a29b3012-09f1-4ef6-ac54-8f30494fea9f"
+      },
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Requirement already satisfied: transformers in /usr/local/lib/python3.12/dist-packages (4.57.1)\n",
+            "Requirement already satisfied: filelock in /usr/local/lib/python3.12/dist-packages (from transformers) (3.20.0)\n",
+            "Requirement already satisfied: huggingface-hub<1.0,>=0.34.0 in /usr/local/lib/python3.12/dist-packages (from transformers) (0.36.0)\n",
+            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.12/dist-packages (from transformers) (2.0.2)\n",
+            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.12/dist-packages (from transformers) (25.0)\n",
+            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.12/dist-packages (from transformers) (6.0.3)\n",
+            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.12/dist-packages (from transformers) (2024.11.6)\n",
+            "Requirement already satisfied: requests in /usr/local/lib/python3.12/dist-packages (from transformers) (2.32.4)\n",
+            "Requirement already satisfied: tokenizers<=0.23.0,>=0.22.0 in /usr/local/lib/python3.12/dist-packages (from transformers) (0.22.1)\n",
+            "Requirement already satisfied: safetensors>=0.4.3 in /usr/local/lib/python3.12/dist-packages (from transformers) (0.6.2)\n",
+            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.12/dist-packages (from transformers) (4.67.1)\n",
+            "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub<1.0,>=0.34.0->transformers) (2025.3.0)\n",
+            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub<1.0,>=0.34.0->transformers) (4.15.0)\n",
+            "Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub<1.0,>=0.34.0->transformers) (1.2.0)\n",
+            "Requirement already satisfied: charset_normalizer<4,>=2 in /usr/local/lib/python3.12/dist-packages (from requests->transformers) (3.4.4)\n",
+            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.12/dist-packages (from requests->transformers) (3.11)\n",
+            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.12/dist-packages (from requests->transformers) (2.5.0)\n",
+            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.12/dist-packages (from requests->transformers) (2025.10.5)\n",
+            "Requirement already satisfied: safetensors in /usr/local/lib/python3.12/dist-packages (0.6.2)\n"
+          ]
+        }
+      ],
+      "source": [
+        "!pip install transformers\n",
+        "!pip install safetensors"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Import and Configure"
+      ],
+      "metadata": {
+        "id": "7HB6A3RDMrQ0"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import torch\n",
+        "import torch.nn as nn\n",
+        "from safetensors.torch import load_file\n",
+        "from torchvision import transforms\n",
+        "from transformers import CLIPProcessor, CLIPModel\n",
+        "from PIL import Image\n",
+        "\n",
+        "\n",
+        "DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
+        "EMBED_DIM = 512     # Embedding dims = 512 for classifiers\n",
+        "ENCODER_FILE_PATH = f\"./olf_encoder_{EMBED_DIM}_c.pt\"\n",
+        "GNN_FILE_PATH = f\"./gnn_{EMBED_DIM}_c.pt\""
+      ],
+      "metadata": {
+        "id": "BHUd6n3bLbqo"
+      },
+      "execution_count": 2,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Embeddings Function"
+      ],
+      "metadata": {
+        "id": "Tad9Pu6PMn9g"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def get_embeddings(clip_model, olf_encoder, graph_model, image, olf_vec):\n",
+        "    \"\"\"\n",
+        "    Gets joint olfaction-vision-language embeddings for a given image and olfaction vector.\n",
+        "\n",
+        "    :param clip_model: vision-language model\n",
+        "    :param olf_encoder: olfactory encoder from aromas/molecules\n",
+        "    :param graph_model: cross-modal associator\n",
+        "    :param image: PIL image\n",
+        "    :param olf_vec: olfaction vector\n",
+        "    :return: joint olfaction-vision-language embeddings\n",
+        "    \"\"\"\n",
+        "    clip_model.eval()\n",
+        "    olf_encoder.eval()\n",
+        "    graph_model.eval()\n",
+        "\n",
+        "    transform = transforms.Compose([\n",
+        "        transforms.Resize((224, 224)),\n",
+        "        transforms.ToTensor(),\n",
+        "    ])\n",
+        "\n",
+        "    image_tensor = transform(image).unsqueeze(0).to(DEVICE)\n",
+        "    olf_tensor = torch.tensor(olf_vec, dtype=torch.float32).unsqueeze(0).to(DEVICE)\n",
+        "\n",
+        "    with torch.no_grad():\n",
+        "        vision_embeds = clip_model.get_image_features(pixel_values=image_tensor)\n",
+        "        if EMBED_DIM != 768 and EMBED_DIM != 512:\n",
+        "            projection = nn.Linear(vision_embeds.shape[-1], EMBED_DIM).to(DEVICE)\n",
+        "            vision_embeds = projection(vision_embeds).to(DEVICE)\n",
+        "        vision_embeds = vision_embeds.to(DEVICE)\n",
+        "        olf_embeds = olf_encoder(olf_tensor).to(DEVICE)\n",
+        "        ovl_logits = graph_model(vision_embeds, olf_embeds).squeeze()\n",
+        "\n",
+        "    return ovl_logits"
+      ],
+      "metadata": {
+        "id": "UzhGG8CzMmBs"
+      },
+      "execution_count": 3,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Get Joint Embeddings from a Data Sample"
+      ],
+      "metadata": {
+        "id": "fQIb0yeiMwRN"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Load the models\n",
+        "olf_encoder = torch.jit.load(ENCODER_FILE_PATH)\n",
+        "graph_model = torch.jit.load(GNN_FILE_PATH)\n",
+        "clip_model = CLIPModel.from_pretrained(\"openai/clip-vit-base-patch32\").to(DEVICE)\n",
+        "\n",
+        "# Build example vision-olfaction sample with dummy data\n",
+        "example_image = Image.new('RGB', (224, 224))\n",
+        "example_image.save(f\"/tmp/image_example.jpg\")\n",
+        "example_olf_vec = torch.randn(112)\n",
+        "\n",
+        "# Run inference\n",
+        "logits = get_embeddings(\n",
+        "    clip_model,\n",
+        "    olf_encoder,\n",
+        "    graph_model,\n",
+        "    example_image,\n",
+        "    example_olf_vec\n",
+        ")\n",
+        "print(\"Logits\", logits)"
+      ],
+      "metadata": {
+        "id": "_U5qqxn8Mibo",
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "outputId": "23b27505-539d-4262-d1a2-45186367afcb"
+      },
+      "execution_count": 4,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_auth.py:86: UserWarning: \n",
+            "Access to the secret `HF_TOKEN` has not been granted on this notebook.\n",
+            "You will not be requested again.\n",
+            "Please restart the session if you want to be prompted again.\n",
+            "  warnings.warn(\n",
+            "/tmp/ipython-input-2356812152.py:22: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.detach().clone() or sourceTensor.detach().clone().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
+            "  olf_tensor = torch.tensor(olf_vec, dtype=torch.float32).unsqueeze(0).to(DEVICE)\n"
+          ]
+        },
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Logits tensor(3.8031e+23, device='cuda:0')\n"
+          ]
+        }
+      ]
+    }
+  ]
+}

requirements.txt

@@ -0,0 +1,33 @@
+hf-xet==1.1.9
+hf_transfer==0.1.9
+huggingface-hub==0.34.4
+matplotlib==3.10.0
+numpy==2.0.2
+openai==1.106.1
+opencv-contrib-python==4.12.0.88
+opencv-python==4.12.0.88
+opencv-python-headless==4.12.0.88
+openpyxl==3.1.5
+opt_einsum==3.4.0
+pandas==2.2.2
+pandas-datareader==0.10.0
+requests==2.32.4
+safetensors==0.6.2
+scikit-image==0.25.2
+scikit-learn==1.6.1
+scipy==1.16.1
+sklearn-pandas==2.2.0
+tiktoken==0.11.0
+tokenizers==0.22.0
+torch==2.8.0+cu126
+torch-geometric==2.6.1
+torchao==0.10.0
+torchaudio==2.8.0+cu126
+torchdata==0.11.0
+torchsummary==1.5.1
+torchtune==0.6.1
+torchvision==0.23.0+cu126
+tqdm==4.67.1
+transformers==4.56.1
+urllib3==2.5.0
+uvicorn==0.35.0

src/base_model.py

@@ -0,0 +1,52 @@
+import torch
+from torchvision import transforms
+from transformers import CLIPModel, SiglipModel
+
+from src import constants
+
+
+# INFERENCE
+def run_inference(vision_lang_encoder, olf_encoder, graph_model, image, olf_vec):
+    vision_lang_encoder.eval()
+    olf_encoder.eval()
+    graph_model.eval()
+
+    transform = transforms.Compose([
+        transforms.Resize((constants.IMG_DIM, constants.IMG_DIM)),
+        transforms.ToTensor(),
+    ])
+
+    image_tensor = transform(image).unsqueeze(0).to(constants.DEVICE)
+    olf_tensor = torch.tensor(olf_vec, dtype=torch.float32).unsqueeze(0).to(constants.DEVICE)
+
+    with torch.no_grad():
+        vision_embed = vision_lang_encoder.get_image_features(pixel_values=image_tensor)
+        olf_embed = olf_encoder(olf_tensor)
+
+        nodes = torch.cat([vision_embed, olf_embed], dim=0)
+        edge_index = torch.cartesian_prod(torch.arange(nodes.size(0)), torch.arange(nodes.size(0))).T.to(constants.DEVICE)
+        logits = graph_model(nodes, edge_index)
+
+    return logits
+
+
+def load_model():
+    # Use CLIP as default baseline
+    clip_model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32").to(constants.DEVICE)
+    clip_model.eval()
+    """
+    Or, you can also use SigLIP:
+        SiglipModel.from_pretrained(
+            "google/siglip-so400m-patch14-384",
+            attn_implementation="flash_attention_2",
+            dtype=torch.float16,
+            device_map=constants.DEVICE,
+        )
+    """
+    olf_encoder = torch.jit.load(constants.ENCODER_SMALL_GRAPH_PATH).to(constants.DEVICE)
+    olf_encoder.eval()
+    graph_model = torch.jit.load(constants.OVLE_SMALL_GRAPH_PATH).to(constants.DEVICE)
+    graph_model.eval()
+
+    return clip_model, olf_encoder, graph_model
+

src/constants.py

@@ -0,0 +1,19 @@
+import torch
+
+
+# Number of features in each aroma vector
+AROMA_VEC_LENGTH: int = 138
+# All images are normalized to this
+IMG_DIM: int = 224
+# Each model was trained with these hyperparams
+BATCH_SIZE: int = 16
+EMBED_DIM: int = 512
+
+# CPU or GPU?
+DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+# Paths to models
+OVLE_SMALL_BASE_PATH: str = f"./model/ovlc/base/gnn_{EMBED_DIM}_c.pt"
+ENCODER_SMALL_BASE_PATH: str = f"./model/ovlc/base/olf_encoder_{EMBED_DIM}_c.pt"
+OVLE_LARGE_GRAPH_PATH: str = f"./model/ovlc/graph/gat_gnn_{EMBED_DIM}_c.pt"
+ENCODER_LARGE_GRAPH_PATH: str = f"./model/ovlc/graph/gat_olf_encoder_{EMBED_DIM}_c.pt"

src/graph_model.py

@@ -0,0 +1,48 @@
+import torch
+from torchvision import transforms
+from transformers import CLIPModel, SiglipModel
+
+from src import constants
+
+
+# INFERENCE
+def run_inference(vision_lang_encoder, olf_encoder, graph_model, image, olf_vec):
+    vision_lang_encoder.eval()
+    olf_encoder.eval()
+    graph_model.eval()
+
+    transform = transforms.Compose([
+        transforms.Resize((constants.IMG_DIM, constants.IMG_DIM)),
+        transforms.ToTensor(),
+    ])
+
+    image_tensor = transform(image).unsqueeze(0).to(constants.DEVICE)
+    olf_tensor = torch.tensor(olf_vec, dtype=torch.float32).unsqueeze(0).to(constants.DEVICE)
+
+    with torch.no_grad():
+        vision_embed = vision_lang_encoder.get_image_features(pixel_values=image_tensor)
+        olf_embed = olf_encoder(olf_tensor)
+        logits = graph_model(vision_embed, olf_embed).squeeze()
+
+    return logits
+
+
+def load_model():
+    # Use CLIP as default baseline
+    clip_model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32").to(constants.DEVICE)
+    clip_model.eval()
+    """
+    Or, you can also use SigLIP:
+        SiglipModel.from_pretrained(
+            "google/siglip-so400m-patch14-384",
+            attn_implementation="flash_attention_2",
+            dtype=torch.float16,
+            device_map=constants.DEVICE,
+        )
+    """
+    olf_encoder = torch.jit.load(constants.ENCODER_SMALL_BASE_PATH).to(constants.DEVICE)
+    olf_encoder.eval()
+    graph_model = torch.jit.load(constants.OVLE_SMALL_BASE_PATH).to(constants.DEVICE)
+    graph_model.eval()
+
+    return clip_model, olf_encoder, graph_model

src/main.py

@@ -0,0 +1,41 @@
+import torch
+from PIL import Image
+
+from src import constants
+from src import base_model as bm
+from src import graph_model as gm
+
+
+if __name__ == "__main__":
+
+    # Build example vision-olfaction sample with dummy data
+    example_image = Image.new('RGB', (constants.IMG_DIM, constants.IMG_DIM))
+    example_image.save(f"/tmp/image_example.jpg")
+    example_olf_vec = torch.randn(constants.AROMA_VEC_LENGTH)
+
+    # -------- Option A --------
+    # Load the base models
+    vision_lang_encoder, olf_encoder, graph_model = bm.load_model()
+    # Get probability from base models
+    ovl_classifier_base = bm.run_inference(
+        vision_lang_encoder=vision_lang_encoder,
+        olf_encoder=olf_encoder,
+        graph_model=graph_model,
+        image=example_image,
+        olf_vec=example_olf_vec
+    )
+    print(f"Olfaction-Vision-Language Logits from Base Model: {ovl_classifier_base}")
+
+    # -------- Option B --------
+    # Load the graph attention models
+    vision_lang_encoder, olf_encoder, graph_model = gm.load_model()
+    # Get probability from graph attention models
+    ovl_classifier_graph = gm.run_inference(
+        vision_lang_encoder=vision_lang_encoder,
+        olf_encoder=olf_encoder,
+        graph_model=graph_model,
+        image=example_image,
+        olf_vec=example_olf_vec
+    )
+    print(f"Olfaction-Vision-Language Logits from Graph Attention Model: {ovl_classifier_graph}")
+

src/utils.py

@@ -0,0 +1,29 @@
+import torch
+from torch.utils.data import DataLoader, Dataset
+from torchvision import transforms
+from PIL import Image
+
+from src import constants
+
+
+# DATASET EXAMPLE
+class OlfactionVisionDataset(Dataset):
+    def __init__(self, image_paths, olfaction_vectors, labels):
+        self.image_paths = image_paths
+        self.olfaction_vectors = olfaction_vectors
+        self.labels = labels
+        self.transform = transforms.Compose([
+            transforms.Resize((constants.IMG_DIM, constants.IMG_DIM)),
+            transforms.ToTensor(),
+        ])
+
+    def __len__(self):
+        return len(self.image_paths)
+
+    def __getitem__(self, idx):
+        img_path = self.image_paths[idx]
+        image = self.transform(Image.open(img_path).convert('RGB'))
+        olf_vec = self.olfaction_vectors[idx]
+        label = self.labels[idx]
+        return image, torch.tensor(olf_vec, dtype=torch.float32), label
+