Upload Hyper3-CLIP beta ViT-B scratch checkpoint

.hfignore

@@ -0,0 +1,3 @@
+checkpoint_step_500000.pt
+*.tmp
+*.partial

0_Hyper3CLIP/config.json

@@ -0,0 +1,7 @@
+{
+  "model_config_file": "config.yaml",
+  "weights_file": "model.safetensors",
+  "image_size": 224,
+  "max_text_length": 77,
+  "normalize_output": true
+}

LICENSE

@@ -0,0 +1,49 @@
+OpenMDW License Agreement, version 1.0 (OpenMDW-1.0)
+
+By exercising rights granted to you under this agreement, you accept and agree
+to its terms.
+
+As used in this agreement, "Model Materials" means the materials provided to
+you under this agreement, consisting of: (1) one or more machine learning
+models (including architecture and parameters); and (2) all related artifacts
+(including associated data, documentation and software) that are provided to
+you hereunder.
+
+Subject to your compliance with this agreement, permission is hereby granted,
+free of charge, to deal in the Model Materials without restriction, including
+under all copyright, patent, database, and trade secret rights included or
+embodied therein.
+
+If you distribute any portion of the Model Materials, you shall retain in your
+distribution (1) a copy of this agreement, and (2) all copyright notices and
+other notices of origin included in the Model Materials that are applicable to
+your distribution.
+
+If you file, maintain, or voluntarily participate in a lawsuit against any
+person or entity asserting that the Model Materials directly or indirectly
+infringe any patent, then all rights and grants made to you hereunder are
+terminated, unless that lawsuit was in response to a corresponding lawsuit
+first brought against you.
+
+This agreement does not impose any restrictions or obligations with respect to
+any use, modification, or sharing of any outputs generated by using the Model
+Materials.
+
+THE MODEL MATERIALS ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE, TITLE, NONINFRINGEMENT, ACCURACY, OR THE
+ABSENCE OF LATENT OR OTHER DEFECTS OR ERRORS, WHETHER OR NOT DISCOVERABLE, ALL
+TO THE GREATEST EXTENT PERMISSIBLE UNDER APPLICABLE LAW.
+
+YOU ARE SOLELY RESPONSIBLE FOR (1) CLEARING RIGHTS OF OTHER PERSONS THAT MAY
+APPLY TO THE MODEL MATERIALS OR ANY USE THEREOF, INCLUDING WITHOUT LIMITATION
+ANY PERSON'S COPYRIGHTS OR OTHER RIGHTS INCLUDED OR EMBODIED IN THE MODEL
+MATERIALS; (2) OBTAINING ANY NECESSARY CONSENTS, PERMISSIONS OR OTHER RIGHTS
+REQUIRED FOR ANY USE OF THE MODEL MATERIALS; OR (3) PERFORMING ANY DUE
+DILIGENCE OR UNDERTAKING ANY OTHER INVESTIGATIONS INTO THE MODEL MATERIALS OR
+ANYTHING INCORPORATED OR EMBODIED THEREIN.
+
+IN NO EVENT SHALL THE PROVIDERS OF THE MODEL MATERIALS BE LIABLE FOR ANY CLAIM,
+DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE MODEL MATERIALS, THE
+USE THEREOF OR OTHER DEALINGS THEREIN.

NOTICE

@@ -0,0 +1,16 @@
+Hyper3-CLIP beta
+Copyright hyper³labs.
+
+Hyper3-CLIP beta was created by hyper³labs.
+
+This notice identifies the original source of the model materials. Redistributions
+of this model or derivative model materials should preserve this notice, the
+accompanying LICENSE file, and the original model card when practical.
+
+hyper³labs and Hyper3-CLIP are names associated with hyper³labs. No trademark
+license is granted. Modified or derived checkpoints must not use the hyper³labs
+or Hyper3-CLIP names in a way that suggests they are official hyper³labs releases
+or endorsed by hyper³labs.
+
+Please cite and link to the original hyper³labs model repository when publishing
+benchmarks, papers, derivative checkpoints, or public demos based on this model.

README.md

@@ -0,0 +1,187 @@
+---
+license: openmdw-1.0
+library_name: sentence-transformers
+pipeline_tag: feature-extraction
+tags:
+- vision-language
+- multimodal
+- image-text-retrieval
+- hyperbolic-embeddings
+- clip
+- sentence-transformers
+- transformers
+- haystack
+- safetensors
+- research
+- scratch-training
+---
+
+# Hyper3-CLIP beta
+
+Hyper3-CLIP beta is an open-weight hyperbolic vision-language checkpoint from
+hyper³labs. It places image and text representations in a Lorentz space and was
+trained with compositional entailment constraints for hierarchy-sensitive
+image-text retrieval.
+
+This beta release is intended as an open baseline and research artifact.
+
+## Model
+
+- Architecture: ViT-B scale vision-language model
+- Vision backbone: `vit_base_patch16_224`
+- Text backbone architecture/tokenizer: `openai/clip-vit-base-patch32`
+- Embedding dimension: 512
+- Training steps: 500,000
+- Global batch size: 768
+- Weights artifact: `model.safetensors`
+
+The original full training checkpoint included optimizer, scheduler, AMP scaler,
+RNG state, config, and step metadata. This repository publishes the weights-only
+`model.safetensors` artifact for inference and downstream research from the
+ViT-B scratch training run.
+
+## Quick Start: Sentence Transformers
+
+The default way to use this checkpoint is through Sentence Transformers. The
+adapter in this repository returns 512-dimensional L2-normalized tangent-space
+embeddings for standard cosine/dot-product vector stores.
+
+Install the runtime dependencies:
+
+```bash
+pip install "sentence-transformers>=5.5.1" timm safetensors pyyaml Pillow
+```
+
+If you are using the gated Hugging Face repository from a fresh machine, accept
+access on the model page and set `HF_TOKEN`.
+
+```python
+from PIL import Image
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer("hyper3labs/hyper3-clip-beta", trust_remote_code=True)
+
+image_embedding = model.encode([Image.open("/path/to/image.jpg")], normalize_embeddings=True)
+text_embedding = model.encode(["machined metal part"], normalize_embeddings=True)
+```
+
+## Transformers
+
+```python
+from PIL import Image
+import torch
+from transformers import AutoModel, AutoTokenizer
+
+model = AutoModel.from_pretrained("hyper3labs/hyper3-clip-beta", trust_remote_code=True).eval()
+tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32")
+
+image = model.preprocess_image(Image.open("/path/to/image.jpg")).unsqueeze(0)
+text = tokenizer(
+    ["machined metal part"],
+    padding=True,
+    truncation=True,
+    max_length=model.config.max_text_length,
+    return_tensors="pt",
+)
+
+with torch.no_grad():
+    outputs = model(
+        pixel_values=image,
+        input_ids=text["input_ids"],
+        attention_mask=text["attention_mask"],
+    )
+
+image_embedding = outputs.image_embeds
+text_embedding = outputs.text_embeds
+```
+
+<details>
+<summary>Haystack image retrieval pipeline</summary>
+
+For indexing images in a Haystack retrieval pipeline, use
+`SentenceTransformersDocumentImageEmbedder` with image paths in
+`Document.meta["file_path"]`, paired with `SentenceTransformersTextEmbedder` for
+text queries.
+
+```bash
+pip install "haystack-ai>=2.30.1" "sentence-transformers>=5.5.1" timm safetensors pyyaml Pillow
+```
+
+```python
+from haystack import Document
+from haystack.components.embedders import SentenceTransformersTextEmbedder
+from haystack.components.embedders.image import SentenceTransformersDocumentImageEmbedder
+
+model_id = "hyper3labs/hyper3-clip-beta"
+
+documents = [
+    Document(
+        content="front view of a machined metal part",
+        meta={"file_path": "/path/to/image.jpg"},
+    )
+]
+
+image_embedder = SentenceTransformersDocumentImageEmbedder(
+    model=model_id,
+    trust_remote_code=True,
+    batch_size=8,
+    normalize_embeddings=True,
+)
+documents = image_embedder.run(documents=documents)["documents"]
+
+text_embedder = SentenceTransformersTextEmbedder(
+    model=model_id,
+    trust_remote_code=True,
+    normalize_embeddings=True,
+)
+query_embedding = text_embedder.run("machined metal part")["embedding"]
+```
+
+</details>
+
+## Evaluation
+
+The numbers below use the official evaluator convention for R@10. Higher is
+better except for TIE and LCA.
+
+| Model | Comparable setting | ImageNet top-1 | COCO text R@10 | COCO image R@10 | Flickr text R@10 | Flickr image R@10 | TIE | LCA | Jaccard | H-Prec | H-Rec |
+|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| MERU-B/16 | same-family baseline | 40.1 | 82.0 | 68.6 | 96.2 | 90.0 | 3.630 | 2.220 | 0.780 | 0.850 | 0.850 |
+| HyCoCLIP-B/16 | official checkpoint | 45.8 | 82.0 | 69.3 | 95.4 | 90.3 | 3.172 | 2.047 | 0.814 | 0.874 | 0.874 |
+| UNCHA-B/16 | official checkpoint | 48.8 | 82.6 | 71.0 | 95.9 | 91.2 | 2.945 | 1.961 | 0.828 | 0.883 | 0.884 |
+| PHyCLIP-B/16 | related reported result | 44.4 | 80.4 | 68.7 | 95.6 | 89.9 | 3.285 | 2.088 | 0.807 | 0.868 | 0.868 |
+| Hyper3-CLIP beta | this release | 48.5 | 84.0 | 72.8 | 97.5 | 92.4 | 2.972 | 1.986 | 0.828 | 0.882 | 0.883 |
+
+Raw evaluation files are included:
+
+- `eval_coco_karpathy_final.json`
+- `eval_flickr30k_final.json`
+- `eval_imagenet_final.json`
+- `eval_hycoclip_uncha_intersection_final.json`
+
+## License And Attribution
+
+The model materials in this repository are released under OpenMDW-1.0. See
+`LICENSE`.
+
+Redistributions should preserve `NOTICE`, `LICENSE`, and the original model card
+when practical. Modified or derived checkpoints should use a distinct name and
+must not imply endorsement by hyper³labs.
+
+Please cite and link to the original hyper³labs model repository when publishing
+benchmarks, papers, derivative checkpoints, or public demos based on this model.
+
+## Intended Use
+
+This release is intended for:
+
+- hierarchy-sensitive image-text retrieval research
+- zero-shot and retrieval evaluation
+- multimodal embedding baselines
+- downstream experiments with hyperbolic representation learning
+
+This model has not been validated for safety-critical use.
+
+## Citation
+
+If you use Hyper3-CLIP beta, cite the original model repository and hyper³labs.

SHA256SUMS

@@ -0,0 +1,19 @@
+127022078882a067d3729185679645a173b7831f22454e02f1d48bf4caca454e  ./.hfignore
+00858698a701445ca5e8e0a753832834d605ae5ce0fc029cc1f4b41e8bf334be  ./0_Hyper3CLIP/config.json
+22d5488faff30b2e34e934779d2a6f033712ec6d1eaaee7e774e9cbe0c3c7587  ./LICENSE
+bc06204c8f542252f901c42929aa964a670f65837d961c947e2b404e9ada57e9  ./NOTICE
+2524472f3efc94e5215beac98fdc8b3e7ff1296e948e0259a3fe09ac44b43041  ./README.md
+88e0cd5e7a6f13394196662fda3291058ddd5b738d8c57a3c05f635c366ae9fd  ./config.json
+9639dadcd1152dc059ff7b158b3d9a87395ea9ef99917f0e254b79f93fe72efa  ./config.yaml
+bc149f4539e0fabd8d4392324f8d4e3d13a3bea0c41cc504349c51be3215bac2  ./config_sentence_transformers.json
+7ee8d05bf0e690b40930f561c51ada9d54badc82602f279e9e1222fc51d89c0e  ./configuration_hyper3_clip.py
+d4ec8249d963614c6bea43812b71157b5d01be7aeee23ea828d77fca8bb26314  ./eval_coco_karpathy_final.json
+4bff22e2f2ea6a5451847577f05238002888ad2670be8a4f788203879dd2bd1d  ./eval_flickr30k_final.json
+589fd4d0711b266aa77255737d767f184006fac03f65a69ed1a7632deee812b0  ./eval_hycoclip_uncha_intersection_final.json
+b6a68d90cae001ccc422220d371d717d30cce6d993e8c45f36a6a8c7c2dd38b3  ./eval_imagenet_final.json
+13b30904fbfdcfad9b968ba1bd5853fe5de0d3bf0dccfa9d3baab0f82c6ea5f4  ./hyper3_clip_sentence_transformers.py
+c86353fda268117808086af62808f54896c1ce954f9a653cafa84b97172effd2  ./metadata.json
+b199fc5574ed85df39c3b0de0d8b5a3e998fcddd29dd181125dd2274aae98555  ./model.safetensors
+605bda2e201ef962387cac2ed64d8735b82b29da57c24ded54ced5f9e3ece04f  ./modeling_hyper3_clip.py
+14c3e2e85d941bac307a35600d1ad8377fddd9cde85b0d706c19a623b9d311d2  ./modules.json
+8ace6250857b88214db892908ee8b85d80da96dd9f7020be8bfd4de327326e31  ./requirements.txt

config.json

@@ -0,0 +1,19 @@
+{
+  "model_type": "hyper3_clip",
+  "architectures": [
+    "Hyper3CLIPModel"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_hyper3_clip.Hyper3CLIPConfig",
+    "AutoModel": "modeling_hyper3_clip.Hyper3CLIPModel"
+  },
+  "vision_backbone": "vit_base_patch16_224",
+  "text_model_name": "openai/clip-vit-base-patch32",
+  "embed_dim": 512,
+  "curv_init": 1.0,
+  "learn_curv": true,
+  "image_size": 224,
+  "max_text_length": 77,
+  "torch_dtype": "float32",
+  "transformers_version": "4.49.0"
+}

config.yaml

@@ -0,0 +1,45 @@
+project:
+  name: hyper3-clip
+  experiment: hyper3_clip_vit_b_8xh100_full
+seed: 7
+output_dir: /sc/projects/sci-aisc/matin.mahmood/runs/hyper3_clip_vit_b_8xh100_full
+model:
+  vision_backbone: vit_base_patch16_224
+  text_model_name: openai/clip-vit-base-patch32
+  embed_dim: 512
+  curv_init: 1.0
+  learn_curv: true
+  entail_weight: 0.2
+  inter_aperture_scale: 0.7
+  intra_aperture_scale: 1.2
+training:
+  total_steps: 500000
+  global_batch_size: 768
+  grad_accum_steps: 1
+  num_workers: 8
+  lr: 0.0005
+  weight_decay: 0.2
+  betas:
+  - 0.9
+  - 0.98
+  warmup_steps: 4000
+  log_interval: 20
+  ckpt_interval: 10000
+  amp: true
+  max_grad_norm: 1.0
+  resume: true
+optimizer:
+  no_decay_params:
+  - logit_scale
+  - visual_alpha
+  - textual_alpha
+  - log_curv
+data:
+  type: processed_grit
+  part_sampling: random_one
+  tarfiles:
+  - /sc/projects/sci-aisc/matin.mahmood/datasets/hycoclip/train/GRIT/processed/*.tar
+  shuffle_buffer: 4000
+  image_size: 224
+  max_text_length: 77
+  num_workers: 8

config_sentence_transformers.json

@@ -0,0 +1,7 @@
+{
+  "__version__": {
+    "sentence_transformers": "5.5.1"
+  },
+  "model_type": "SentenceTransformer",
+  "similarity_fn_name": "cosine"
+}

configuration_hyper3_clip.py

@@ -0,0 +1,27 @@
+from __future__ import annotations
+
+from transformers import PretrainedConfig
+
+
+class Hyper3CLIPConfig(PretrainedConfig):
+    model_type = "hyper3_clip"
+
+    def __init__(
+        self,
+        vision_backbone: str = "vit_base_patch16_224",
+        text_model_name: str = "openai/clip-vit-base-patch32",
+        embed_dim: int = 512,
+        curv_init: float = 1.0,
+        learn_curv: bool = True,
+        image_size: int = 224,
+        max_text_length: int = 77,
+        **kwargs,
+    ) -> None:
+        super().__init__(**kwargs)
+        self.vision_backbone = vision_backbone
+        self.text_model_name = text_model_name
+        self.embed_dim = int(embed_dim)
+        self.curv_init = float(curv_init)
+        self.learn_curv = bool(learn_curv)
+        self.image_size = int(image_size)
+        self.max_text_length = int(max_text_length)

eval_coco_karpathy_final.json

@@ -0,0 +1,21 @@
+{
+  "checkpoint": "/sc/projects/sci-aisc/matin.mahmood/runs/hyper3_clip_vit_b_8xh100_full/checkpoint_final.pt",
+  "config": "configs/eval_common_coco_karpathy_8xh100_full_final.yaml",
+  "model_config": "configs/hyper3_clip_vit_b_8xh100_full.yaml",
+  "results": {
+    "coco_karpathy_retrieval": {
+      "i2t_r1": 50.260000000000005,
+      "i2t_r10": 83.96000000000001,
+      "i2t_r5": 76.02,
+      "image_to_text_r1": 0.5026,
+      "image_to_text_r10": 0.8396,
+      "image_to_text_r5": 0.7602,
+      "t2i_r1": 36.93722486495972,
+      "t2i_r10": 73.15473556518555,
+      "t2i_r5": 62.64294385910034,
+      "text_to_image_r1": 0.36937224864959717,
+      "text_to_image_r10": 0.7315473556518555,
+      "text_to_image_r5": 0.6264294385910034
+    }
+  }
+}

eval_flickr30k_final.json

@@ -0,0 +1,21 @@
+{
+  "checkpoint": "/sc/projects/sci-aisc/matin.mahmood/runs/hyper3_clip_vit_b_8xh100_full/checkpoint_final.pt",
+  "config": "configs/eval_common_flickr30k_8xh100_full_final.yaml",
+  "model_config": "configs/hyper3_clip_vit_b_8xh100_full.yaml",
+  "results": {
+    "flickr30k_retrieval": {
+      "i2t_r1": 79.0,
+      "i2t_r10": 97.5,
+      "i2t_r5": 94.5,
+      "image_to_text_r1": 0.79,
+      "image_to_text_r10": 0.975,
+      "image_to_text_r5": 0.945,
+      "t2i_r1": 64.80000019073486,
+      "t2i_r10": 92.35999584197998,
+      "t2i_r5": 86.79999709129333,
+      "text_to_image_r1": 0.6480000019073486,
+      "text_to_image_r10": 0.9235999584197998,
+      "text_to_image_r5": 0.8679999709129333
+    }
+  }
+}

eval_hycoclip_uncha_intersection_final.json

@@ -0,0 +1,47 @@
+{
+  "checkpoint": "/sc/projects/sci-aisc/matin.mahmood/runs/hyper3_clip_vit_b_8xh100_full/checkpoint_final.pt",
+  "config": "configs/eval_hycoclip_uncha_intersection_8xh100_full_final.yaml",
+  "model_config": "configs/hyper3_clip_vit_b_8xh100_full.yaml",
+  "results": {
+    "coco_retrieval": {
+      "i2t_r1": 50.3,
+      "i2t_r10": 84.0,
+      "i2t_r5": 75.86,
+      "image_to_text_r1": 0.503,
+      "image_to_text_r10": 0.84,
+      "image_to_text_r5": 0.7586,
+      "t2i_r1": 35.67602038383484,
+      "t2i_r10": 72.81522154808044,
+      "t2i_r5": 62.113213539123535,
+      "text_to_image_r1": 0.3567602038383484,
+      "text_to_image_r10": 0.7281522154808044,
+      "text_to_image_r5": 0.6211321353912354
+    },
+    "flickr30k_retrieval": {
+      "i2t_r1": 79.0,
+      "i2t_r10": 97.5,
+      "i2t_r5": 94.5,
+      "image_to_text_r1": 0.79,
+      "image_to_text_r10": 0.975,
+      "image_to_text_r5": 0.945,
+      "t2i_r1": 64.80000019073486,
+      "t2i_r10": 92.35999584197998,
+      "t2i_r5": 86.79999709129333,
+      "text_to_image_r1": 0.6480000019073486,
+      "text_to_image_r10": 0.9235999584197998,
+      "text_to_image_r5": 0.8679999709129333
+    },
+    "imagenet_hierarchical": {
+      "hierarchical_precision": 0.8819191582306588,
+      "hierarchical_recall": 0.8834472284937281,
+      "jaccard": 0.8284612410570121,
+      "lca": 1.98578,
+      "tie": 2.97216
+    },
+    "imagenet_zero_shot": {
+      "mean_per_class_acc_pct": 48.496,
+      "top1": 0.48496,
+      "top1_pct": 48.496
+    }
+  }
+}

eval_imagenet_final.json

@@ -0,0 +1,12 @@
+{
+  "checkpoint": "/sc/projects/sci-aisc/matin.mahmood/runs/hyper3_clip_vit_b_8xh100_full/checkpoint_final.pt",
+  "config": "configs/eval_common_imagenet_8xh100_full_final.yaml",
+  "model_config": "configs/hyper3_clip_vit_b_8xh100_full.yaml",
+  "results": {
+    "imagenet_zero_shot": {
+      "mean_per_class_acc_pct": 48.496,
+      "top1": 0.48496,
+      "top1_pct": 48.496
+    }
+  }
+}

hyper3_clip_sentence_transformers.py

@@ -0,0 +1,264 @@
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+import timm
+import torch
+import torch.nn.functional as F
+from PIL import Image
+from safetensors.torch import load_file
+from sentence_transformers.base.modules.module import Module
+from torch import nn
+from transformers import AutoTokenizer, CLIPTextConfig, CLIPTextModel
+
+try:
+    import yaml
+except ImportError as exc:  # pragma: no cover - produces a clear error for missing deps.
+    raise ImportError("Hyper3-CLIP requires pyyaml to load config.yaml") from exc
+
+
+IMAGENET_MEAN = torch.tensor([0.485, 0.456, 0.406], dtype=torch.float32).view(3, 1, 1)
+IMAGENET_STD = torch.tensor([0.229, 0.224, 0.225], dtype=torch.float32).view(3, 1, 1)
+
+
+class Hyper3CLIPSentenceTransformerModule(Module):
+    """Sentence Transformers module for the Hyper3-CLIP beta checkpoint."""
+
+    config_keys = [
+        "model_config_file",
+        "weights_file",
+        "image_size",
+        "max_text_length",
+        "normalize_output",
+    ]
+    modalities = ["text", "image"]
+
+    def __init__(
+        self,
+        model_config: dict[str, Any],
+        weights_path: str | None = None,
+        model_config_file: str = "config.yaml",
+        weights_file: str = "model.safetensors",
+        image_size: int = 224,
+        max_text_length: int = 77,
+        normalize_output: bool = True,
+    ) -> None:
+        super().__init__()
+        self.model_config_file = model_config_file
+        self.weights_file = weights_file
+        self.image_size = int(image_size)
+        self.max_text_length = int(max_text_length)
+        self.normalize_output = bool(normalize_output)
+
+        model_args = dict(model_config["model"])
+        data_args = model_config.get("data", {})
+        self.image_size = int(data_args.get("image_size", self.image_size))
+        self.max_text_length = int(data_args.get("max_text_length", self.max_text_length))
+
+        self.model = _Hyper3CLIPInference(
+            vision_backbone=model_args["vision_backbone"],
+            text_model_name=model_args["text_model_name"],
+            embed_dim=int(model_args["embed_dim"]),
+            curv_init=float(model_args.get("curv_init", 1.0)),
+            learn_curv=bool(model_args.get("learn_curv", True)),
+        )
+        if weights_path is not None:
+            state_dict = load_file(weights_path, device="cpu")
+            self.model.load_state_dict(state_dict, strict=True)
+        self.model.eval()
+        self.tokenizer = self.model.tokenizer
+
+    @classmethod
+    def load(
+        cls,
+        model_name_or_path: str,
+        subfolder: str = "",
+        token: bool | str | None = None,
+        cache_folder: str | None = None,
+        revision: str | None = None,
+        local_files_only: bool = False,
+        **kwargs: Any,
+    ) -> "Hyper3CLIPSentenceTransformerModule":
+        config = cls.load_config(
+            model_name_or_path,
+            subfolder=subfolder,
+            token=token,
+            cache_folder=cache_folder,
+            revision=revision,
+            local_files_only=local_files_only,
+        )
+        model_config_file = config.get("model_config_file", "config.yaml")
+        weights_file = config.get("weights_file", "model.safetensors")
+        model_config_path = cls.load_file_path(
+            model_name_or_path,
+            model_config_file,
+            subfolder="",
+            token=token,
+            cache_folder=cache_folder,
+            revision=revision,
+            local_files_only=local_files_only,
+        )
+        weights_path = cls.load_file_path(
+            model_name_or_path,
+            weights_file,
+            subfolder="",
+            token=token,
+            cache_folder=cache_folder,
+            revision=revision,
+            local_files_only=local_files_only,
+        )
+        if model_config_path is None:
+            raise FileNotFoundError(f"Could not find {model_config_file!r} in {model_name_or_path!r}")
+        if weights_path is None:
+            raise FileNotFoundError(f"Could not find {weights_file!r} in {model_name_or_path!r}")
+
+        with open(model_config_path, encoding="utf-8") as f:
+            model_config = yaml.safe_load(f)
+
+        return cls(
+            model_config=model_config,
+            weights_path=weights_path,
+            model_config_file=model_config_file,
+            weights_file=weights_file,
+            image_size=int(config.get("image_size", 224)),
+            max_text_length=int(config.get("max_text_length", 77)),
+            normalize_output=bool(config.get("normalize_output", True)),
+        )
+
+    def preprocess(
+        self,
+        inputs: list[Any],
+        prompt: str | None = None,
+        **kwargs: Any,
+    ) -> dict[str, torch.Tensor | str]:
+        if not inputs:
+            return {}
+
+        if all(_is_image(item) for item in inputs):
+            images = torch.stack([self._preprocess_image(item) for item in inputs])
+            return {"pixel_values": images, "modality": "image"}
+
+        if all(isinstance(item, str) for item in inputs):
+            texts = [f"{prompt or ''}{item}" for item in inputs]
+            tokens = self.tokenizer(
+                texts,
+                padding=True,
+                truncation=True,
+                max_length=self.max_text_length,
+                return_tensors="pt",
+            )
+            return {
+                "input_ids": tokens["input_ids"],
+                "attention_mask": tokens["attention_mask"],
+                "modality": "text",
+            }
+
+        raise TypeError("Hyper3-CLIP beta supports batches containing only PIL images or only text strings.")
+
+    def tokenize(self, texts: list[Any], **kwargs: Any) -> dict[str, torch.Tensor | str]:
+        return self.preprocess(texts, **kwargs)
+
+    def forward(self, features: dict[str, torch.Tensor | Any], **kwargs: Any) -> dict[str, torch.Tensor | Any]:
+        if "pixel_values" in features:
+            embeddings = self.model.encode_image(features["pixel_values"], project=False)
+        elif "input_ids" in features and "attention_mask" in features:
+            embeddings = self.model.encode_text(features["input_ids"], features["attention_mask"], project=False)
+        else:
+            raise ValueError("Expected either pixel_values or input_ids/attention_mask in features.")
+
+        embeddings = embeddings.float()
+        if self.normalize_output:
+            embeddings = F.normalize(embeddings, p=2, dim=-1)
+        features["sentence_embedding"] = embeddings
+        return features
+
+    def get_embedding_dimension(self) -> int:
+        return int(self.model.embed_dim)
+
+    def save(self, output_path: str, *args: Any, safe_serialization: bool = True, **kwargs: Any) -> None:
+        output_dir = Path(output_path)
+        output_dir.mkdir(parents=True, exist_ok=True)
+        with open(output_dir / self.config_file_name, "w", encoding="utf-8") as f:
+            json.dump(self.get_config_dict(), f, indent=2)
+
+    def _preprocess_image(self, image: Any) -> torch.Tensor:
+        if isinstance(image, np.ndarray):
+            image = Image.fromarray(image)
+        elif torch.is_tensor(image):
+            array = image.detach().cpu()
+            if array.ndim == 3 and array.shape[0] in {1, 3, 4}:
+                array = array.permute(1, 2, 0)
+            image = Image.fromarray(array.numpy())
+        if not isinstance(image, Image.Image):
+            raise TypeError(f"Expected PIL image, numpy array, or torch tensor, got {type(image)!r}")
+
+        resampling = getattr(Image, "Resampling", Image).BICUBIC
+        image = image.convert("RGB").resize((self.image_size, self.image_size), resampling)
+        array = np.asarray(image, dtype=np.float32) / 255.0
+        tensor = torch.from_numpy(array).permute(2, 0, 1)
+        return (tensor - IMAGENET_MEAN) / IMAGENET_STD
+
+
+class _Hyper3CLIPInference(nn.Module):
+    def __init__(
+        self,
+        vision_backbone: str,
+        text_model_name: str,
+        embed_dim: int,
+        curv_init: float,
+        learn_curv: bool,
+    ) -> None:
+        super().__init__()
+        self.vision_encoder = _VisionEncoder(vision_backbone)
+        self.text_encoder = _TextEncoder(text_model_name)
+        self.tokenizer = self.text_encoder.tokenizer
+        self.embed_dim = int(embed_dim)
+        self.image_proj = nn.Linear(self.vision_encoder.output_dim, self.embed_dim)
+        self.text_proj = nn.Linear(self.text_encoder.output_dim, self.embed_dim)
+        self.logit_scale = nn.Parameter(torch.tensor(1 / 0.07).log())
+        self.visual_alpha = nn.Parameter(torch.full((), self.embed_dim**-0.5).log())
+        self.textual_alpha = nn.Parameter(torch.full((), self.embed_dim**-0.5).log())
+        self.log_curv = nn.Parameter(torch.full((), curv_init).log(), requires_grad=learn_curv)
+
+    def encode_image(self, image: torch.Tensor, project: bool = False) -> torch.Tensor:
+        feats = self.image_proj(self.vision_encoder(image))
+        if project:
+            feats = feats * self.visual_alpha.exp().float()
+        return feats
+
+    def encode_text(self, input_ids: torch.Tensor, attention_mask: torch.Tensor, project: bool = False) -> torch.Tensor:
+        feats = self.text_proj(self.text_encoder(input_ids=input_ids, attention_mask=attention_mask))
+        if project:
+            feats = feats * self.textual_alpha.exp().float()
+        return feats
+
+
+class _VisionEncoder(nn.Module):
+    def __init__(self, backbone_name: str) -> None:
+        super().__init__()
+        self.backbone = timm.create_model(backbone_name, pretrained=False, num_classes=0, global_pool="avg")
+        self.output_dim = self.backbone.num_features
+
+    def forward(self, image: torch.Tensor) -> torch.Tensor:
+        return self.backbone(image)
+
+
+class _TextEncoder(nn.Module):
+    def __init__(self, model_name: str) -> None:
+        super().__init__()
+        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
+        self.backbone = CLIPTextModel(CLIPTextConfig.from_pretrained(model_name))
+        self.output_dim = self.backbone.config.hidden_size
+
+    def forward(self, input_ids: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
+        out = self.backbone(input_ids=input_ids, attention_mask=attention_mask)
+        if hasattr(out, "pooler_output") and out.pooler_output is not None:
+            return out.pooler_output
+        return out.last_hidden_state[:, 0]
+
+
+def _is_image(item: Any) -> bool:
+    return isinstance(item, Image.Image) or isinstance(item, np.ndarray) or torch.is_tensor(item)

metadata.json

@@ -0,0 +1,24 @@
+{
+  "end_time": "2026-05-06T10:53:23+00:00",
+  "env": {
+    "hostname": "gx08",
+    "rank": "0",
+    "world_size": "8"
+  },
+  "experiment_name": "hyper3-clip",
+  "final_step": 500000,
+  "job": {
+    "gpus": "0,1,2,3,4,5,6,7",
+    "node_list": "gx08",
+    "num_nodes": "1",
+    "partition": "aisc-batch",
+    "slurm_job_id": "1916964"
+  },
+  "run_id": "hyper3_clip_vit_b_8xh100_full",
+  "start_time": "2026-05-05T09:57:08+00:00",
+  "status": "completed",
+  "tags": {
+    "data": "processed_grit",
+    "model": "vit_base_patch16_224"
+  }
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b199fc5574ed85df39c3b0de0d8b5a3e998fcddd29dd181125dd2274aae98555
+size 598529912

modeling_hyper3_clip.py

@@ -0,0 +1,119 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+import numpy as np
+import timm
+import torch
+import torch.nn.functional as F
+from PIL import Image
+from torch import nn
+from transformers import CLIPTextConfig, CLIPTextModel, PreTrainedModel
+from transformers.modeling_outputs import ModelOutput
+
+from .configuration_hyper3_clip import Hyper3CLIPConfig
+
+
+IMAGENET_MEAN = torch.tensor([0.485, 0.456, 0.406], dtype=torch.float32).view(3, 1, 1)
+IMAGENET_STD = torch.tensor([0.229, 0.224, 0.225], dtype=torch.float32).view(3, 1, 1)
+
+
+@dataclass
+class Hyper3CLIPOutput(ModelOutput):
+    image_embeds: torch.FloatTensor | None = None
+    text_embeds: torch.FloatTensor | None = None
+
+
+class Hyper3CLIPModel(PreTrainedModel):
+    config_class = Hyper3CLIPConfig
+    main_input_name = "pixel_values"
+    supports_gradient_checkpointing = False
+
+    def __init__(self, config: Hyper3CLIPConfig) -> None:
+        super().__init__(config)
+        self.vision_encoder = _VisionEncoder(config.vision_backbone)
+        self.text_encoder = _TextEncoder(config.text_model_name)
+        self.embed_dim = int(config.embed_dim)
+        self.image_proj = nn.Linear(self.vision_encoder.output_dim, self.embed_dim)
+        self.text_proj = nn.Linear(self.text_encoder.output_dim, self.embed_dim)
+        self.logit_scale = nn.Parameter(torch.tensor(1 / 0.07).log())
+        self.visual_alpha = nn.Parameter(torch.full((), self.embed_dim**-0.5).log())
+        self.textual_alpha = nn.Parameter(torch.full((), self.embed_dim**-0.5).log())
+        self.log_curv = nn.Parameter(torch.full((), config.curv_init).log(), requires_grad=config.learn_curv)
+
+    def encode_image(self, pixel_values: torch.Tensor, normalize: bool = True) -> torch.Tensor:
+        embeddings = self.image_proj(self.vision_encoder(pixel_values))
+        return F.normalize(embeddings.float(), p=2, dim=-1) if normalize else embeddings
+
+    def encode_text(
+        self,
+        input_ids: torch.Tensor,
+        attention_mask: torch.Tensor,
+        normalize: bool = True,
+    ) -> torch.Tensor:
+        embeddings = self.text_proj(self.text_encoder(input_ids=input_ids, attention_mask=attention_mask))
+        return F.normalize(embeddings.float(), p=2, dim=-1) if normalize else embeddings
+
+    def preprocess_image(self, image: Any) -> torch.Tensor:
+        return _preprocess_image(image, image_size=self.config.image_size)
+
+    def forward(
+        self,
+        pixel_values: torch.Tensor | None = None,
+        input_ids: torch.Tensor | None = None,
+        attention_mask: torch.Tensor | None = None,
+        normalize: bool = True,
+        return_dict: bool | None = None,
+    ) -> Hyper3CLIPOutput | tuple[torch.Tensor | None, torch.Tensor | None]:
+        image_embeds = self.encode_image(pixel_values, normalize=normalize) if pixel_values is not None else None
+        text_embeds = None
+        if input_ids is not None:
+            if attention_mask is None:
+                attention_mask = torch.ones_like(input_ids)
+            text_embeds = self.encode_text(input_ids, attention_mask, normalize=normalize)
+
+        if return_dict is False:
+            return image_embeds, text_embeds
+        return Hyper3CLIPOutput(image_embeds=image_embeds, text_embeds=text_embeds)
+
+
+class _VisionEncoder(nn.Module):
+    def __init__(self, backbone_name: str) -> None:
+        super().__init__()
+        self.backbone = timm.create_model(backbone_name, pretrained=False, num_classes=0, global_pool="avg")
+        self.output_dim = self.backbone.num_features
+
+    def forward(self, image: torch.Tensor) -> torch.Tensor:
+        return self.backbone(image)
+
+
+class _TextEncoder(nn.Module):
+    def __init__(self, model_name: str) -> None:
+        super().__init__()
+        self.backbone = CLIPTextModel(CLIPTextConfig.from_pretrained(model_name))
+        self.output_dim = self.backbone.config.hidden_size
+
+    def forward(self, input_ids: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
+        out = self.backbone(input_ids=input_ids, attention_mask=attention_mask)
+        if hasattr(out, "pooler_output") and out.pooler_output is not None:
+            return out.pooler_output
+        return out.last_hidden_state[:, 0]
+
+
+def _preprocess_image(image: Any, image_size: int = 224) -> torch.Tensor:
+    if isinstance(image, np.ndarray):
+        image = Image.fromarray(image)
+    elif torch.is_tensor(image):
+        array = image.detach().cpu()
+        if array.ndim == 3 and array.shape[0] in {1, 3, 4}:
+            array = array.permute(1, 2, 0)
+        image = Image.fromarray(array.numpy())
+    if not isinstance(image, Image.Image):
+        raise TypeError(f"Expected PIL image, numpy array, or torch tensor, got {type(image)!r}")
+
+    resampling = getattr(Image, "Resampling", Image).BICUBIC
+    image = image.convert("RGB").resize((image_size, image_size), resampling)
+    array = np.asarray(image, dtype=np.float32) / 255.0
+    tensor = torch.from_numpy(array).permute(2, 0, 1)
+    return (tensor - IMAGENET_MEAN) / IMAGENET_STD

modules.json

@@ -0,0 +1,8 @@
+[
+  {
+    "idx": 0,
+    "name": "0_Hyper3CLIP",
+    "path": "0_Hyper3CLIP",
+    "type": "hyper3_clip_sentence_transformers.Hyper3CLIPSentenceTransformerModule"
+  }
+]

requirements.txt

@@ -0,0 +1,7 @@
+sentence-transformers>=5.5.1
+haystack-ai>=2.30.1
+transformers>=4.49.0
+timm>=1.0.0
+safetensors>=0.4.0
+pyyaml>=6.0.0
+Pillow>=10.0.0