Replace with Hyper3-CLIP beta hier-beta scratch checkpoint

.hfignore

@@ -1,3 +0,0 @@
-checkpoint_step_500000.pt
-*.tmp
-*.partial

0_Hyper3CLIP/config.json

@@ -1,7 +0,0 @@
-{
-  "model_config_file": "config.yaml",
-  "weights_file": "model.safetensors",
-  "image_size": 224,
-  "max_text_length": 77,
-  "normalize_output": true
-}

LICENSE

@@ -1,49 +1,3 @@
-OpenMDW License Agreement, version 1.0 (OpenMDW-1.0)
+OpenMDW-1.0
 
-By exercising rights granted to you under this agreement, you accept and agree
-to its terms.
-
-As used in this agreement, "Model Materials" means the materials provided to
-you under this agreement, consisting of: (1) one or more machine learning
-models (including architecture and parameters); and (2) all related artifacts
-(including associated data, documentation and software) that are provided to
-you hereunder.
-
-Subject to your compliance with this agreement, permission is hereby granted,
-free of charge, to deal in the Model Materials without restriction, including
-under all copyright, patent, database, and trade secret rights included or
-embodied therein.
-
-If you distribute any portion of the Model Materials, you shall retain in your
-distribution (1) a copy of this agreement, and (2) all copyright notices and
-other notices of origin included in the Model Materials that are applicable to
-your distribution.
-
-If you file, maintain, or voluntarily participate in a lawsuit against any
-person or entity asserting that the Model Materials directly or indirectly
-infringe any patent, then all rights and grants made to you hereunder are
-terminated, unless that lawsuit was in response to a corresponding lawsuit
-first brought against you.
-
-This agreement does not impose any restrictions or obligations with respect to
-any use, modification, or sharing of any outputs generated by using the Model
-Materials.
-
-THE MODEL MATERIALS ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
-OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE, TITLE, NONINFRINGEMENT, ACCURACY, OR THE
-ABSENCE OF LATENT OR OTHER DEFECTS OR ERRORS, WHETHER OR NOT DISCOVERABLE, ALL
-TO THE GREATEST EXTENT PERMISSIBLE UNDER APPLICABLE LAW.
-
-YOU ARE SOLELY RESPONSIBLE FOR (1) CLEARING RIGHTS OF OTHER PERSONS THAT MAY
-APPLY TO THE MODEL MATERIALS OR ANY USE THEREOF, INCLUDING WITHOUT LIMITATION
-ANY PERSON'S COPYRIGHTS OR OTHER RIGHTS INCLUDED OR EMBODIED IN THE MODEL
-MATERIALS; (2) OBTAINING ANY NECESSARY CONSENTS, PERMISSIONS OR OTHER RIGHTS
-REQUIRED FOR ANY USE OF THE MODEL MATERIALS; OR (3) PERFORMING ANY DUE
-DILIGENCE OR UNDERTAKING ANY OTHER INVESTIGATIONS INTO THE MODEL MATERIALS OR
-ANYTHING INCORPORATED OR EMBODIED THEREIN.
-
-IN NO EVENT SHALL THE PROVIDERS OF THE MODEL MATERIALS BE LIABLE FOR ANY CLAIM,
-DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
-OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE MODEL MATERIALS, THE
-USE THEREOF OR OTHER DEALINGS THEREIN.
+See the original OpenMDW-1.0 license terms for permitted use and redistribution.

NOTICE

@@ -1,16 +1,4 @@
 Hyper3-CLIP beta
 Copyright hyper³labs.
 
-Hyper3-CLIP beta was created by hyper³labs.
-
-This notice identifies the original source of the model materials. Redistributions
-of this model or derivative model materials should preserve this notice, the
-accompanying LICENSE file, and the original model card when practical.
-
-hyper³labs and Hyper3-CLIP are names associated with hyper³labs. No trademark
-license is granted. Modified or derived checkpoints must not use the hyper³labs
-or Hyper3-CLIP names in a way that suggests they are official hyper³labs releases
-or endorsed by hyper³labs.
-
-Please cite and link to the original hyper³labs model repository when publishing
-benchmarks, papers, derivative checkpoints, or public demos based on this model.
+Hyper3-CLIP beta was created by hyper³labs. Modified or derived checkpoints must not imply endorsement by hyper³labs.

README.md

@@ -1,6 +1,5 @@
 ---
 license: openmdw-1.0
-library_name: sentence-transformers
 pipeline_tag: feature-extraction
 tags:
 - vision-language
@@ -8,180 +7,72 @@ tags:
 - image-text-retrieval
 - hyperbolic-embeddings
 - clip
-- sentence-transformers
-- transformers
-- haystack
-- safetensors
 - research
 - scratch-training
+- hier-beta
+- argent
 ---
 
 # Hyper3-CLIP beta
 
-Hyper3-CLIP beta is an open-weight hyperbolic vision-language checkpoint from
-hyper³labs. It places image and text representations in a Lorentz space and was
-trained with compositional entailment constraints for hierarchy-sensitive
-image-text retrieval.
+Hyper3-CLIP beta is the hyper³labs ViT-B scratch checkpoint trained with the
+hier-beta ARGENT objective.
 
-This beta release is intended as an open baseline and research artifact.
+This repository publishes the raw PyTorch training checkpoint for the completed
+500k-step paper-scratch run. It is not the older Hyper3-CLIP v0.5
+SentenceTransformers package.
 
-## Model
+## Artifact
 
-- Architecture: ViT-B scale vision-language model
+- Checkpoint: `checkpoint_final.pt`
+- Config: `config.yaml`
+- Training metadata: `metadata.json`
+- Run: `hyper3_vitb_clip_uncha_hier_beta_argent_mp5_paper_scratch_8x500k_s31`
+- Objective: `uncha` with `uncha_entailment_loss: hier_beta_argent`
 - Vision backbone: `vit_base_patch16_224`
-- Text backbone architecture/tokenizer: `openai/clip-vit-base-patch32`
+- Vision pretrained: `false`
+- Text model architecture/tokenizer: `openai/clip-vit-base-patch32`
+- Text pretrained: `false`
 - Embedding dimension: 512
 - Training steps: 500,000
 - Global batch size: 768
-- Weights artifact: `model.safetensors`
 
-The original full training checkpoint included optimizer, scheduler, AMP scaler,
-RNG state, config, and step metadata. This repository publishes the weights-only
-`model.safetensors` artifact for inference and downstream research from the
-ViT-B scratch training run.
-
-## Quick Start: Sentence Transformers
+## Evaluation
 
-The default way to use this checkpoint is through Sentence Transformers. The
-adapter in this repository returns 512-dimensional L2-normalized tangent-space
-embeddings for standard cosine/dot-product vector stores.
+The `eval/` directory includes the paper-comparable full benchmark table and the
+raw wide summary row used for the current model comparison.
 
-Install the runtime dependencies:
+Headline row from the local full eval:
 
-```bash
-pip install "sentence-transformers>=5.5.1" timm safetensors pyyaml Pillow
-```
+- ImageNet top-1: 46.984%
+- COCO I2T/T2I R@10: 84.30 / 73.19
+- Flickr I2T/T2I R@10: 97.60 / 91.44
+- WordNet hierarchy: TIE 3.1597, LCA 2.0786, Jaccard 0.8179
+- PEP AUC/AP: 96.07 / 69.36
 
-If you are using the gated Hugging Face repository from a fresh machine, accept
-access on the model page and set `HF_TOKEN`.
+The checkpoint is strong on retrieval in the paper-comparable table, but weak on
+several flat/fine-grained zero-shot datasets such as Food101, CUB, Flowers102,
+Cars, and Aircraft. Treat this release as a research checkpoint, not a polished
+production model.
 
-```python
-from PIL import Image
-from sentence_transformers import SentenceTransformer
-
-model = SentenceTransformer("hyper3labs/hyper3-clip-beta", trust_remote_code=True)
-
-image_embedding = model.encode([Image.open("/path/to/image.jpg")], normalize_embeddings=True)
-text_embedding = model.encode(["machined metal part"], normalize_embeddings=True)
-```
+## Loading
 
-## Transformers
+This is a raw training checkpoint. Use the hyper³labs `hyper3-clip` codebase and
+the included `config.yaml` to instantiate the model, then load
+`checkpoint_final.pt`.
 
 ```python
-from PIL import Image
 import torch
-from transformers import AutoModel, AutoTokenizer
-
-model = AutoModel.from_pretrained("hyper3labs/hyper3-clip-beta", trust_remote_code=True).eval()
-tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32")
-
-image = model.preprocess_image(Image.open("/path/to/image.jpg")).unsqueeze(0)
-text = tokenizer(
-    ["machined metal part"],
-    padding=True,
-    truncation=True,
-    max_length=model.config.max_text_length,
-    return_tensors="pt",
-)
-
-with torch.no_grad():
-    outputs = model(
-        pixel_values=image,
-        input_ids=text["input_ids"],
-        attention_mask=text["attention_mask"],
-    )
-
-image_embedding = outputs.image_embeds
-text_embedding = outputs.text_embeds
-```
-
-<details>
-<summary>Haystack image retrieval pipeline</summary>
 
-For indexing images in a Haystack retrieval pipeline, use
-`SentenceTransformersDocumentImageEmbedder` with image paths in
-`Document.meta["file_path"]`, paired with `SentenceTransformersTextEmbedder` for
-text queries.
-
-```bash
-pip install "haystack-ai>=2.30.1" "sentence-transformers>=5.5.1" timm safetensors pyyaml Pillow
+checkpoint = torch.load("checkpoint_final.pt", map_location="cpu", weights_only=False)
+state_dict = checkpoint.get("model", checkpoint)
 ```
 
-```python
-from haystack import Document
-from haystack.components.embedders import SentenceTransformersTextEmbedder
-from haystack.components.embedders.image import SentenceTransformersDocumentImageEmbedder
-
-model_id = "hyper3labs/hyper3-clip-beta"
-
-documents = [
-    Document(
-        content="front view of a machined metal part",
-        meta={"file_path": "/path/to/image.jpg"},
-    )
-]
-
-image_embedder = SentenceTransformersDocumentImageEmbedder(
-    model=model_id,
-    trust_remote_code=True,
-    batch_size=8,
-    normalize_embeddings=True,
-)
-documents = image_embedder.run(documents=documents)["documents"]
-
-text_embedder = SentenceTransformersTextEmbedder(
-    model=model_id,
-    trust_remote_code=True,
-    normalize_embeddings=True,
-)
-query_embedding = text_embedder.run("machined metal part")["embedding"]
-```
-
-</details>
-
-## Evaluation
-
-The numbers below use the official evaluator convention for R@10. Higher is
-better except for TIE and LCA.
-
-| Model | Comparable setting | ImageNet top-1 | COCO text R@10 | COCO image R@10 | Flickr text R@10 | Flickr image R@10 | TIE | LCA | Jaccard | H-Prec | H-Rec |
-|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
-| MERU-B/16 | same-family baseline | 40.1 | 82.0 | 68.6 | 96.2 | 90.0 | 3.630 | 2.220 | 0.780 | 0.850 | 0.850 |
-| HyCoCLIP-B/16 | official checkpoint | 45.8 | 82.0 | 69.3 | 95.4 | 90.3 | 3.172 | 2.047 | 0.814 | 0.874 | 0.874 |
-| UNCHA-B/16 | official checkpoint | 48.8 | 82.6 | 71.0 | 95.9 | 91.2 | 2.945 | 1.961 | 0.828 | 0.883 | 0.884 |
-| PHyCLIP-B/16 | related reported result | 44.4 | 80.4 | 68.7 | 95.6 | 89.9 | 3.285 | 2.088 | 0.807 | 0.868 | 0.868 |
-| Hyper3-CLIP beta | this release | 48.5 | 84.0 | 72.8 | 97.5 | 92.4 | 2.972 | 1.986 | 0.828 | 0.882 | 0.883 |
-
-Raw evaluation files are included:
-
-- `eval_coco_karpathy_final.json`
-- `eval_flickr30k_final.json`
-- `eval_imagenet_final.json`
-- `eval_hycoclip_uncha_intersection_final.json`
-
 ## License And Attribution
 
-The model materials in this repository are released under OpenMDW-1.0. See
-`LICENSE`.
-
-Redistributions should preserve `NOTICE`, `LICENSE`, and the original model card
-when practical. Modified or derived checkpoints should use a distinct name and
-must not imply endorsement by hyper³labs.
+The model materials in this repository are released under OpenMDW-1.0.
+Redistributions should preserve `NOTICE`, `LICENSE`, and the model card when
+practical.
 
 Please cite and link to the original hyper³labs model repository when publishing
 benchmarks, papers, derivative checkpoints, or public demos based on this model.
-
-## Intended Use
-
-This release is intended for:
-
-- hierarchy-sensitive image-text retrieval research
-- zero-shot and retrieval evaluation
-- multimodal embedding baselines
-- downstream experiments with hyperbolic representation learning
-
-This model has not been validated for safety-critical use.
-
-## Citation
-
-If you use Hyper3-CLIP beta, cite the original model repository and hyper³labs.

SHA256SUMS

@@ -1,19 +1,7 @@
-127022078882a067d3729185679645a173b7831f22454e02f1d48bf4caca454e  ./.hfignore
-00858698a701445ca5e8e0a753832834d605ae5ce0fc029cc1f4b41e8bf334be  ./0_Hyper3CLIP/config.json
-22d5488faff30b2e34e934779d2a6f033712ec6d1eaaee7e774e9cbe0c3c7587  ./LICENSE
-bc06204c8f542252f901c42929aa964a670f65837d961c947e2b404e9ada57e9  ./NOTICE
-2524472f3efc94e5215beac98fdc8b3e7ff1296e948e0259a3fe09ac44b43041  ./README.md
-88e0cd5e7a6f13394196662fda3291058ddd5b738d8c57a3c05f635c366ae9fd  ./config.json
-9639dadcd1152dc059ff7b158b3d9a87395ea9ef99917f0e254b79f93fe72efa  ./config.yaml
-bc149f4539e0fabd8d4392324f8d4e3d13a3bea0c41cc504349c51be3215bac2  ./config_sentence_transformers.json
-7ee8d05bf0e690b40930f561c51ada9d54badc82602f279e9e1222fc51d89c0e  ./configuration_hyper3_clip.py
-d4ec8249d963614c6bea43812b71157b5d01be7aeee23ea828d77fca8bb26314  ./eval_coco_karpathy_final.json
-4bff22e2f2ea6a5451847577f05238002888ad2670be8a4f788203879dd2bd1d  ./eval_flickr30k_final.json
-589fd4d0711b266aa77255737d767f184006fac03f65a69ed1a7632deee812b0  ./eval_hycoclip_uncha_intersection_final.json
-b6a68d90cae001ccc422220d371d717d30cce6d993e8c45f36a6a8c7c2dd38b3  ./eval_imagenet_final.json
-13b30904fbfdcfad9b968ba1bd5853fe5de0d3bf0dccfa9d3baab0f82c6ea5f4  ./hyper3_clip_sentence_transformers.py
-c86353fda268117808086af62808f54896c1ce954f9a653cafa84b97172effd2  ./metadata.json
-b199fc5574ed85df39c3b0de0d8b5a3e998fcddd29dd181125dd2274aae98555  ./model.safetensors
-605bda2e201ef962387cac2ed64d8735b82b29da57c24ded54ced5f9e3ece04f  ./modeling_hyper3_clip.py
-14c3e2e85d941bac307a35600d1ad8377fddd9cde85b0d706c19a623b9d311d2  ./modules.json
-8ace6250857b88214db892908ee8b85d80da96dd9f7020be8bfd4de327326e31  ./requirements.txt
+1cb17ef6982d024531296071cc43af7195871a0c3f26231d527ef297547d940c  LICENSE
+26e93e9b7fdccd87e8dc93fcd1488f43ef48bd8f99af225354294b0ee75bf5fb  NOTICE
+c22ec054cda1b2391cf23b987827075d6668ab3c78bab4981d5f9d47c218691a  README.md
+b6011edf5a198e57a77a029477eee092fd09e205a86d4010b78695adba79edb3  checkpoint_final.pt
+b02548b0f2b7b1e914a003946e434b653353569dbdadcde7d222c6dd27a1332d  config.yaml
+e23c2cdec7b1ffefecfc5779529f9a5ae340fd4caa784e737e36652ac84e6602  eval/summary_wide.csv
+48a11aa060bdfd87d980a6914a801c7c1f8fde97e0ba9c6e6c476bb48f50ad7f  metadata.json

model.safetensors → checkpoint_final.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b199fc5574ed85df39c3b0de0d8b5a3e998fcddd29dd181125dd2274aae98555
-size 598529912
+oid sha256:b6011edf5a198e57a77a029477eee092fd09e205a86d4010b78695adba79edb3
+size 1884510012

config.json

@@ -1,19 +0,0 @@
-{
-  "model_type": "hyper3_clip",
-  "architectures": [
-    "Hyper3CLIPModel"
-  ],
-  "auto_map": {
-    "AutoConfig": "configuration_hyper3_clip.Hyper3CLIPConfig",
-    "AutoModel": "modeling_hyper3_clip.Hyper3CLIPModel"
-  },
-  "vision_backbone": "vit_base_patch16_224",
-  "text_model_name": "openai/clip-vit-base-patch32",
-  "embed_dim": 512,
-  "curv_init": 1.0,
-  "learn_curv": true,
-  "image_size": 224,
-  "max_text_length": 77,
-  "torch_dtype": "float32",
-  "transformers_version": "4.49.0"
-}

config.yaml

@@ -1,17 +1,54 @@
 project:
   name: hyper3-clip
-  experiment: hyper3_clip_vit_b_8xh100_full
-seed: 7
-output_dir: /sc/projects/sci-aisc/matin.mahmood/runs/hyper3_clip_vit_b_8xh100_full
+  experiment: hyper3_vitb_clip_uncha_hier_beta_argent_mp5_paper_scratch_8x500k_s31
+seed: 31
+output_dir: /sc/projects/sci-aisc/matin.mahmood/runs/hyper3_vitb_hierbeta_argent_mp5_paper_scratch_500k_v1/hyper3_vitb_clip_uncha_hier_beta_argent_mp5_paper_scratch_8x500k_s31
 model:
+  objective: uncha
   vision_backbone: vit_base_patch16_224
+  vision_pretrained: false
+  vision_global_pool: token
+  vision_use_sincos2d_pos: true
+  vision_timm_norm_layer: layer_norm
   text_model_name: openai/clip-vit-base-patch32
+  text_pretrained: false
+  text_pooling: auto
   embed_dim: 512
   curv_init: 1.0
   learn_curv: true
   entail_weight: 0.2
   inter_aperture_scale: 0.7
   intra_aperture_scale: 1.2
+  uncha_piecewise_factor: 0.1
+  uncha_calibration_alpha: 10.0
+  uncha_stop_grad_calibration: true
+  uncha_entailment_geometry: lorentz
+  uncha_aggregate_weight: 0.0
+  uncha_entailment_loss: hier_beta_argent
+  uncha_argent_beta: 1.0
+  uncha_argent_norm_weight: 0.1
+  uncha_argent_aux_weight: 0.5
+  uncha_argent_aggregation: uncha
+  uncha_part_weight_power: 0.0
+  uncha_contrastive_loss: ce
+  uncha_sigmoid_bias_init: -10.0
+  uncha_sigmoid_negative_weight: 1.0
+  uncha_part_quality_mode: none
+  uncha_part_quality_topk: 5
+  uncha_part_quality_temperature: 4.0
+  uncha_entailment_warmup_steps: 0
+  uncha_global_local_mode: repeat
+  beta_clip_global_weight: 0.0
+  beta_clip_weight: 0.0
+  beta_clip_beta: 0.5
+  beta_clip_similarity: dot
+  beta_clip_num_heads: 8
+  beta_clip_mlp_ratio: 4.0
+  beta_clip_drop_cls_token: true
+  fuse_beta_query_encoder_forwards: true
+  group_beta_query_pooling: true
+  beta_clip_variant: ce
+  phyclip_product_metric: l1
 training:
   total_steps: 500000
   global_batch_size: 768
@@ -28,18 +65,36 @@ training:
   amp: true
   max_grad_norm: 1.0
   resume: true
+  resume_from: null
+  resume_from_env: RESUME_FROM_CHECKPOINT
+  find_unused_parameters: true
 optimizer:
   no_decay_params:
   - logit_scale
+  - global_logit_scale
+  - local_logit_scale
+  - global_local_logit_scale
   - visual_alpha
   - textual_alpha
   - log_curv
+  - global_logit_bias
+  - local_logit_bias
+  - global_local_logit_bias
 data:
   type: processed_grit
-  part_sampling: random_one
+  part_sampling: all
+  max_parts: 5
+  train_transform: tight_crop_color_jitter_gray
   tarfiles:
   - /sc/projects/sci-aisc/matin.mahmood/datasets/hycoclip/train/GRIT/processed/*.tar
   shuffle_buffer: 4000
   image_size: 224
   max_text_length: 77
   num_workers: 8
+  image_normalization: imagenet
+  beta_clip:
+    enabled: true
+    max_sentences: 5
+    max_phrases: 30
+    max_queries_per_image: 6
+    use_part_texts: true

config_sentence_transformers.json

@@ -1,7 +0,0 @@
-{
-  "__version__": {
-    "sentence_transformers": "5.5.1"
-  },
-  "model_type": "SentenceTransformer",
-  "similarity_fn_name": "cosine"
-}

configuration_hyper3_clip.py

@@ -1,27 +0,0 @@
-from __future__ import annotations
-
-from transformers import PretrainedConfig
-
-
-class Hyper3CLIPConfig(PretrainedConfig):
-    model_type = "hyper3_clip"
-
-    def __init__(
-        self,
-        vision_backbone: str = "vit_base_patch16_224",
-        text_model_name: str = "openai/clip-vit-base-patch32",
-        embed_dim: int = 512,
-        curv_init: float = 1.0,
-        learn_curv: bool = True,
-        image_size: int = 224,
-        max_text_length: int = 77,
-        **kwargs,
-    ) -> None:
-        super().__init__(**kwargs)
-        self.vision_backbone = vision_backbone
-        self.text_model_name = text_model_name
-        self.embed_dim = int(embed_dim)
-        self.curv_init = float(curv_init)
-        self.learn_curv = bool(learn_curv)
-        self.image_size = int(image_size)
-        self.max_text_length = int(max_text_length)

eval/summary_wide.csv

@@ -0,0 +1,2 @@
+model_id,aircraft_zero_shot.mean_per_class_acc_pct,aircraft_zero_shot.num_classes,aircraft_zero_shot.num_images,aircraft_zero_shot.top1,aircraft_zero_shot.top1_pct,caltech101_zero_shot.mean_per_class_acc_pct,caltech101_zero_shot.num_classes,caltech101_zero_shot.num_images,caltech101_zero_shot.top1,caltech101_zero_shot.top1_pct,cars_zero_shot.mean_per_class_acc_pct,cars_zero_shot.num_classes,cars_zero_shot.num_images,cars_zero_shot.top1,cars_zero_shot.top1_pct,cifar100_zero_shot.mean_per_class_acc_pct,cifar100_zero_shot.num_classes,cifar100_zero_shot.num_images,cifar100_zero_shot.top1,cifar100_zero_shot.top1_pct,cifar10_zero_shot.mean_per_class_acc_pct,cifar10_zero_shot.num_classes,cifar10_zero_shot.num_images,cifar10_zero_shot.top1,cifar10_zero_shot.top1_pct,coco_karpathy_retrieval.i2t_r1,coco_karpathy_retrieval.i2t_r10,coco_karpathy_retrieval.i2t_r5,coco_karpathy_retrieval.image_to_i2t_r1,coco_karpathy_retrieval.image_to_i2t_r10,coco_karpathy_retrieval.image_to_i2t_r5,coco_karpathy_retrieval.t2i_r1,coco_karpathy_retrieval.t2i_r10,coco_karpathy_retrieval.t2i_r5,coco_karpathy_retrieval.text_to_t2i_r1,coco_karpathy_retrieval.text_to_t2i_r10,coco_karpathy_retrieval.text_to_t2i_r5,country211_zero_shot.mean_per_class_acc_pct,country211_zero_shot.num_classes,country211_zero_shot.num_images,country211_zero_shot.top1,country211_zero_shot.top1_pct,cub_zero_shot.mean_per_class_acc_pct,cub_zero_shot.num_classes,cub_zero_shot.num_images,cub_zero_shot.top1,cub_zero_shot.top1_pct,dtd_zero_shot.mean_per_class_acc_pct,dtd_zero_shot.num_classes,dtd_zero_shot.num_images,dtd_zero_shot.top1,dtd_zero_shot.top1_pct,eurosat_zero_shot.mean_per_class_acc_pct,eurosat_zero_shot.num_classes,eurosat_zero_shot.num_images,eurosat_zero_shot.top1,eurosat_zero_shot.top1_pct,flickr30k_retrieval.i2t_r1,flickr30k_retrieval.i2t_r10,flickr30k_retrieval.i2t_r5,flickr30k_retrieval.image_to_i2t_r1,flickr30k_retrieval.image_to_i2t_r10,flickr30k_retrieval.image_to_i2t_r5,flickr30k_retrieval.t2i_r1,flickr30k_retrieval.t2i_r10,flickr30k_retrieval.t2i_r5,flickr30k_retrieval.text_to_t2i_r1,flickr30k_retrieval.text_to_t2i_r10,flickr30k_retrieval.text_to_t2i_r5,flowers_zero_shot.mean_per_class_acc_pct,flowers_zero_shot.num_classes,flowers_zero_shot.num_images,flowers_zero_shot.top1,flowers_zero_shot.top1_pct,food101_zero_shot.mean_per_class_acc_pct,food101_zero_shot.num_classes,food101_zero_shot.num_images,food101_zero_shot.top1,food101_zero_shot.top1_pct,imagenet_hierarchical.hierarchical_precision,imagenet_hierarchical.hierarchical_recall,imagenet_hierarchical.jaccard,imagenet_hierarchical.lca,imagenet_hierarchical.num_images,imagenet_hierarchical.tie,imagenet_zero_shot.mean_per_class_acc_pct,imagenet_zero_shot.num_classes,imagenet_zero_shot.num_images,imagenet_zero_shot.top1,imagenet_zero_shot.top1_pct,pep_entailment.auc_roc,pep_entailment.auc_roc_pct,pep_entailment.average_precision,pep_entailment.average_precision_pct,pep_entailment.mean_negative_score,pep_entailment.mean_positive_score,pep_entailment.num_negative_pairs,pep_entailment.num_pairs,pep_entailment.num_positive_pairs,pep_entailment.num_samples,pets_zero_shot.mean_per_class_acc_pct,pets_zero_shot.num_classes,pets_zero_shot.num_images,pets_zero_shot.top1,pets_zero_shot.top1_pct,resisc45_zero_shot.mean_per_class_acc_pct,resisc45_zero_shot.num_classes,resisc45_zero_shot.num_images,resisc45_zero_shot.top1,resisc45_zero_shot.top1_pct,stl10_zero_shot.mean_per_class_acc_pct,stl10_zero_shot.num_classes,stl10_zero_shot.num_images,stl10_zero_shot.top1,stl10_zero_shot.top1_pct,sun397_zero_shot.mean_per_class_acc_pct,sun397_zero_shot.num_classes,sun397_zero_shot.num_images,sun397_zero_shot.top1,sun397_zero_shot.top1_pct
+hier_beta_argent_vit_b_paper_scratch_500k_s31,4.33065953654189,100.0,3333.0,0.043504350435043505,4.35043504350435,83.41844657464664,102.0,6084.0,0.8091715976331361,80.9171597633136,16.12101061121831,196.0,8041.0,0.16055217012809353,16.055217012809354,65.64,100.0,10000.0,0.6564,65.64,91.13999999999999,10.0,10000.0,0.9114,91.14,50.86000000000001,84.3,75.68,0.5086,0.843,0.7568,36.617353558540344,73.19072484970093,62.99480199813843,0.36617353558540344,0.7319072484970093,0.6299480199813843,6.056872037914692,211.0,21100.0,0.06056872037914692,6.056872037914692,1.5132183908045977,200.0,5794.0,0.015360717984121504,1.5360717984121504,29.414893617021278,47.0,1880.0,0.29414893617021276,29.414893617021278,28.52,10.0,5000.0,0.2852,28.52,78.10000000000001,97.6,95.39999999999999,0.781,0.976,0.954,63.80000114440918,91.43999814987183,86.39999628067017,0.6380000114440918,0.9143999814987183,0.8639999628067017,2.565877951202066,102.0,6149.0,0.03691657180029273,3.691657180029273,5.687128712871287,101.0,25250.0,0.05687128712871287,5.687128712871288,0.8737958987678984,0.875414806637806,0.8179272225669888,2.0786,50000.0,3.1597,46.983999999999995,1000.0,50000.0,0.46984,46.983999999999995,0.96066314625,96.066314625,0.6936434133338044,69.36434133338044,0.11403495481073857,0.3067990660965443,100000.0,104000.0,4000.0,1000.0,62.566094514390926,37.0,3669.0,0.628236576723903,62.8236576723903,48.94444444444445,45.0,25200.0,0.48944444444444446,48.94444444444444,96.5625,10.0,8000.0,0.965625,96.5625,59.38539042821159,397.0,19850.0,0.5938539042821158,59.38539042821158

eval_coco_karpathy_final.json

@@ -1,21 +0,0 @@
-{
-  "checkpoint": "/sc/projects/sci-aisc/matin.mahmood/runs/hyper3_clip_vit_b_8xh100_full/checkpoint_final.pt",
-  "config": "configs/eval_common_coco_karpathy_8xh100_full_final.yaml",
-  "model_config": "configs/hyper3_clip_vit_b_8xh100_full.yaml",
-  "results": {
-    "coco_karpathy_retrieval": {
-      "i2t_r1": 50.260000000000005,
-      "i2t_r10": 83.96000000000001,
-      "i2t_r5": 76.02,
-      "image_to_text_r1": 0.5026,
-      "image_to_text_r10": 0.8396,
-      "image_to_text_r5": 0.7602,
-      "t2i_r1": 36.93722486495972,
-      "t2i_r10": 73.15473556518555,
-      "t2i_r5": 62.64294385910034,
-      "text_to_image_r1": 0.36937224864959717,
-      "text_to_image_r10": 0.7315473556518555,
-      "text_to_image_r5": 0.6264294385910034
-    }
-  }
-}

eval_flickr30k_final.json

@@ -1,21 +0,0 @@
-{
-  "checkpoint": "/sc/projects/sci-aisc/matin.mahmood/runs/hyper3_clip_vit_b_8xh100_full/checkpoint_final.pt",
-  "config": "configs/eval_common_flickr30k_8xh100_full_final.yaml",
-  "model_config": "configs/hyper3_clip_vit_b_8xh100_full.yaml",
-  "results": {
-    "flickr30k_retrieval": {
-      "i2t_r1": 79.0,
-      "i2t_r10": 97.5,
-      "i2t_r5": 94.5,
-      "image_to_text_r1": 0.79,
-      "image_to_text_r10": 0.975,
-      "image_to_text_r5": 0.945,
-      "t2i_r1": 64.80000019073486,
-      "t2i_r10": 92.35999584197998,
-      "t2i_r5": 86.79999709129333,
-      "text_to_image_r1": 0.6480000019073486,
-      "text_to_image_r10": 0.9235999584197998,
-      "text_to_image_r5": 0.8679999709129333
-    }
-  }
-}

eval_hycoclip_uncha_intersection_final.json

@@ -1,47 +0,0 @@
-{
-  "checkpoint": "/sc/projects/sci-aisc/matin.mahmood/runs/hyper3_clip_vit_b_8xh100_full/checkpoint_final.pt",
-  "config": "configs/eval_hycoclip_uncha_intersection_8xh100_full_final.yaml",
-  "model_config": "configs/hyper3_clip_vit_b_8xh100_full.yaml",
-  "results": {
-    "coco_retrieval": {
-      "i2t_r1": 50.3,
-      "i2t_r10": 84.0,
-      "i2t_r5": 75.86,
-      "image_to_text_r1": 0.503,
-      "image_to_text_r10": 0.84,
-      "image_to_text_r5": 0.7586,
-      "t2i_r1": 35.67602038383484,
-      "t2i_r10": 72.81522154808044,
-      "t2i_r5": 62.113213539123535,
-      "text_to_image_r1": 0.3567602038383484,
-      "text_to_image_r10": 0.7281522154808044,
-      "text_to_image_r5": 0.6211321353912354
-    },
-    "flickr30k_retrieval": {
-      "i2t_r1": 79.0,
-      "i2t_r10": 97.5,
-      "i2t_r5": 94.5,
-      "image_to_text_r1": 0.79,
-      "image_to_text_r10": 0.975,
-      "image_to_text_r5": 0.945,
-      "t2i_r1": 64.80000019073486,
-      "t2i_r10": 92.35999584197998,
-      "t2i_r5": 86.79999709129333,
-      "text_to_image_r1": 0.6480000019073486,
-      "text_to_image_r10": 0.9235999584197998,
-      "text_to_image_r5": 0.8679999709129333
-    },
-    "imagenet_hierarchical": {
-      "hierarchical_precision": 0.8819191582306588,
-      "hierarchical_recall": 0.8834472284937281,
-      "jaccard": 0.8284612410570121,
-      "lca": 1.98578,
-      "tie": 2.97216
-    },
-    "imagenet_zero_shot": {
-      "mean_per_class_acc_pct": 48.496,
-      "top1": 0.48496,
-      "top1_pct": 48.496
-    }
-  }
-}

eval_imagenet_final.json

@@ -1,12 +0,0 @@
-{
-  "checkpoint": "/sc/projects/sci-aisc/matin.mahmood/runs/hyper3_clip_vit_b_8xh100_full/checkpoint_final.pt",
-  "config": "configs/eval_common_imagenet_8xh100_full_final.yaml",
-  "model_config": "configs/hyper3_clip_vit_b_8xh100_full.yaml",
-  "results": {
-    "imagenet_zero_shot": {
-      "mean_per_class_acc_pct": 48.496,
-      "top1": 0.48496,
-      "top1_pct": 48.496
-    }
-  }
-}

hyper3_clip_sentence_transformers.py

@@ -1,264 +0,0 @@
-from __future__ import annotations
-
-import json
-from pathlib import Path
-from typing import Any
-
-import numpy as np
-import timm
-import torch
-import torch.nn.functional as F
-from PIL import Image
-from safetensors.torch import load_file
-from sentence_transformers.base.modules.module import Module
-from torch import nn
-from transformers import AutoTokenizer, CLIPTextConfig, CLIPTextModel
-
-try:
-    import yaml
-except ImportError as exc:  # pragma: no cover - produces a clear error for missing deps.
-    raise ImportError("Hyper3-CLIP requires pyyaml to load config.yaml") from exc
-
-
-IMAGENET_MEAN = torch.tensor([0.485, 0.456, 0.406], dtype=torch.float32).view(3, 1, 1)
-IMAGENET_STD = torch.tensor([0.229, 0.224, 0.225], dtype=torch.float32).view(3, 1, 1)
-
-
-class Hyper3CLIPSentenceTransformerModule(Module):
-    """Sentence Transformers module for the Hyper3-CLIP beta checkpoint."""
-
-    config_keys = [
-        "model_config_file",
-        "weights_file",
-        "image_size",
-        "max_text_length",
-        "normalize_output",
-    ]
-    modalities = ["text", "image"]
-
-    def __init__(
-        self,
-        model_config: dict[str, Any],
-        weights_path: str | None = None,
-        model_config_file: str = "config.yaml",
-        weights_file: str = "model.safetensors",
-        image_size: int = 224,
-        max_text_length: int = 77,
-        normalize_output: bool = True,
-    ) -> None:
-        super().__init__()
-        self.model_config_file = model_config_file
-        self.weights_file = weights_file
-        self.image_size = int(image_size)
-        self.max_text_length = int(max_text_length)
-        self.normalize_output = bool(normalize_output)
-
-        model_args = dict(model_config["model"])
-        data_args = model_config.get("data", {})
-        self.image_size = int(data_args.get("image_size", self.image_size))
-        self.max_text_length = int(data_args.get("max_text_length", self.max_text_length))
-
-        self.model = _Hyper3CLIPInference(
-            vision_backbone=model_args["vision_backbone"],
-            text_model_name=model_args["text_model_name"],
-            embed_dim=int(model_args["embed_dim"]),
-            curv_init=float(model_args.get("curv_init", 1.0)),
-            learn_curv=bool(model_args.get("learn_curv", True)),
-        )
-        if weights_path is not None:
-            state_dict = load_file(weights_path, device="cpu")
-            self.model.load_state_dict(state_dict, strict=True)
-        self.model.eval()
-        self.tokenizer = self.model.tokenizer
-
-    @classmethod
-    def load(
-        cls,
-        model_name_or_path: str,
-        subfolder: str = "",
-        token: bool | str | None = None,
-        cache_folder: str | None = None,
-        revision: str | None = None,
-        local_files_only: bool = False,
-        **kwargs: Any,
-    ) -> "Hyper3CLIPSentenceTransformerModule":
-        config = cls.load_config(
-            model_name_or_path,
-            subfolder=subfolder,
-            token=token,
-            cache_folder=cache_folder,
-            revision=revision,
-            local_files_only=local_files_only,
-        )
-        model_config_file = config.get("model_config_file", "config.yaml")
-        weights_file = config.get("weights_file", "model.safetensors")
-        model_config_path = cls.load_file_path(
-            model_name_or_path,
-            model_config_file,
-            subfolder="",
-            token=token,
-            cache_folder=cache_folder,
-            revision=revision,
-            local_files_only=local_files_only,
-        )
-        weights_path = cls.load_file_path(
-            model_name_or_path,
-            weights_file,
-            subfolder="",
-            token=token,
-            cache_folder=cache_folder,
-            revision=revision,
-            local_files_only=local_files_only,
-        )
-        if model_config_path is None:
-            raise FileNotFoundError(f"Could not find {model_config_file!r} in {model_name_or_path!r}")
-        if weights_path is None:
-            raise FileNotFoundError(f"Could not find {weights_file!r} in {model_name_or_path!r}")
-
-        with open(model_config_path, encoding="utf-8") as f:
-            model_config = yaml.safe_load(f)
-
-        return cls(
-            model_config=model_config,
-            weights_path=weights_path,
-            model_config_file=model_config_file,
-            weights_file=weights_file,
-            image_size=int(config.get("image_size", 224)),
-            max_text_length=int(config.get("max_text_length", 77)),
-            normalize_output=bool(config.get("normalize_output", True)),
-        )
-
-    def preprocess(
-        self,
-        inputs: list[Any],
-        prompt: str | None = None,
-        **kwargs: Any,
-    ) -> dict[str, torch.Tensor | str]:
-        if not inputs:
-            return {}
-
-        if all(_is_image(item) for item in inputs):
-            images = torch.stack([self._preprocess_image(item) for item in inputs])
-            return {"pixel_values": images, "modality": "image"}
-
-        if all(isinstance(item, str) for item in inputs):
-            texts = [f"{prompt or ''}{item}" for item in inputs]
-            tokens = self.tokenizer(
-                texts,
-                padding=True,
-                truncation=True,
-                max_length=self.max_text_length,
-                return_tensors="pt",
-            )
-            return {
-                "input_ids": tokens["input_ids"],
-                "attention_mask": tokens["attention_mask"],
-                "modality": "text",
-            }
-
-        raise TypeError("Hyper3-CLIP beta supports batches containing only PIL images or only text strings.")
-
-    def tokenize(self, texts: list[Any], **kwargs: Any) -> dict[str, torch.Tensor | str]:
-        return self.preprocess(texts, **kwargs)
-
-    def forward(self, features: dict[str, torch.Tensor | Any], **kwargs: Any) -> dict[str, torch.Tensor | Any]:
-        if "pixel_values" in features:
-            embeddings = self.model.encode_image(features["pixel_values"], project=False)
-        elif "input_ids" in features and "attention_mask" in features:
-            embeddings = self.model.encode_text(features["input_ids"], features["attention_mask"], project=False)
-        else:
-            raise ValueError("Expected either pixel_values or input_ids/attention_mask in features.")
-
-        embeddings = embeddings.float()
-        if self.normalize_output:
-            embeddings = F.normalize(embeddings, p=2, dim=-1)
-        features["sentence_embedding"] = embeddings
-        return features
-
-    def get_embedding_dimension(self) -> int:
-        return int(self.model.embed_dim)
-
-    def save(self, output_path: str, *args: Any, safe_serialization: bool = True, **kwargs: Any) -> None:
-        output_dir = Path(output_path)
-        output_dir.mkdir(parents=True, exist_ok=True)
-        with open(output_dir / self.config_file_name, "w", encoding="utf-8") as f:
-            json.dump(self.get_config_dict(), f, indent=2)
-
-    def _preprocess_image(self, image: Any) -> torch.Tensor:
-        if isinstance(image, np.ndarray):
-            image = Image.fromarray(image)
-        elif torch.is_tensor(image):
-            array = image.detach().cpu()
-            if array.ndim == 3 and array.shape[0] in {1, 3, 4}:
-                array = array.permute(1, 2, 0)
-            image = Image.fromarray(array.numpy())
-        if not isinstance(image, Image.Image):
-            raise TypeError(f"Expected PIL image, numpy array, or torch tensor, got {type(image)!r}")
-
-        resampling = getattr(Image, "Resampling", Image).BICUBIC
-        image = image.convert("RGB").resize((self.image_size, self.image_size), resampling)
-        array = np.asarray(image, dtype=np.float32) / 255.0
-        tensor = torch.from_numpy(array).permute(2, 0, 1)
-        return (tensor - IMAGENET_MEAN) / IMAGENET_STD
-
-
-class _Hyper3CLIPInference(nn.Module):
-    def __init__(
-        self,
-        vision_backbone: str,
-        text_model_name: str,
-        embed_dim: int,
-        curv_init: float,
-        learn_curv: bool,
-    ) -> None:
-        super().__init__()
-        self.vision_encoder = _VisionEncoder(vision_backbone)
-        self.text_encoder = _TextEncoder(text_model_name)
-        self.tokenizer = self.text_encoder.tokenizer
-        self.embed_dim = int(embed_dim)
-        self.image_proj = nn.Linear(self.vision_encoder.output_dim, self.embed_dim)
-        self.text_proj = nn.Linear(self.text_encoder.output_dim, self.embed_dim)
-        self.logit_scale = nn.Parameter(torch.tensor(1 / 0.07).log())
-        self.visual_alpha = nn.Parameter(torch.full((), self.embed_dim**-0.5).log())
-        self.textual_alpha = nn.Parameter(torch.full((), self.embed_dim**-0.5).log())
-        self.log_curv = nn.Parameter(torch.full((), curv_init).log(), requires_grad=learn_curv)
-
-    def encode_image(self, image: torch.Tensor, project: bool = False) -> torch.Tensor:
-        feats = self.image_proj(self.vision_encoder(image))
-        if project:
-            feats = feats * self.visual_alpha.exp().float()
-        return feats
-
-    def encode_text(self, input_ids: torch.Tensor, attention_mask: torch.Tensor, project: bool = False) -> torch.Tensor:
-        feats = self.text_proj(self.text_encoder(input_ids=input_ids, attention_mask=attention_mask))
-        if project:
-            feats = feats * self.textual_alpha.exp().float()
-        return feats
-
-
-class _VisionEncoder(nn.Module):
-    def __init__(self, backbone_name: str) -> None:
-        super().__init__()
-        self.backbone = timm.create_model(backbone_name, pretrained=False, num_classes=0, global_pool="avg")
-        self.output_dim = self.backbone.num_features
-
-    def forward(self, image: torch.Tensor) -> torch.Tensor:
-        return self.backbone(image)
-
-
-class _TextEncoder(nn.Module):
-    def __init__(self, model_name: str) -> None:
-        super().__init__()
-        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
-        self.backbone = CLIPTextModel(CLIPTextConfig.from_pretrained(model_name))
-        self.output_dim = self.backbone.config.hidden_size
-
-    def forward(self, input_ids: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
-        out = self.backbone(input_ids=input_ids, attention_mask=attention_mask)
-        if hasattr(out, "pooler_output") and out.pooler_output is not None:
-            return out.pooler_output
-        return out.last_hidden_state[:, 0]
-
-
-def _is_image(item: Any) -> bool:
-    return isinstance(item, Image.Image) or isinstance(item, np.ndarray) or torch.is_tensor(item)

metadata.json

@@ -1,7 +1,7 @@
 {
-  "end_time": "2026-05-06T10:53:23+00:00",
+  "end_time": "2026-06-13T06:09:58+00:00",
   "env": {
-    "hostname": "gx08",
+    "hostname": "gx12",
     "rank": "0",
     "world_size": "8"
   },
@@ -9,16 +9,17 @@
   "final_step": 500000,
   "job": {
     "gpus": "0,1,2,3,4,5,6,7",
-    "node_list": "gx08",
+    "job_id": "2061433",
+    "node_list": "gx12",
     "num_nodes": "1",
-    "partition": "aisc-batch",
-    "slurm_job_id": "1916964"
+    "partition": "aisc-batch"
   },
-  "run_id": "hyper3_clip_vit_b_8xh100_full",
-  "start_time": "2026-05-05T09:57:08+00:00",
+  "run_id": "hyper3_vitb_clip_uncha_hier_beta_argent_mp5_paper_scratch_8x500k_s31",
+  "start_time": "2026-06-11T08:58:47+00:00",
   "status": "completed",
   "tags": {
     "data": "processed_grit",
-    "model": "vit_base_patch16_224"
+    "model": "vit_base_patch16_224",
+    "objective": "uncha"
   }
 }

modeling_hyper3_clip.py

@@ -1,119 +0,0 @@
-from __future__ import annotations
-
-from dataclasses import dataclass
-from typing import Any
-
-import numpy as np
-import timm
-import torch
-import torch.nn.functional as F
-from PIL import Image
-from torch import nn
-from transformers import CLIPTextConfig, CLIPTextModel, PreTrainedModel
-from transformers.modeling_outputs import ModelOutput
-
-from .configuration_hyper3_clip import Hyper3CLIPConfig
-
-
-IMAGENET_MEAN = torch.tensor([0.485, 0.456, 0.406], dtype=torch.float32).view(3, 1, 1)
-IMAGENET_STD = torch.tensor([0.229, 0.224, 0.225], dtype=torch.float32).view(3, 1, 1)
-
-
-@dataclass
-class Hyper3CLIPOutput(ModelOutput):
-    image_embeds: torch.FloatTensor | None = None
-    text_embeds: torch.FloatTensor | None = None
-
-
-class Hyper3CLIPModel(PreTrainedModel):
-    config_class = Hyper3CLIPConfig
-    main_input_name = "pixel_values"
-    supports_gradient_checkpointing = False
-
-    def __init__(self, config: Hyper3CLIPConfig) -> None:
-        super().__init__(config)
-        self.vision_encoder = _VisionEncoder(config.vision_backbone)
-        self.text_encoder = _TextEncoder(config.text_model_name)
-        self.embed_dim = int(config.embed_dim)
-        self.image_proj = nn.Linear(self.vision_encoder.output_dim, self.embed_dim)
-        self.text_proj = nn.Linear(self.text_encoder.output_dim, self.embed_dim)
-        self.logit_scale = nn.Parameter(torch.tensor(1 / 0.07).log())
-        self.visual_alpha = nn.Parameter(torch.full((), self.embed_dim**-0.5).log())
-        self.textual_alpha = nn.Parameter(torch.full((), self.embed_dim**-0.5).log())
-        self.log_curv = nn.Parameter(torch.full((), config.curv_init).log(), requires_grad=config.learn_curv)
-
-    def encode_image(self, pixel_values: torch.Tensor, normalize: bool = True) -> torch.Tensor:
-        embeddings = self.image_proj(self.vision_encoder(pixel_values))
-        return F.normalize(embeddings.float(), p=2, dim=-1) if normalize else embeddings
-
-    def encode_text(
-        self,
-        input_ids: torch.Tensor,
-        attention_mask: torch.Tensor,
-        normalize: bool = True,
-    ) -> torch.Tensor:
-        embeddings = self.text_proj(self.text_encoder(input_ids=input_ids, attention_mask=attention_mask))
-        return F.normalize(embeddings.float(), p=2, dim=-1) if normalize else embeddings
-
-    def preprocess_image(self, image: Any) -> torch.Tensor:
-        return _preprocess_image(image, image_size=self.config.image_size)
-
-    def forward(
-        self,
-        pixel_values: torch.Tensor | None = None,
-        input_ids: torch.Tensor | None = None,
-        attention_mask: torch.Tensor | None = None,
-        normalize: bool = True,
-        return_dict: bool | None = None,
-    ) -> Hyper3CLIPOutput | tuple[torch.Tensor | None, torch.Tensor | None]:
-        image_embeds = self.encode_image(pixel_values, normalize=normalize) if pixel_values is not None else None
-        text_embeds = None
-        if input_ids is not None:
-            if attention_mask is None:
-                attention_mask = torch.ones_like(input_ids)
-            text_embeds = self.encode_text(input_ids, attention_mask, normalize=normalize)
-
-        if return_dict is False:
-            return image_embeds, text_embeds
-        return Hyper3CLIPOutput(image_embeds=image_embeds, text_embeds=text_embeds)
-
-
-class _VisionEncoder(nn.Module):
-    def __init__(self, backbone_name: str) -> None:
-        super().__init__()
-        self.backbone = timm.create_model(backbone_name, pretrained=False, num_classes=0, global_pool="avg")
-        self.output_dim = self.backbone.num_features
-
-    def forward(self, image: torch.Tensor) -> torch.Tensor:
-        return self.backbone(image)
-
-
-class _TextEncoder(nn.Module):
-    def __init__(self, model_name: str) -> None:
-        super().__init__()
-        self.backbone = CLIPTextModel(CLIPTextConfig.from_pretrained(model_name))
-        self.output_dim = self.backbone.config.hidden_size
-
-    def forward(self, input_ids: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
-        out = self.backbone(input_ids=input_ids, attention_mask=attention_mask)
-        if hasattr(out, "pooler_output") and out.pooler_output is not None:
-            return out.pooler_output
-        return out.last_hidden_state[:, 0]
-
-
-def _preprocess_image(image: Any, image_size: int = 224) -> torch.Tensor:
-    if isinstance(image, np.ndarray):
-        image = Image.fromarray(image)
-    elif torch.is_tensor(image):
-        array = image.detach().cpu()
-        if array.ndim == 3 and array.shape[0] in {1, 3, 4}:
-            array = array.permute(1, 2, 0)
-        image = Image.fromarray(array.numpy())
-    if not isinstance(image, Image.Image):
-        raise TypeError(f"Expected PIL image, numpy array, or torch tensor, got {type(image)!r}")
-
-    resampling = getattr(Image, "Resampling", Image).BICUBIC
-    image = image.convert("RGB").resize((image_size, image_size), resampling)
-    array = np.asarray(image, dtype=np.float32) / 255.0
-    tensor = torch.from_numpy(array).permute(2, 0, 1)
-    return (tensor - IMAGENET_MEAN) / IMAGENET_STD

modules.json

@@ -1,8 +0,0 @@
-[
-  {
-    "idx": 0,
-    "name": "0_Hyper3CLIP",
-    "path": "0_Hyper3CLIP",
-    "type": "hyper3_clip_sentence_transformers.Hyper3CLIPSentenceTransformerModule"
-  }
-]

requirements.txt

@@ -1,7 +0,0 @@
-sentence-transformers>=5.5.1
-haystack-ai>=2.30.1
-transformers>=4.49.0
-timm>=1.0.0
-safetensors>=0.4.0
-pyyaml>=6.0.0
-Pillow>=10.0.0