Hugging Face's logo

jeeit17
/
HyFL-CLIP

Feature Extraction
OpenCLIP
English
clip
open-clip
vision-language
image-text-retrieval
cross-modal-retrieval
long-context
hyperbolic-learning
hyperbolic-geometry
lorentz-model
eccv-2026
Model card Files
xet
 Community
HyFL-CLIP / README.md
jeeit17's picture
jeeit17
Update README.md
ad7a2f5 verified
preview code
|
Raw
History Blame Contribute Delete
9.85 kB
metadata 
license: mit
library_name: open_clip
pipeline_tag: feature-extraction
base_model_relation: finetune
language:
  - en
tags:
  - clip
  - open-clip
  - vision-language
  - image-text-retrieval
  - cross-modal-retrieval
  - long-context
  - hyperbolic-learning
  - hyperbolic-geometry
  - lorentz-model
  - eccv-2026

HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding

ECCV 2026

• Project Page
• Paper
• Code

Overview

HyFL-CLIP improves the robustness of long-context vision-language understanding by transferring OpenCLIP’s pretrained Euclidean alignment into a hyperbolic representation space. The model reports improvements of up to 19.5% in long-text cross-modal retrieval under perturbations including:

  • caption reordering;
  • caption summarization;
  • caption condensation; and
  • partial text deletion.

Model Description

HyFL-CLIP is a hyperbolic fine-tuning framework that improves the robustness of CLIP to long and compositional text descriptions.

Standard CLIP models are primarily trained using short captions and rely on Euclidean contrastive learning. Their image–text alignment can therefore degrade when long descriptions are reordered, summarized, condensed, or partially deleted.

HyFL-CLIP addresses this limitation by projecting OpenCLIP representations into hyperbolic space, specifically the Lorentz model. Hyperbolic geometry allows the model to represent hierarchical semantic relationships among token-level information, partial descriptions, short captions, long-form descriptions, and images.

This checkpoint is fine-tuned from an OpenCLIP pretrained model.

Method

HyFL-CLIP incorporates three main components.

Cross-Manifold Similarity Distillation

Similarity knowledge from a pretrained Euclidean OpenCLIP model is distilled into a hyperbolic student model. This preserves the alignment knowledge of the pretrained model while changing the underlying representation geometry.

Hierarchical Semantic Modeling

Summarized tokens, short captions, long-form descriptions, and images are embedded in a shared hyperbolic space. This enables the model to represent part–whole and summary–detail relationships.

Einstein Midpoint Aggregation

Token- and phrase-level representations are aggregated using the Lorentzian centroid, also known as the Einstein midpoint. This provides a geometry-consistent method for combining partial representations into whole-level representations.

Results

The evaluation covers:

  • long-context cross-modal retrieval;
  • retrieval under caption perturbations;
  • intra-modality retrieval;
  • short-text cross-modal retrieval; and
  • integration with downstream generators such as Stable Diffusion XL.

Please refer to the paper for the complete quantitative results and ablation studies.

Evaluation Datasets

HyFL-CLIP is evaluated on the following datasets:

Dataset Evaluation setting
Urban1K Long-context image–text retrieval
DOCCI Dense and compositional caption retrieval
Long-DCI Long-description image–text retrieval
ShareGPT4V Detailed vision-language descriptions
COCO Short-text image–text retrieval
Flickr30K Short-text image–text retrieval

The datasets are not distributed as part of this model repository. Users must obtain them from their respective sources and comply with their individual licenses and terms of use.

Installation

Clone the official HyFL-CLIP repository and prepare the required environment.

git clone https://github.com/janeyeon/hyfl-clip.git
cd hyfl-clip

Install the required dependencies:

pip install torch torchvision
pip install open_clip_torch
pip install huggingface_hub

Please refer to the official code repository for the complete environment configuration.

Download

Two HyFL-CLIP checkpoints are available:

  • HyFL-CLIP_ViTB.pt: HyFL-CLIP with a ViT-B backbone
  • HyFL-CLIP_ViTL.pt: HyFL-CLIP with a ViT-L backbone

The checkpoints can be downloaded using huggingface_hub:

from huggingface_hub import hf_hub_download

vit_b_checkpoint = hf_hub_download(
    repo_id="jeeit17/HyFL-CLIP",
    filename="HyFL-CLIP_ViTB.pt",
)

vit_l_checkpoint = hf_hub_download(
    repo_id="jeeit17/HyFL-CLIP",
    filename="HyFL-CLIP_ViTL.pt",
)

print("ViT-B checkpoint:", vit_b_checkpoint)
print("ViT-L checkpoint:", vit_l_checkpoint)

Usage

HyFL-CLIP checkpoints should be loaded using the model architecture and Lorentz-geometry implementation provided in the official HyFL-CLIP codebase.

A general checkpoint-loading pattern is shown below:

import torch

checkpoint = torch.load(
    "hyfl_clip.pt",
    map_location="cpu",
)

model = build_hyfl_clip_model()

if "state_dict" in checkpoint:
    model.load_state_dict(checkpoint["state_dict"])
else:
    model.load_state_dict(checkpoint)

model.eval()

The model must be initialized using the same configuration used during training, including:

  • OpenCLIP backbone architecture;
  • pretrained OpenCLIP checkpoint;
  • text context length;
  • embedding dimension;
  • hyperbolic curvature;
  • checkpoint state-dictionary format; and
  • image and text preprocessing configuration.

Please refer to the official HyFL-CLIP repository for the exact model initialization and inference code.

Evaluation

Cross-modal retrieval evaluation is implemented in eval/retrieval/.

Run the provided evaluation script:

bash eval.sh

The underlying command follows this format:

python eval/retrieval/unified_eval.py \
    --ckpt_path /path/to/hyfl_clip.pt \
    --datasets urban1k docci dci sharegpt4v

Supported long-context evaluation datasets include:

urban1k
docci
dci
sharegpt4v

Before running the evaluation, update the dataset image_root and json_path fields in eval/retrieval/unified_eval.py.

Dataset preparation instructions are available from:

  1. Urban1K
  2. TULIP for Long-DCI
  3. DOCCI

Training

Training is performed using train/train.py with distributed PyTorch execution.

Run the provided launcher:

bash train.sh

The launcher internally uses torchrun. Update CUDA_VISIBLE_DEVICES and --nproc_per_node according to the target machine.

Checkpoints are saved in the following format:

output/ckpts/<experiment_name>/ep=<epoch>_hyfl.pt

Intended Uses

HyFL-CLIP is intended for research involving:

  • long-context image–text retrieval;
  • robust cross-modal retrieval;
  • compositional vision-language understanding;
  • hierarchical multimodal representation learning;
  • hyperbolic representation learning;
  • retrieval under textual perturbations;
  • long-caption image search; and
  • multimodal representation analysis.

Out-of-Scope Uses

HyFL-CLIP is not designed or validated for:

  • safety-critical decision-making;
  • medical diagnosis;
  • legal or financial decision-making;
  • biometric identification;
  • surveillance;
  • autonomous content moderation; or
  • applications requiring guaranteed robustness or fairness.

Retrieval performance should not be interpreted as general-purpose long-context reasoning ability.

Authors

1 Department of Electrical and Computer Engineering, Seoul National University 2 Interdisciplinary Program in Artificial Intelligence, Seoul National University 3 INMC and AIIS, Seoul National University

* Equal contribution

Acknowledgements

HyFL-CLIP builds on:

We thank the authors and maintainers of these projects for making their research, models, and implementations publicly available.

License

HyFL-CLIP is released under the MIT License. See the LICENSE file for the full license text.

This model is a fine-tuned version of an OpenCLIP pretrained model. Users are responsible for complying with the license terms applicable to:

  • the specific pretrained OpenCLIP checkpoint;
  • incorporated third-party source code;
  • training and evaluation datasets; and
  • other third-party components used with the model.

The relevant upstream copyright and license notices must be preserved in accordance with their respective license terms.

The MIT License applied to HyFL-CLIP does not replace or override separate licenses, attribution requirements, or usage restrictions applicable to third-party checkpoints, code, or datasets.

Citation

Please cite the following paper when using HyFL-CLIP:

@misc{jang2026hyflclip,
  title         = {HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding},
  author        = {Ji Ha Jang and Hayeon Kim and Chulwon Lee and Junghun James Kim and Se Young Chun},
  year          = {2026},
  eprint        = {2607.00428},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2607.00428}
}