Upload folder using huggingface_hub

README.md

@@ -0,0 +1,87 @@
+---
+language:
+- en
+library_name: transformers
+tags:
+- computer-vision
+- image-segmentation
+- semantic-segmentation
+- co-segmentation
+- part-segmentation
+- multi-image-reasoning
+- vision-language
+pipeline_tag: image-segmentation
+datasets:
+- PLAN-Lab/MixedParts
+---
+
+# CALICO
+
+CALICO is a large vision-language model for part-focused semantic co-segmentation. Given a pair of images and a natural-language prompt, CALICO identifies common objects, common parts, or unique parts and predicts segmentation masks for the referenced regions.
+
+This repository contains the released CALICO checkpoint. It is intended to be loaded with the [CALICO codebase](https://github.com/PLAN-Lab/CALICO).
+
+## Model Details
+
+- **Model name:** CALICO
+- **Dataset:** Mixed Parts
+- **Task:** part-focused semantic co-segmentation and segmentation-grounded vision-language reasoning
+- **Checkpoint format:** merged Hugging Face checkpoint with safetensors shards
+- **Vision-language image encoder:** Q-Former over EVA-CLIP-G visual features
+- **Grounding decoder:** SAM ViT-H mask decoder
+- **Correspondence features:** frozen DINOv2 features used by CALICO correspondence modules
+
+CALICO includes two correspondence modules in the language-model forward pass:
+
+- **Correspondence Extraction Module (CEM):** fuses Q-Former visual patch embeddings with frozen DINOv2 correspondence features.
+- **Correspondence Adaptation Module (CAM):** adapts Q-Former queries from the current language state and reintegrates correspondence-aware visual features into image-token hidden states.
+
+## Intended Use
+
+Use CALICO for research on multi-image part-focused segmentation, semantic co-segmentation, and vision-language grounding. The model is released for evaluation and fine-tuning with the CALICO repository.
+
+## Quick Start
+
+Install the CALICO environment and prepare Mixed Parts following the repository docs:
+
+- [Installation](https://github.com/PLAN-Lab/CALICO/blob/main/docs/INSTALL.md)
+- [Data preparation](https://github.com/PLAN-Lab/CALICO/blob/main/docs/DATA.md)
+- [Mixed Parts dataset](https://huggingface.co/datasets/PLAN-Lab/MixedParts)
+
+Run evaluation from the CALICO repository root:
+
+```bash
+python evaluate.py \
+  --merged_ckpt_path PLAN-Lab/CALICO \
+  --dataset_dir ./data \
+  --output_save_path ./evaluate_results/calico_mixed_parts \
+  --val_dataset "MixedPartsObjectVal|MixedPartsPartVal" \
+  --multi_image_filepath_prefix ./data/mixed_parts_data/mixed_parts_test.json \
+  --mode test \
+  --compute_metrics
+```
+
+`--dataset_dir ./data` should contain:
+
+```text
+data/
+├── coco_2017/
+└── mixed_parts_data/
+```
+
+## Training and Fine-Tuning
+
+Fine-tuning is supported through `train.py` in the CALICO repository. See [docs/TRAINING.md](https://github.com/PLAN-Lab/CALICO/blob/main/docs/TRAINING.md) for recommended arguments and data layout.
+
+## Citation
+
+If you use CALICO or Mixed Parts, please cite:
+
+```bibtex
+@article{nguyen2025calico,
+  title={CALICO: Part-Focused Semantic Co-Segmentation with Large Vision-Language Models},
+  author={Nguyen, Kiet A. and Juvekar, Adheesh and Yu, Tianjiao and Wahed, Muntasir and Lourentzou, Ismini},
+  journal={In Proceedings for the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
+  year={2025}
+}
+```

added_tokens.json

@@ -0,0 +1,9 @@
+{
+  "</p>": 32006,
+  "<bbox>": 32002,
+  "<im_end>": 32001,
+  "<im_start>": 32000,
+  "<p>": 32005,
+  "<point>": 32003,
+  "[SEG]": 32004
+}

config.json

@@ -0,0 +1,72 @@
+{
+  "_name_or_path": "PLAN-Lab/CALICO",
+  "architectures": [
+    "CALICOForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_initialize_adaptors": true,
+  "auto_initialize_qformer": true,
+  "bbox_token_idx": 32002,
+  "bos_token_id": 1,
+  "dino_hidden_size": 768,
+  "dino_model_name": "dinov2_vitb14_reg",
+  "eos_token_id": 2,
+  "global_image_size": 224,
+  "grounding_encoder": "sam_vit_h",
+  "grounding_image_size": 1024,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "image_aspect": "square",
+  "image_aspect_ratio": "square",
+  "initializer_range": 0.02,
+  "intermediate_size": 11008,
+  "layer_type": "linear",
+  "max_position_embeddings": 4096,
+  "mlp_bias": false,
+  "mm_projector_type": "linear",
+  "model_type": "calico",
+  "num_attention_heads": 32,
+  "num_attn_heads": 8,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 32,
+  "num_level_reg_features": 4,
+  "num_query_tokens": 32,
+  "out_dim": 256,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "q_former_model": "",
+  "qformer_hidden_size": 768,
+  "qformer_vision_encoder": "eva_clip_g",
+  "qformer_vision_width": 1408,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "seg_image_tokens": [
+    [
+      32004,
+      313,
+      2382,
+      29896
+    ],
+    [
+      32004,
+      313,
+      2382,
+      29906
+    ]
+  ],
+  "seg_token_idx": 32004,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "train_mask_decoder": true,
+  "transformers_version": "4.42.3",
+  "update_layers": [
+    11,
+    22
+  ],
+  "use_cache": false,
+  "use_mm_proj": true,
+  "vocab_size": 32007,
+  "with_region": true
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "max_length": 4096,
+  "pad_token_id": 0,
+  "transformers_version": "4.42.3",
+  "use_cache": false
+}

model-00001-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a417a84fac860be7e6c606a3b1b66462dc481199df410655162415343aee6585
+size 4939091944

model-00002-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:272d9186b3b45696e68c1ffa0dce3ab7aba618de3fa61126fd04e5e9cdea462c
+size 4947390880

model-00003-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f71b674f01ebb20a2f9d44c3ec83803a6e95a285c97d3e70d498f710829008a
+size 4991144144

model-00004-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:399403a0245d505e31b04076d65155999909907766e558d856e3d26e2a456b8b
+size 2147217712

special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<unk>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
+size 499723

tokenizer_config.json

@@ -0,0 +1,99 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": true,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32000": {
+      "content": "<im_start>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32001": {
+      "content": "<im_end>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32002": {
+      "content": "<bbox>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32003": {
+      "content": "<point>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32004": {
+      "content": "[SEG]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32005": {
+      "content": "<p>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32006": {
+      "content": "</p>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": false,
+  "model_max_length": 2300,
+  "pad_token": "<unk>",
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}