rsomani95 commited on
Commit
9121dcc
·
verified ·
1 Parent(s): b37fdf1

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +32 -46
README.md CHANGED
@@ -3,7 +3,7 @@ library_name: cinemaclip
3
  pipeline_tag: zero-shot-image-classification
4
  tags:
5
  - clip
6
- - mobile-clip
7
  - cinema
8
  - film
9
  - movies
@@ -13,21 +13,20 @@ tags:
13
  - domain-specific
14
  - image-classification
15
  - zero-shot
16
- base_model: apple/MobileCLIP-S1-OpenCLIP
17
  base_model_relation: finetune
18
- license: apple-amlr
19
- license_name: aplle-ascl
20
- license_link: https://github.com/apple/ml-mobileclip/blob/main/LICENSE_MODELS
21
  ---
22
 
23
  # CinemaCLIP-1.0.0
24
 
25
- **CinemaCLIP** is a [MobileCLIP-S1](https://huggingface.co/apple/MobileCLIP-S1-OpenCLIP) fine-tune specialized for understanding the visual language of cinema at a frame level. It is a hybrid CLIP model with 23 classifier heads that represent a comprehensive taxonomy built with domain experts. For more info, see our [launch blog post](https://www.ozu.ai/cinemaclip).
26
 
27
  This repository ships three serialized forms of the same model:
28
- - **Torch** (`model.safetensors`): load via the `cinemaclip` Python package.
29
- - **CoreML** (`ImageEncoder.mlmodel`, `ImageEncoder.mlpackage` and `TextEncoder.mlpackage`): on-device Apple Neural Engine inference.
30
- - **ONNX** (`ImageEncoder.onnx`, `TextEncoder.onnx`, plus `_fp16` variants): cross-platform inference.
31
 
32
  ## Install
33
 
@@ -123,45 +122,32 @@ Two inference modes are reported for CinemaCLIP:
123
 
124
  | Category | CinemaCLIP 0-shot | CinemaCLIP Classifier | Qwen3.5-4B | Gemma4-4B | InternVL3.5-4B | Molmo2-4B | DFN ViT-H-14 | MetaCLIP PE-bigG | OpenAI ViT-L-14 | MobileCLIP-S1 | DFN ViT-L-14 | SigLIP2 SO400M | SigLIP2 ViT-gopt |
125
  |---|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|
126
- | **Mean** | **82.9** | **87.6** | **57.6** | **56.7** | **55.3** | **55.3** | **45.9** | **45.2** | **44.8** | **44.2** | **39.0** | **38.7** | **36.5** |
127
- | Color Contrast | 89.6 | 86.8 | 33.7 | 35.3 | 33.7 | 35.3 | 34.0 | 33.1 | 49.4 | 38.7 | 37.1 | 57.7 | 25.2 |
128
- | Color Key | 84.9 | 92.9 | 78.1 | 78.1 | 80.3 | 64.3 | 58.2 | 50.2 | 53.2 | 59.4 | 48.3 | 22.8 | 52.6 |
129
- | Color Saturation | 82.6 | 82.6 | 66.5 | 65.4 | 72.1 | 45.9 | 55.1 | 61.8 | 58.1 | 35.8 | 46.8 | 33.3 | 31.8 |
130
- | Color Theory | 71.3 | 72.7 | 54.0 | 51.7 | 50.7 | 48.7 | 54.7 | 51.7 | 50.7 | 47.3 | 47.7 | 31.3 | 31.7 |
131
- | Color Tones | 86.0 | 86.5 | 50.2 | 62.6 | 70.6 | 62.1 | 58.5 | 50.2 | 52.0 | 55.7 | 47.2 | 24.0 | 17.7 |
132
- | Lighting Cast | 85.9 | 90.4 | 38.3 | 53.3 | 39.8 | 35.7 | 25.4 | 29.3 | 28.8 | 35.7 | 22.8 | 37.8 | 18.2 |
133
- | Lighting Contrast | 93.9 | 95.3 | 29.8 | 39.1 | 38.7 | 46.1 | 35.3 | 35.5 | 32.6 | 39.0 | 39.4 | 48.4 | 37.6 |
134
- | Lighting Edge | 87.6 | 90.4 | 22.8 | 38.8 | 31.2 | 40.4 | 22.4 | 31.6 | 41.6 | 34.0 | 21.2 | 26.0 | 25.6 |
135
- | Lighting Silhouette | 88.4 | 93.1 | 80.9 | 63.0 | 48.9 | 48.8 | 66.6 | 67.1 | 67.4 | 58.4 | 43.5 | 46.2 | 78.9 |
136
- | Shot Angle | 73.4 | 82.3 | 41.9 | 49.2 | 33.2 | 49.9 | 28.0 | 13.7 | 19.0 | 19.6 | 25.9 | 21.3 | 17.2 |
137
- | Shot Composition | 95.5 | 96.0 | 46.0 | 54.5 | 55.7 | 60.5 | 27.8 | 24.3 | 21.3 | 22.0 | 25.2 | 31.4 | 11.4 |
138
- | Shot Dutch Angle | 61.9 | 78.5 | 62.2 | 65.1 | 46.7 | 49.3 | 27.3 | 44.5 | 38.4 | 56.6 | 25.9 | 47.6 | 68.7 |
139
- | Shot Focus | 71.3 | 71.2 | 19.9 | 26.6 | 26.3 | 25.1 | 32.9 | 31.2 | 24.4 | 31.3 | 37.3 | 48.2 | 12.6 |
140
- | Shot Framing | 79.2 | 83.8 | 38.0 | 29.6 | 40.1 | 34.6 | 33.6 | 24.9 | 23.5 | 23.9 | 33.0 | 7.3 | 9.8 |
141
- | Shot Height | 90.5 | 91.8 | 38.1 | 37.4 | 41.2 | 53.0 | 37.6 | 33.7 | 28.9 | 24.0 | 33.6 | 29.6 | 23.9 |
142
- | Shot Lens Size | 67.9 | 70.6 | 49.6 | 28.0 | 43.6 | 46.6 | 32.1 | 28.0 | 34.5 | 30.1 | 25.7 | 30.1 | 17.6 |
143
- | Shot Location | 90.9 | 93.9 | 81.0 | 82.2 | 81.5 | 79.2 | 73.0 | 68.4 | 68.0 | 75.6 | 66.1 | 65.0 | 46.7 |
144
- | Shot Symmetry | 88.3 | 92.9 | 90.2 | 86.7 | 76.0 | 80.2 | 76.6 | 78.0 | 54.0 | 39.3 | 24.9 | 46.0 | 82.4 |
145
- | Shot Time of Day | 69.2 | 89.0 | 75.1 | 66.1 | 70.7 | 70.7 | 68.1 | 69.6 | 60.3 | 73.7 | 71.2 | 48.5 | 42.7 |
146
- | Shot Type | 81.8 | 90.5 | 81.3 | 61.2 | 57.0 | 57.4 | 52.8 | 40.4 | 36.5 | 35.7 | 56.7 | 46.5 | 29.7 |
147
- | Shot Type - Crowd | 91.5 | 99.6 | 97.2 | 88.2 | 94.3 | 94.8 | 55.9 | 69.1 | 68.6 | 77.2 | 37.3 | 52.4 | 69.3 |
148
- | Shot Type - OTS | 92.0 | 95.5 | 92.5 | 85.0 | 83.9 | 87.6 | 53.2 | 57.0 | 73.9 | 60.3 | 42.1 | 50.5 | 51.2 |
149
 
150
  The `shot.lighting.direction` head ships in the classifier heads but has been excluded from the table above being a multi-label classifier.
151
 
152
- ## Files in this repo
153
-
154
- | File | Purpose |
155
- |---|---|
156
- | `model.safetensors` | Blended (α=0.75) torch weights — `CinemaCLIP.from_pretrained` target |
157
- | `config.json` | Autogenerated `__init__` kwargs for `CinemaCLIP` |
158
- | `CinemaNetSchema.json` | Schema detailing classifier head metadata, confidence thresholds, preprocessing info |
159
- | `ImageEncoder.mlmodel` | CoreML `"neuralnetwork"` ImageEncoder (unified embedding + probabilities) |
160
- | `ImageEncoder.mlpackage` | CoreML ImageEncoder (unified embedding + probabilities) |
161
- | `TextEncoder.mlpackage` | CoreML TextEncoder |
162
- | `ImageEncoder.onnx` / `_fp16.onnx` | ONNX ImageEncoder |
163
- | `TextEncoder.onnx` / `_fp16.onnx` | ONNX TextEncoder |
164
-
165
 
166
  ## Citation
167
 
@@ -170,7 +156,7 @@ The `shot.lighting.direction` head ships in the classifier heads but has been ex
170
  title = {CinemaCLIP: A hybrid CLIP model and taxonomy for the visual language of cinema},
171
  author = {Somani, Rahul and Marini, Anton and Stewart, Damian},
172
  year = {2026},
173
- publisher = {HuggingFace},
174
  doi = {10.57967/hf/8539},
175
  howpublished = {\url{https://huggingface.co/OZU-Technology/CinemaCLIP}},
176
  note = {Model weights and taxonomy}
 
3
  pipeline_tag: zero-shot-image-classification
4
  tags:
5
  - clip
6
+ - vit
7
  - cinema
8
  - film
9
  - movies
 
13
  - domain-specific
14
  - image-classification
15
  - zero-shot
16
+ base_model: laion/CLIP-ViT-B-32-256x256-DataComp-s34B-b86K
17
  base_model_relation: finetune
18
+ license: openrail
19
+ license_link: LICENSE.md
 
20
  ---
21
 
22
  # CinemaCLIP-1.0.0
23
 
24
+ **CinemaCLIP** is a [ViT-B-32-256](https://huggingface.co/laion/CLIP-ViT-B-32-256x256-DataComp-s34B-b86K) fine-tune specialized for understanding the visual language of cinema at a frame level. It is a hybrid CLIP model with 23 classifier heads that represent a comprehensive taxonomy built with domain experts. For more info, see our [launch blog post](https://www.ozu.ai/cinemaclip).
25
 
26
  This repository ships three serialized forms of the same model:
27
+ - **Torch** (`model.safetensors`) load via the `cinemaclip` Python package.
28
+ - **CoreML** (`ImageEncoder.mlmodel`, `ImageEncoder.mlpackage` and `TextEncoder.mlpackage`) — for on-device Apple Neural Engine inference.
29
+ - **ONNX** (`ImageEncoder.onnx`, `TextEncoder.onnx`, plus `_fp16` variants) — for cross-platform inference.
30
 
31
  ## Install
32
 
 
122
 
123
  | Category | CinemaCLIP 0-shot | CinemaCLIP Classifier | Qwen3.5-4B | Gemma4-4B | InternVL3.5-4B | Molmo2-4B | DFN ViT-H-14 | MetaCLIP PE-bigG | OpenAI ViT-L-14 | MobileCLIP-S1 | DFN ViT-L-14 | SigLIP2 SO400M | SigLIP2 ViT-gopt |
124
  |---|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|
125
+ | **Mean** | **83.2** | **87.7** | **57.6** | **56.7** | **55.3** | **55.3** | **45.9** | **45.2** | **44.8** | **44.2** | **39.0** | **38.7** | **36.5** |
126
+ | Color Contrast | 89.3 | 87.4 | 33.7 | 35.3 | 33.7 | 35.3 | 34.0 | 33.1 | 49.4 | 38.7 | 37.1 | 57.7 | 25.2 |
127
+ | Color Key | 86.8 | 95.7 | 78.1 | 78.1 | 80.3 | 64.3 | 58.2 | 50.2 | 53.2 | 59.4 | 48.3 | 22.8 | 52.6 |
128
+ | Color Saturation | 83.0 | 84.3 | 66.5 | 65.4 | 72.1 | 45.9 | 55.1 | 61.8 | 58.1 | 35.8 | 46.8 | 33.3 | 31.8 |
129
+ | Color Theory | 75.3 | 73.3 | 54.0 | 51.7 | 50.7 | 48.7 | 54.7 | 51.7 | 50.7 | 47.3 | 47.7 | 31.3 | 31.7 |
130
+ | Color Tones | 87.3 | 89.3 | 50.2 | 62.6 | 70.6 | 62.1 | 58.5 | 50.2 | 52.0 | 55.7 | 47.2 | 24.0 | 17.7 |
131
+ | Lighting Cast | 81.2 | 87.8 | 38.3 | 53.3 | 39.8 | 35.7 | 25.4 | 29.3 | 28.8 | 35.7 | 22.8 | 37.8 | 18.2 |
132
+ | Lighting Contrast | 91.6 | 93.2 | 29.8 | 39.1 | 38.7 | 46.1 | 35.3 | 35.5 | 32.6 | 39.0 | 39.4 | 48.4 | 37.6 |
133
+ | Lighting Edge | 80.4 | 93.6 | 22.8 | 38.8 | 31.2 | 40.4 | 22.4 | 31.6 | 41.6 | 34.0 | 21.2 | 26.0 | 25.6 |
134
+ | Lighting Silhouette | 88.2 | 92.0 | 80.9 | 63.0 | 48.9 | 48.8 | 66.6 | 67.1 | 67.4 | 58.4 | 43.5 | 46.2 | 78.9 |
135
+ | Shot Angle | 79.5 | 84.4 | 41.9 | 49.2 | 33.2 | 49.9 | 28.0 | 13.7 | 19.0 | 19.6 | 25.9 | 21.3 | 17.2 |
136
+ | Shot Composition | 94.0 | 97.0 | 46.0 | 54.5 | 55.7 | 60.5 | 27.8 | 24.3 | 21.3 | 22.0 | 25.2 | 31.4 | 11.4 |
137
+ | Shot Dutch Angle | 67.6 | 73.6 | 62.2 | 65.1 | 46.7 | 49.3 | 27.3 | 44.5 | 38.4 | 56.6 | 25.9 | 47.6 | 68.7 |
138
+ | Shot Focus | 59.1 | 71.8 | 19.9 | 26.6 | 26.3 | 25.1 | 32.9 | 31.2 | 24.4 | 31.3 | 37.3 | 48.2 | 12.6 |
139
+ | Shot Framing | 83.1 | 82.3 | 38.0 | 29.6 | 40.1 | 34.6 | 33.6 | 24.9 | 23.5 | 23.9 | 33.0 | 7.3 | 9.8 |
140
+ | Shot Height | 89.2 | 92.8 | 38.1 | 37.4 | 41.2 | 53.0 | 37.6 | 33.7 | 28.9 | 24.0 | 33.6 | 29.6 | 23.9 |
141
+ | Shot Lens Size | 73.3 | 76.7 | 49.6 | 28.0 | 43.6 | 46.6 | 32.1 | 28.0 | 34.5 | 30.1 | 25.7 | 30.1 | 17.6 |
142
+ | Shot Location | 86.5 | 92.9 | 81.0 | 82.2 | 81.5 | 79.2 | 73.0 | 68.4 | 68.0 | 75.6 | 66.1 | 65.0 | 46.7 |
143
+ | Shot Symmetry | 87.8 | 91.0 | 90.2 | 86.7 | 76.0 | 80.2 | 76.6 | 78.0 | 54.0 | 39.3 | 24.9 | 46.0 | 82.4 |
144
+ | Shot Time of Day | 75.7 | 87.6 | 75.1 | 66.1 | 70.7 | 70.7 | 68.1 | 69.6 | 60.3 | 73.7 | 71.2 | 48.5 | 42.7 |
145
+ | Shot Type | 80.7 | 86.7 | 81.3 | 61.2 | 57.0 | 57.4 | 52.8 | 40.4 | 36.5 | 35.7 | 56.7 | 46.5 | 29.7 |
146
+ | Shot Type - Crowd | 96.9 | 99.1 | 97.2 | 88.2 | 94.3 | 94.8 | 55.9 | 69.1 | 68.6 | 77.2 | 37.3 | 52.4 | 69.3 |
147
+ | Shot Type - OTS | 94.1 | 96.4 | 92.5 | 85.0 | 83.9 | 87.6 | 53.2 | 57.0 | 73.9 | 60.3 | 42.1 | 50.5 | 51.2 |
148
 
149
  The `shot.lighting.direction` head ships in the classifier heads but has been excluded from the table above being a multi-label classifier.
150
 
 
 
 
 
 
 
 
 
 
 
 
 
 
151
 
152
  ## Citation
153
 
 
156
  title = {CinemaCLIP: A hybrid CLIP model and taxonomy for the visual language of cinema},
157
  author = {Somani, Rahul and Marini, Anton and Stewart, Damian},
158
  year = {2026},
159
+ publisher = {Hugging Face},
160
  doi = {10.57967/hf/8539},
161
  howpublished = {\url{https://huggingface.co/OZU-Technology/CinemaCLIP}},
162
  note = {Model weights and taxonomy}