README.md

---
license: apache-2.0
base_model:
- google/owlv2-base-patch16
pipeline_tag: object-detection
---
# **NoctOWL: Fine-Grained Open-Vocabulary Object Detector**  


## **Model Description**  

**NoctOWL** (***N***ot **o**nly **c**oarse-**t**ext **OWL**) is an adaptation of **OWL-ViT** (*NoctOWL*) and **OWLv2** (*NoctOWLv2*), designed for **Fine-Grained Open-Vocabulary Detection (FG-OVD)**. Unlike standard open-vocabulary object detectors, which focus primarily on class-level recognition, NoctOWL enhances the ability to detect and distinguish fine-grained object attributes such as color, material, transparency, and pattern. 

It maintains a balanced **trade-off between fine- and coarse-grained detection**, making it particularly effective in scenarios requiring detailed object descriptions.  

You can find the original code to train and evaluate the model [here](https://github.com/lorebianchi98/FG-OVD/tree/main/benchmarks).

### **Model Variants**  
- **NoctOWL Base** (`lorebianchi98/NoctOWL-base-patch16`)  
- **NoctOWLv2 Base** (`lorebianchi98/NoctOWLv2-base-patch16`)  
- **NoctOWL Large** (`lorebianchi98/NoctOWL-large-patch14`)  
- **NoctOWLv2 Large** (`lorebianchi98/NoctOWLv2-large-patch14`)  

## **Usage**  

### **Loading the Model**  
```python
from transformers import OwlViTForObjectDetection, Owlv2ForObjectDetection, OwlViTProcessor, Owlv2Processor

# Load NoctOWL model
model = OwlViTForObjectDetection.from_pretrained("lorebianchi98/NoctOWL-base-patch16")
processor = OwlViTProcessor.from_pretrained("google/owlvit-base-patch16")

# Load NoctOWLv2 model
model_v2 = Owlv2ForObjectDetection.from_pretrained("lorebianchi98/NoctOWLv2-base-patch16")
processor_v2 = Owlv2Processor.from_pretrained("google/owlv2-base-patch16")
```  

### **Inference Example**  
```python
from PIL import Image
import torch

# Load image
image = Image.open("example.jpg")

# Define text prompts (fine-grained descriptions)
text_queries = ["a red patterned dress", "a dark brown wooden chair"]

# Process inputs
inputs = processor(images=image, text=text_queries, return_tensors="pt")

# Run inference
outputs = model(**inputs)

# Extract detected objects
logits = outputs.logits
boxes = outputs.pred_boxes

# Post-processing can be applied to visualize results
```

## Results
We report the mean Average Precision (**mAP**) on the Fine-Grained Open-Vocabulary Detection ([FG-OVD](https://lorebianchi98.github.io/FG-OVD/)) benchmarks across different difficulty levels, as well as performance on rare classes from the LVIS dataset.
| Model | LVIS (Rare) | Trivial | Easy | Medium | Hard | Color | Material | Pattern | Transparency |  
|-------|------------|----------------|---------------|---------------|---------------|-------|----------|---------|--------------|  
| OWL (B/16) | 20.6 | 53.9 | 38.4 | 39.8 | 26.2 | 45.3 | 37.3 | 26.6 | 34.1 |  
| OWL (L/14) | 31.2 | 65.1 | 44.0 | 39.3 | 26.5 | 43.8 | 44.9 | 36.0 | 29.2 |  
| OWLv2 (B/16) | 29.6 | 52.9 | 40.0 | 38.5 | 25.3 | 45.1 | 33.5 | 19.2 | 28.5 |  
| OWLv2 (L/14) | **34.9** | 63.2 | 42.8 | 41.2 | 25.4 | 53.3 | 36.9 | 23.3 | 12.2 |  
| **NoctOWL (B/16)** | 11.6 | 46.6 | 44.4 | 45.6 | 40.0 | 44.7 | 46.0 | 46.1 | 53.6 |  
| **NoctOWL (L/14)** | 26.0 | 57.4 | 54.2 | 54.8 | 48.6 | 53.1 | 56.9 | **49.8** | **57.2** |  
| **NoctOWLv2 (B/16)** | 17.5 | 48.3 | 49.1 | 47.1 | 42.1 | 46.8 | 48.2 | 42.2 | 50.2 |  
| **NoctOWLv2 (L/14)** | 27.2 | **57.5** | **55.5** | **57.2** | **50.2** | **55.6** | **57.0** | 49.2 | 55.9 |