README.md

---
title: SAM3 Promptable Concept Segmentation
emoji: 🎯
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: false
license: apache-2.0
short_description: SAM3 inference with text prompts and SAM2 API compatibility
---

# SAM3 Promptable Concept Segmentation

This Space provides both a **web interface** and **REST API** for SAM3 (Segment Anything Model 3) inference, featuring:

## 🚀 Key Features

- **🆕 Text Prompts**: Segment objects using natural language descriptions (e.g., "kitten", "car", "person wearing red shirt")
- **🔄 SAM2 Compatible**: Drop-in replacement for existing SAM2 inference endpoints
- **📊 High Quality**: Uses official SAM3 post-processing for single high-confidence masks
- **🔌 Dual APIs**: Simple Gradio API + SAM2-compatible inference endpoint format
- **⚡ Fast**: Optimized for production use with proper confidence thresholding

## 📖 Usage

### Web Interface
Simply upload an image, enter a text description of what you want to segment, and adjust the confidence threshold.

### API Usage

#### 1. Simple Text API (Gradio format)
```python
import requests
import base64

# Encode your image to base64
with open("image.jpg", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

# Make API request
response = requests.post(
    "https://your-username-sam3-api.hf.space/api/predict",
    json={
        "data": [image_b64, "kitten", 0.5]
    }
)

result = response.json()
```

#### 2. SAM2/SAM3 Compatible API (Inference Endpoint format)
```python
import requests
import base64

# Encode your image to base64
with open("image.jpg", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

# SAM3 Text Prompts (NEW)
response = requests.post(
    "https://your-username-sam3-api.hf.space/api/sam2_compatible",
    json={
        "data": [{
            "inputs": {
                "image": image_b64,
                "text_prompts": ["kitten", "toy"],
                "confidence_threshold": 0.5
            }
        }]
    }
)

# SAM2 Compatible (Points/Boxes)
response = requests.post(
    "https://your-username-sam3-api.hf.space/api/sam2_compatible",
    json={
        "data": [{
            "inputs": {
                "image": image_b64,
                "boxes": [[100, 100, 200, 200]],
                "confidence_threshold": 0.5
            }
        }]
    }
)

result = response.json()
```

## 🔧 API Parameters

### SAM2-Compatible API Input
```json
{
  "inputs": {
    "image": "base64_encoded_image_string",

    // SAM3 NEW: Text-based prompts
    "text_prompts": ["person", "car"],  // List of text descriptions

    // SAM2 COMPATIBLE: Point-based prompts
    "points": [[[x1, y1]], [[x2, y2]]],  // Points for each object
    "point_labels": [[1], [1]],  // Labels for each point (1=foreground, 0=background)

    // SAM2 COMPATIBLE: Bounding box prompts
    "boxes": [[x1, y1, x2, y2], [x1, y1, x2, y2]],  // Bounding boxes
    "box_labels": [1, 0],  // Labels for each box (1=positive, 0=negative/exclude)

    "multimask_output": false,  // Optional, defaults to False
    "confidence_threshold": 0.5  // Optional, minimum confidence for returned masks
  }
}
```

### API Response
```json
{
  "masks": ["base64_encoded_mask_1", "base64_encoded_mask_2"],
  "scores": [0.95, 0.87],
  "num_objects": 2,
  "sam_version": "3.0",
  "success": true
}
```

## 🆚 SAM3 vs SAM2

| Feature | SAM2 | SAM3 |
|---------|------|------|
| **Text Prompts** | ❌ | ✅ Natural language descriptions |
| **Point Prompts** | ✅ | ✅ (compatible) |
| **Box Prompts** | ✅ | ✅ (compatible) |
| **Quality** | High | Higher (concept-aware) |
| **API Format** | HF Inference Endpoints | ✅ Compatible + Extensions |

## 🔬 Technical Details

- **Model**: `facebook/sam3` from HuggingFace Transformers
- **Post-processing**: Official `post_process_instance_segmentation()` API
- **Framework**: Gradio 5.49.1 with automatic API generation
- **Dependencies**: Latest transformers with SAM3 support
- **Deployment**: HuggingFace Spaces (avoids Inference Toolkit compatibility issues)

## 📚 References

- [SAM3 Model Card](https://huggingface.co/facebook/sam3)
- [SAM3 Paper](https://ai.meta.com/research/publications/segment-anything-model-3/)
- [Transformers SAM3 Documentation](https://huggingface.co/docs/transformers/model_doc/sam3)