phongphamthe/inference-endpoint-poc

HF Vision Inference Harness

This directory is intended to be pushed as the root of a dedicated Hugging Face model repository and deployed as a custom Hugging Face Inference Endpoint.

It answers the production decision question for the current SAM3 batch-processing POC by combining:

• a custom EndpointHandler
• explicit workload and SLO configuration
• transport and result contracts
• deployment and smoke-test artifacts

Hub-ready repository layout

Publish these contents at the root of the Hugging Face Hub repository:

.
├── .env.example
├── contracts/
│   ├── request.schema.json
│   └── result.schema.json
├── config/
│   └── inference-requirements.yaml
├── examples/
│   ├── depth.request.json
│   └── segmentation.request.json
├── DEPLOYMENT_CHECKLIST.md
├── handler.py
├── README.md
└── requirements.txt

The Hugging Face Inference Endpoint service only requires handler.py and requirements.txt at the repo root. The rest of the files are there to keep deployment, contracts, and smoke testing reproducible.

Environment variables

Set these in the Endpoint configuration UI rather than committing secrets:

• HF_TOKEN: required for gated model access such as facebook/sam3
• HF_SAM_MODEL_ID: defaults to facebook/sam3
• HF_DEPTH_MODEL_ID: defaults to depth-anything/Depth-Anything-V2-Small-hf

See .env.example for the expected values.

Request shape

The handler expects a payload like:

{
  "inputs": [
    {
      "image_id": "image-0",
      "image_base64": "..."
    }
  ],
  "task": "object_segmentation",
  "parameters": {
    "text_query": "pavement crack",
    "confidence_threshold": 0.5
  }
}

See examples/segmentation.request.json and examples/depth.request.json for ready-to-send payload templates.

Runtime behavior

The current handler supports two task paths:

• object_segmentation: SAM3 segmentation with query-driven mask output, bounding-box derivation, and base64 PNG mask serialization
• depth_estimation: depth model inference with resized grayscale depth PNG output and min/max depth metadata

The batch result shape intentionally matches the normalized response style already used in the existing Space app, including item-level status, retryable, and timing blocks.

Endpoint deployment target

For the first controlled benchmark deployment, use these settings from config/inference-requirements.yaml:

• custom handler endpoint
• min replicas: 2
• max replicas: 4
• scale to zero: disabled
• initial benchmark focus: segmentation first, then depth

If Datadog export is part of the acceptance criteria, use a Team or Enterprise plan so the endpoint can expose OpenMetrics.

Smoke test sequence

1. Deploy the repository as a custom endpoint.
2. Set HF_TOKEN in endpoint environment variables.
3. Send a one-image segmentation request with examples/segmentation.request.json.
4. Verify status, results[0].sam.predictions, and timing fields.
5. Only after segmentation succeeds, send a one-image depth request.

Manual local check

From the workspace root, run:

python -m pytest tests/test_endpoint_handler.py

If the local environment lacks Python or the required packages, deploy directly to a Hugging Face endpoint and use the smoke test sequence above.