Spaces:

Shanur
/

oculus-ui-detector

Runtime error

import gradio as gr
from modules.element_processing import process_screenshot
from modules.element_detector import initialize_models
import asyncio
import io
import os
import numpy as np
from PIL import Image

# Initialize models on startup
async def init():
    await initialize_models()

loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(init())

# Create a mock background tasks class
class MockBackgroundTasks:
    def add_task(self, *args, **kwargs):
        pass

# Define interface
def process_image(image):
    # Convert image to bytes
    img_byte_arr = io.BytesIO()
    image.save(img_byte_arr, format='PNG')
    image_data = img_byte_arr.getvalue()
    
    # Process the screenshot
    background_tasks = MockBackgroundTasks()
    elements, image_path = loop.run_until_complete(
        process_screenshot(image_data, background_tasks)
    )
    
    # Format output
    result_text = "\n".join([
        f"icon {element['code']}: {{"
        f"'type': '{element['type']}', "
        f"'centerX': {element['center_x']}, "
        f"'centerY': {element['center_y']}, "
        f"'content': '{element.get('text_content', element.get('object_label', ''))}'}}"
        for element in elements
    ])
    
    # Load annotated image
    annotated_img = None
    if image_path and os.path.exists(image_path):
        annotated_img = Image.open(image_path)
    
    return annotated_img, result_text

# Create interface
demo = gr.Interface(
    fn=process_image,
    inputs=gr.Image(type="pil"),
    outputs=[
        gr.Image(type="pil", label="Annotated Image"),
        gr.Textbox(label="Detected Elements")
    ],
    title="UI Element Detection",
    description="Upload a screenshot to detect UI elements"
)

# Launch the app
demo.launch()

Update requirements.txt

Update the requirements.txt file to include Gradio and other necessary dependencies:

fastapi==0.110.0
uvicorn==0.27.1
pydantic==2.6.3
pillow==10.2.0
opencv-python==4.9.0.80
easyocr==1.7.2
ultralytics==8.3.97
numpy<2
gradio>=4.0.0

Step 3: Configure Environment

Create a .env file in the root directory with essential configuration:

OCR_BACKEND=easyocr
ANNOTATION_DIR=annotated
FONT_SIZE=24
JPEG_QUALITY=85

Ensure the annotated directory exists:
```
mkdir -p annotated
```

Step 4: Set Up the Model Weights

Make sure your YOLOv8 model weights are included in the repository
If using Git LFS, add a .gitattributes file to track large model files:
```
weights/*.pt filter=lfs diff=lfs merge=lfs -text
```

Step 5: Create a HuggingFace Space

Create a new Space on HuggingFace:

huggingface-cli repo create oculus-ui-detector --type space

Clone the newly created Space:

git clone https://huggingface.co/spaces/YOUR_USERNAME/oculus-ui-detector

Add all your files to the Space:

cp -R . /path/to/oculus-ui-detector
cd /path/to/oculus-ui-detector

Commit and push your changes:

git add .
git commit -m "Initial commit"
git push

Step 6: Configure the Space

Visit your Space on HuggingFace and:

Set the SDK to "Gradio"
Set the Space hardware (recommended: GPU enabled)
Add any required Secrets in the Settings tab if you're using Google OCR or other services requiring API keys

Step 7: API Access

The Gradio interface has API access enabled. There are two ways to access your app via API:

Option 1: Using Gradio's Client API

You can interact with your app programmatically using the Gradio client:

import gradio as gr

# Connect to your HuggingFace Space
client = gr.Client("https://huggingface.co/spaces/YOUR_USERNAME/oculus-ui-detector")

# Call the function with a data URI
result = client.predict(
    "data:image/png;base64,YOUR_BASE64_DATA_HERE",  # data_uri
    api_name="/process_image_data_uri"
)

# Or upload an image directly
with open("screenshot.png", "rb") as f:
    result = client.predict(
        f,  # image file
        api_name="/process_image_upload"
    )

print(result)

Option 2: Using REST API

You can also use the REST API directly:

import requests
import json
import base64

# For data URI endpoint
url = "https://YOUR_USERNAME-oculus-ui-detector.hf.space/api/process_image_data_uri"

# Read and encode image to base64
with open("screenshot.png", "rb") as image_file:
    encoded_string = base64.b64encode(image_file.read()).decode('utf-8')
    data_uri = f"data:image/png;base64,{encoded_string}"

# Make the API request
response = requests.post(
    url,
    json={"data": [data_uri]}
)

print(json.loads(response.content)["data"])

API Documentation

The API endpoints will be automatically documented at:

https://YOUR_USERNAME-oculus-ui-detector.hf.space/docs

This documentation shows all available API endpoints and their parameters.

Step 7: Deployment Options

HuggingFace Spaces provides several deployment options:

Basic Deployment: The app will be built automatically when you push to the repository
Docker Deployment: For advanced configuration, you can use a Dockerfile to customize the environment
Custom Domain: Connect your Space to a custom domain in the Settings tab

Troubleshooting

If you encounter issues during deployment:

Check the Space logs for error messages
Ensure all dependencies are correctly specified
Verify model files are correctly uploaded
Make sure the annotated directory is properly configured

For model size issues:

Consider using huggingface_hub to download your models on startup
Use a smaller version of the model
Use model quantization to reduce size

Deploying Oculus Server to HuggingFace Spaces

Step 1: Prepare for Deployment

Step 2: Create Required Files

Create app.py