README.md

metadata

title: Smart Tiffin Packer
emoji: 🍱
colorFrom: yellow
colorTo: green
sdk: docker
app_port: 7860
pinned: false

Smart Tiffin Packing Environment 🍱🤖

Semantic-aware constrained packing under real-world constraints

An OpenEnv-compliant RL environment where an LLM agent controls a robotic arm to pack an Indian tiffin meal. The agent uses VLM-derived food classification to reason about container compatibility, volume constraints, temperature zones, and fragility — then physically executes packing decisions.

🎯 What is this?

This environment simulates the real-world task of packing an Indian meal into tiffin containers. An AI agent must:

1. Identify food items using a Vision-Language Model (VLM)
2. Reason about which container each item should go into
3. Execute packing commands via a robotic arm
4. Satisfy multiple constraints simultaneously

Why Tiffin Packing?

Every day, millions of people in India pack tiffin boxes for lunch. It's a genuine spatial-reasoning task with real constraints:

• Liquids (sambar, dal) must go in sealed containers
• Fragile items (papad, chapati) shouldn't be crushed
• Hot and cold foods should be separated
• Volume limits mean you can't just stuff everything in one box

🏗️ Architecture

LLM Agent (via OpenAI API)
    │
    ├── observe → See scene description
    ├── identify → VLM classifies food item
    ├── pick → Robotic arm picks up food
    ├── place → Place item in container
    └── pour → Pour liquid into container
    │
    ▼
OpenEnv Server (FastAPI)
    │
    ├── Simulation Engine (logic + PyBullet physics)
    ├── VLM Classifier (cached food_db.json)
    ├── Task Manager (easy/medium/hard)
    └── Deterministic Grader (0.0-1.0)

🎮 Tasks

Task	Items	Containers	Constraints	Difficulty
🟢 Easy	rice, sambar (2)	sealed, flat (2)	Type matching	Straightforward
🟡 Medium	rice, sambar, chapati, pickle (4)	sealed, flat, deep (3)	Types + overflow + temperature	Requires reasoning
🔴 Hard	rice, sambar, curd, chapati, papad, curry (6)	sealed, flat, deep, small_sealed (4)	All constraints active	Genuinely challenging

📊 Scoring (0.0 – 1.0)

Component	Weight	Description
Validity	40%	Food placed in type-compatible container?
Efficiency	30%	Space utilization vs capacity used
Constraints	20%	Temperature, fragility, flavor isolation
Neatness	10%	All items packed? Nothing dropped?

🚀 Quick Start

Run locally

pip install -r requirements.txt
uvicorn server.app:app --host 0.0.0.0 --port 7860

Run inference

export API_BASE_URL=https://api.openai.com/v1
export MODEL_NAME=gpt-4o
export HF_TOKEN=your-api-key
export ENV_URL=http://localhost:7860
python inference.py

Docker

docker build -t tiffin-packer .
docker run -p 7860:7860 tiffin-packer

🔧 Action Space

{
  "command": "identify | pick | place | pour | observe",
  "target_id": 1
}

👁️ Observation Space

{
  "scene_description": "Natural language scene state",
  "food_items": [{"id": 1, "name": "rice", "status": "on_table", ...}],
  "containers": [{"id": 1, "type": "sealed_round", "capacity_ml": 300, ...}],
  "held_item": null,
  "vlm_result": {"type": "solid", "fragility": 0.1, ...},
  "available_commands": ["observe", "identify", "pick"],
  "step_feedback": "Successfully picked up rice"
}

📁 Project Structure

tiffen-packer/
├── openenv.yaml          # OpenEnv manifest
├── inference.py           # LLM inference script (OpenAI Client)
├── Dockerfile             # HF Spaces deployment
├── tiffin_packer/         # Core package
│   ├── models.py          # Pydantic Action/Observation/State
│   ├── simulation/
│   │   ├── engine.py      # Logic simulation engine
│   │   └── pybullet_renderer.py  # Physics visualization
│   ├── vlm/
│   │   ├── classifier.py  # VLM food classifier
│   │   └── food_db.json   # 15 Indian food items
│   ├── tasks.py           # Easy/Medium/Hard task configs
│   └── grader.py          # Deterministic scoring
└── server/
    ├── tiffin_environment.py  # OpenEnv Environment
    └── app.py                 # FastAPI server

🏆 OpenEnv Compliance

• ✅ Typed Pydantic models (Action, Observation, State)
• ✅ step() / reset() / state() API
• ✅ openenv.yaml manifest
• ✅ 3 tasks with deterministic graders (0.0–1.0)
• ✅ Dense reward function with partial progress signals
• ✅ Baseline inference script using OpenAI Client
• ✅ Docker deployment for HF Spaces

👥 Team

CtrlAltWin — Meta PyTorch OpenEnv Hackathon 2026

📄 License

MIT