File size: 5,012 Bytes
f61db78 bbc1784 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 | ---
title: Smart Tiffin Packer
emoji: 🍱
colorFrom: yellow
colorTo: green
sdk: docker
app_port: 7860
pinned: false
---
# Smart Tiffin Packing Environment 🍱🤖
> **Semantic-aware constrained packing under real-world constraints**
>
> An OpenEnv-compliant RL environment where an LLM agent controls a robotic arm
> to pack an Indian tiffin meal. The agent uses VLM-derived food classification
> to reason about container compatibility, volume constraints, temperature zones,
> and fragility — then physically executes packing decisions.
## 🎯 What is this?
This environment simulates the real-world task of **packing an Indian meal into tiffin containers**. An AI agent must:
1. **Identify** food items using a Vision-Language Model (VLM)
2. **Reason** about which container each item should go into
3. **Execute** packing commands via a robotic arm
4. **Satisfy** multiple constraints simultaneously
### Why Tiffin Packing?
Every day, millions of people in India pack tiffin boxes for lunch. It's a genuine spatial-reasoning task with real constraints:
- Liquids (sambar, dal) must go in sealed containers
- Fragile items (papad, chapati) shouldn't be crushed
- Hot and cold foods should be separated
- Volume limits mean you can't just stuff everything in one box
## 🏗️ Architecture
```
LLM Agent (via OpenAI API)
│
├── observe → See scene description
├── identify → VLM classifies food item
├── pick → Robotic arm picks up food
├── place → Place item in container
└── pour → Pour liquid into container
│
▼
OpenEnv Server (FastAPI)
│
├── Simulation Engine (logic + PyBullet physics)
├── VLM Classifier (cached food_db.json)
├── Task Manager (easy/medium/hard)
└── Deterministic Grader (0.0-1.0)
```
## 🎮 Tasks
| Task | Items | Containers | Constraints | Difficulty |
|------|-------|-----------|-------------|------------|
| 🟢 Easy | rice, sambar (2) | sealed, flat (2) | Type matching | Straightforward |
| 🟡 Medium | rice, sambar, chapati, pickle (4) | sealed, flat, deep (3) | Types + overflow + temperature | Requires reasoning |
| 🔴 Hard | rice, sambar, curd, chapati, papad, curry (6) | sealed, flat, deep, small_sealed (4) | All constraints active | Genuinely challenging |
## 📊 Scoring (0.0 – 1.0)
| Component | Weight | Description |
|-----------|--------|-------------|
| Validity | 40% | Food placed in type-compatible container? |
| Efficiency | 30% | Space utilization vs capacity used |
| Constraints | 20% | Temperature, fragility, flavor isolation |
| Neatness | 10% | All items packed? Nothing dropped? |
## 🚀 Quick Start
### Run locally
```bash
pip install -r requirements.txt
uvicorn server.app:app --host 0.0.0.0 --port 7860
```
### Run inference
```bash
export API_BASE_URL=https://api.openai.com/v1
export MODEL_NAME=gpt-4o
export HF_TOKEN=your-api-key
export ENV_URL=http://localhost:7860
python inference.py
```
### Docker
```bash
docker build -t tiffin-packer .
docker run -p 7860:7860 tiffin-packer
```
## 🔧 Action Space
```json
{
"command": "identify | pick | place | pour | observe",
"target_id": 1
}
```
## 👁️ Observation Space
```json
{
"scene_description": "Natural language scene state",
"food_items": [{"id": 1, "name": "rice", "status": "on_table", ...}],
"containers": [{"id": 1, "type": "sealed_round", "capacity_ml": 300, ...}],
"held_item": null,
"vlm_result": {"type": "solid", "fragility": 0.1, ...},
"available_commands": ["observe", "identify", "pick"],
"step_feedback": "Successfully picked up rice"
}
```
## 📁 Project Structure
```
tiffen-packer/
├── openenv.yaml # OpenEnv manifest
├── inference.py # LLM inference script (OpenAI Client)
├── Dockerfile # HF Spaces deployment
├── tiffin_packer/ # Core package
│ ├── models.py # Pydantic Action/Observation/State
│ ├── simulation/
│ │ ├── engine.py # Logic simulation engine
│ │ └── pybullet_renderer.py # Physics visualization
│ ├── vlm/
│ │ ├── classifier.py # VLM food classifier
│ │ └── food_db.json # 15 Indian food items
│ ├── tasks.py # Easy/Medium/Hard task configs
│ └── grader.py # Deterministic scoring
└── server/
├── tiffin_environment.py # OpenEnv Environment
└── app.py # FastAPI server
```
## 🏆 OpenEnv Compliance
- ✅ Typed Pydantic models (Action, Observation, State)
- ✅ `step()` / `reset()` / `state()` API
- ✅ `openenv.yaml` manifest
- ✅ 3 tasks with deterministic graders (0.0–1.0)
- ✅ Dense reward function with partial progress signals
- ✅ Baseline inference script using OpenAI Client
- ✅ Docker deployment for HF Spaces
## 👥 Team
**CtrlAltWin** — Meta PyTorch OpenEnv Hackathon 2026
## 📄 License
MIT
|