processor/README.md

 

processor

English | 中文

Documentation: 2026-09-28 · Backend: 1.19.4 · Frontend: 1.12.3

processor is EgoData's service for data capture, processing, review, and export. It is designed for embodied AI and robot manipulation data. The system consists of a FastAPI backend, an asynchronous Worker, Web review pages, and Workflow Studio.

Beginners: start with the illustrated user guide.

Capture upload → Project matching → Workflow processing → AI/Human review → LeRobot/HDF5 export

Documentation navigation

• System overview
• Installation and startup
• Code layout
• Project data structure
• Video data specification
• Workflow modules
• Web Workflow Studio modules
• Processing result storage
• Configuration
• API
• Testing and troubleshooting

1. System overview

Capture client
  │
  ▼
FastAPI backend
  ├── Project, Episode, workflow, and review APIs
  ├── Upload reception, extraction, normalization, and validation
  └── Worker task scheduling
  │
  ▼
Worker
  ├── MediaPipe hand keypoints
  ├── RGB-D metric 3D hand keypoints
  ├── AI annotation through an API
  └── Video and annotation quality review
  │
  ▼
Web review pages
  ├── Original RGB video
  ├── Browser-side SVG/Canvas keypoint overlays
  └── Pseudo-color depth preview
  │
  ▼
LeRobot v2.1 / v3.0 or HDF5

The local processor/ directory is the only source of frontend and backend code. The remote server provides data storage or SFTP access only; its deprecated frontend and backend code must not be used as the project source.

2. Code layout

Click to expand: code layout

processor/
├── app/                    # FastAPI, storage, workflows, processing, and export logic
├── worker/                 # Asynchronous processing Worker
├── scripts/                # Deployment, startup, checks, and maintenance scripts
├── web/
│   ├── templates/          # Web page templates
│   ├── static/             # CSS, JavaScript, and frontend build artifacts
│   └── workflow-studio/    # React + Vite workflow editor source
├── tests/                  # Automated tests
├── requirements*.txt       # Python dependency manifests
├── deploy.py               # Cross-platform deployment entry point
├── deploy.sh               # Linux deployment entry point
├── deploy.bat              # Windows deployment entry point
└── .env.example            # Configuration template

Runtime data is stored under STORAGE_DIR by default:

data/
├── sessions/               # Project datasets
├── state/                  # System state, queues, run records, and export jobs
└── tmp/                    # Upload and Worker temporary files

Runtime data, model weights, archives, credentials, and virtual environments must not be committed to Git. Dependency manifests such as requirements*.txt and package.json should be kept in the repository.

## 3. Project data structure

Click to expand: project data structure, Episodes, Chunks, and metadata

A project is the root directory of one dataset. Its canonical root contains exactly data/, meta/, and videos/; no additional batch-level data directory is created under the project:

data/sessions/<project>/
├── data/
│   └── chunk-000/
│       ├── episode_000000.parquet
│       └── episode_000001.parquet
├── meta/
│   ├── info.json
│   ├── stats.json
│   ├── tasks.json
│   └── episodes/
│       └── chunk-000/
│           ├── episode_000000.parquet
│           └── episode_000001.parquet
└── videos/
    └── chunk-000/
        ├── observation.images.head_rgb/
        │   └── episode_000000.mp4
        └── observation.images.head_depth/
            └── episode_000000.mp4

Episodes and Chunks

• Episode numbering starts at 000000;
• episode_000000 through episode_000999 are stored in chunk-000;
• episode_001000 starts chunk-001;
• each chunk contains at most 1,000 Episodes;
• data, meta/episodes, and videos use the same Episode/Chunk mapping;
• the capture batch name is used for business tracking and does not replace the dataset's episode_index.

Meta directory

The canonical meta/ directory contains:

• info.json: dataset version, FPS, frame counts, features, and path templates;
• stats.json: statistics for dataset fields;
• tasks.json: task IDs and task descriptions;
• episodes/chunk-XXX/episode_XXXXXX.parquet: per-Episode timestamps, device information, video information, capture metadata, calibration information, and processing state.

Timestamps, device information, and calibration information are written to the corresponding Episode. Review and annotation pages use only the actual videos, depth streams, and metadata of the current Episode.

4. Video data specification

Click to expand: video sources, depth storage, and browser pseudo-color preview

Video sources

Type	Example source name	Description
Mono RGB	head_rgb	Regular RGB video
Metric depth	head_depth	Independent metric-depth video
Stereo RGB	stereo_left_rgb, stereo_right_rgb	Left and right RGB videos
Stereo depth	stereo_depth	Independent depth video

RGB and Depth are always stored as independent streams. A depth-only stream is never treated as RGB and is never duplicated alongside the RGB stream.

Depth storage

New metric-depth videos use:

HEVC (H.265) in MP4 / hvc1 / gray12le / qp=6 / range=full

The depth video stores 12-bit logarithmic depth codes, not pseudo-color images:

100–5000 mm logarithmic depth → 0..4095 gray12le codes

The corresponding feature in meta/info.json is marked with video.is_depth_map=true and the depth encoding parameters. Readers dequantize the codes back to millimeters. The original depth video is preserved during processing and export.

Frontend depth preview

Pseudo-color depth is used only for browser preview and is never written back to the dataset:

HEVC gray12le
      │
      └── Browser Canvas: depth codes → JET pseudo-color

Preview color convention:

• invalid depth: black;
• near range: blue-violet;
• middle range: green and yellow;
• far range: orange and red.

Depth preview data is prefetched when an Episode is opened. RGB video, stereo RGB video, and export data are not pseudo-colorized, and no additional pseudo-color video is generated.

5. Workflow modules

Click to expand: input, processing, review, export modules, and node states

Workflows consist of Input, Process, Review, and Export modules. Ports have explicit data types; each connected branch processes only the stream connected to that port. A branch without an input connection is marked Skipped · Missing Input and does not block other connected branches.

Input modules

• RGB Camera: single-view RGB video;
• RGB-D Camera: RGB video and Depth;
• Stereo RGB Camera: Left RGB Video and Right RGB Video;
• Stereo RGB-D Camera: left/right RGB video and Depth;
• Glove Sensor: glove pressure, joint, or sensor data.

Process modules

• MediaPipe Hand: extracts 2D hand keypoints from RGB video;
• Human Annotation: manual annotation;
• AI Annotation: video/segment annotation through the configured API;
• RGB_TO_2D_BareHand: 2D bare-hand keypoints;
• RGB_TO_2D_BlackGlove: 2D black-glove keypoints;
• RGB-D_3D_BareHand: true metric 3D bare-hand keypoints from RGB video + Depth;
• RGB-D_3D_BlackGlove: true metric 3D black-glove keypoints from RGB video + Depth.

The Spatial button on RGB_TO_2D_* only controls a browser visualization. It does not create fake 3D data or add 3D values to exports. The RGB Video and Depth ports on RGB-D_3D_* are independent inputs and should be connected to the corresponding capture outputs.

Web Workflow Studio frontend module reference

The following modules are the ones currently registered by the frontend palette and canvas. Node names, port names, and connection types are maintained by the frontend registry and are not overwritten by a stale workflow or deprecated backend catalog. The legacy types rgb_hand_3d, black_hand_rgb_3d, stereo_triangulate, and black_glove_hand are mapped only when old workflows are loaded; they are not shown in the new-workflow palette.

Capture modules (Input)

Frontend module	Inputs	Outputs	Purpose and rules
RGB Camera	None	RGB Video	Mono RGB video. The device is selected from the current project's data; blank means auto-match.
RGB-D Camera	None	RGB Video, Depth	Emits RGB and real depth streams from the same device; Depth can connect only to a depth input.
Stereo RGB Camera	None	Left RGB Video, Right RGB Video	Stereo RGB input. Connecting one side creates the corresponding left/right relationship.
Stereo RGB-D Camera	None	Left RGB Video, Right RGB Video, Depth	Stereo RGB plus real depth; Depth is used by RGB-D 3D processing.
Glove Sensor	None	Glove Sensor Data	Pressure, joint, or glove sensor data; no video output. It can connect directly to quality review.

Video and keypoint processing modules (Process)

Frontend module	Inputs	Outputs	Purpose and rules
Human Annotation	RGB Video	Annotation	Frame-level manual annotation on RGB video.
AI Annotation	RGB Video	Annotation	Video/segment annotation through a local or API VLM; API is recommended for deployment.
RGB_TO_2D_BareHand	RGB Video	Hand 2D	Bare-hand 2D keypoints.Spatial is display-only and exports no metric 3D.
RGB_TO_2D_BlackGlove	RGB Video	Hand 2D	Black-glove 2D keypoints. Without Depth, it does not calculate metric 3D.
RGB-D_3D_BareHand	RGB Video, Depth	Hand 3D	Metric 3D bare-hand keypoints from RGB and real Depth; a missing branch is skipped.
RGB-D_3D_BlackGlove	RGB Video, Depth	Hand 3D	Metric 3D black-glove keypoints from RGB and real Depth; it does not search the disk or fall back to RGB estimation.

MediaPipe Hand remains available for historical workflows and backend compatibility but is hidden from the new-workflow palette. All nodes provide hover text describing their purpose and accepted connection types.

Review modules (Review)

Frontend module	Inputs	Outputs	Purpose and rules
Human Review	Review Target	Reviewed Data	Human inspection of video, keypoints, or annotation results before downstream processing.
AI Quality Review	Quality Review Target	Reviewed Data	Checks video decoding, frame continuity, black screens, freezes, annotation coverage, and sensor data.

Export modules (Export)

Frontend module	Inputs	Outputs	Purpose and rules
LeRobot Export	Exportable Data	Dataset	Exports LeRobot v2.1 or v3.0; the version is selected on the node.
HDF5 Export	Exportable Data	Dataset	Exports HDF5 with the configured compression parameters.

Workflow Studio UI components

Click to expand: frontend component reference

UI component	Function
NodePalette	Displays Input, Process, Review, and Export categories; supports search, category collapse, and hover descriptions.
WorkflowCanvas	Supports drag-and-drop node creation, typed-port connections, node movement, selection, zoom, grid snapping, and minimap navigation.
WorkflowNode	Renders the card title, ports, project device selector/input,Spatial button, and API settings button.
WorkflowDrawer	Lists workflows, creates new workflows, loads workflows, and refreshes project-specific input sources.
PipelineToolbar	Provides New, Save, Save As, workflow JSON Export, and Run actions; the yellow dot indicates unsaved changes.
NodeSettingsModal	Configures AI Annotation API vendor, model, endpoint, and key fields.
DeletableEdge	Renders data connections; a selected edge can be deleted.

Device selection is always scoped to the current project. A new empty workflow does not display a device from another project. After data is uploaded, each card's selector shows only real sources matching that card's RGB, RGB-D, stereo, or glove category.

Review and execution rules

Connect inputs and outputs by their port names. An unconnected processing branch is skipped and does not affect other branches. RGB-D 3D produces true metric 3D only when both RGB Video and Depth are available.

Node states are limited to:

Connected
Processing
Completed
Skipped · Missing Input
Completed with Warning

6. Processing result storage

Click to expand: processing results, upload extraction, and cleanup rules

When a Worker finishes processing, its results are merged into the corresponding Episode:

data/chunk-000/episode_000000.parquet
  └── 2D keypoints, metric 3D, sensor, and annotation fields

meta/episodes/chunk-000/episode_000000.parquet
  └── processing_*, run_id, node states, and output indexes

Original RGB/Depth videos are not overwritten. Hand-keypoint previews are overlaid on the original RGB video in the browser; a second skeleton video is not saved. System-level run records are kept under data/state/runs/, and temporary export products under data/state/exports/; neither is part of the project dataset directory.

The upload archive lifecycle is: receive → extract → normalize → write to project → validate → update status. Temporary upload files and extraction directories are cleaned only after extraction, atomic commit, and validation all succeed. On failure, staging content is retained for retry and diagnosis.

7. Installation and startup

Linux

cd "processor"
cp .env.example .env
# Edit .env and set API_KEY, WORKER_API_KEY, JWT_SECRET, and storage settings.
chmod +x deploy.sh
./deploy.sh

Common options:

python deploy.py --check-only       # Check only; do not install, change config, or start services.
python deploy.py --skip-vllm        # Do not start local VLM; use an API for AI annotation.
python deploy.py --no-services      # Run in the foreground without installing startup services.

The current AI annotation deployment uses an API. Use --skip-vllm, then configure the API vendor, model, and endpoint in the AI Annotation settings in Workflow Studio.

Windows

cd /d "processor"
copy .env.example .env && deploy.bat

Start the backend manually

cd "processor"
source .venv-linux/bin/activate
export PYTHONPATH="$PWD"
./scripts/run_backend_linux.sh

Alternatively:

uvicorn app.main:app --host 0.0.0.0 --port 8000

Start the Worker

cd "processor"
export EGODATA_SERVER_URL=http://127.0.0.1:8000
export EGODATA_WORKER_API_KEY='same Worker API key as the server'
./scripts/run_worker_linux.sh

On Windows, use scripts/run_worker_windows.ps1. Linux and Windows virtual environments are independent and must not be copied between operating systems.

Workflow Studio

cd "processor/web/workflow-studio"
npm ci
npm run dev

Production build:

npm run build

The build output is written to web/static/workflow-studio/ and served by FastAPI.

8. Configuration

Click to expand: environment variables and security configuration

Configuration is stored in processor/.env or environment variables and must not be committed:

Setting	Purpose
STORAGE_DIR	Root directory for data and system state
STORAGE_BACKEND	local or sftp
SFTP_*	Remote data directory and SSH connection settings
API_KEY	API key for the capture client and protected endpoints
WORKER_API_KEY	API key used by the Worker to claim and return jobs
JWT_SECRET	Secret used to sign Web login sessions
UPLOAD_STAGING_DIR	Local staging directory for upload archives
HOST / PORT	Web/API listen address and port
PUBLIC_BASE_URL	Public service URL

See .env.example for the complete template. Production environments must use random secrets and restrict .env permissions. Never put passwords, tokens, API keys, or database credentials in logs, workflow JSON, frontend code, or documentation.

9. API

Click to expand: primary APIs and OpenAPI documentation

After startup, these endpoints are available:

GET /health
GET /docs
GET /openapi.json

Primary API groups:

/api/v1/auth/*          Login and authentication
/api/v1/projects/*      Projects and Episodes
/api/v1/workflows/*     Workflow definitions
/api/v1/session/*       Capture-client uploads
/api/v1/video/*         Video, depth, and keypoint previews
/api/v1/annotations/*   Annotation and review
/api/v1/export/*        Dataset export
/api/v1/worker/*        Worker job claiming and callbacks

The running FastAPI OpenAPI document is the authoritative API contract.

10. Testing and troubleshooting

Click to expand: test commands and common troubleshooting steps

Backend and Worker checks:

cd "processor"
python -m compileall app worker
python deploy.py --check-only
PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python -m pytest -q tests/test_depth_codec.py

Frontend check:

cd "processor/web/workflow-studio"
npm run build

Troubleshooting order:

1. Open /health and confirm that the service and storage are available;
2. check that STORAGE_DIR points to the actual data directory;
3. inspect the upload status and the actual files for the current Episode;
4. check that the Worker is online, can claim jobs, and sends heartbeats;
5. verify that Episode numbering matches across data, meta/episodes, and videos;
6. for RGB-D 3D nodes, verify that RGB Video and Depth are connected separately;
7. if depth preview is incorrect, verify that the video uses gray12le and is not a pseudo-color preview saved as source depth data.

11. Versioning

Click to expand: version maintenance rules

• The backend version is maintained in app/version.py and covers the backend only;
• the frontend version is maintained in web/VERSION. It covers every piece of frontend code under processor/web/ (templates/, static/, workflow-studio/) — not just the React SPA. The version inside web/workflow-studio/package.json only represents the SPA itself; all three numbers are independent and do not need to stay in sync.
• Bump web/VERSION — that one file — whenever you change the frontend. A single number serves two purposes: ① the ?v={{ asset_version }} cache key on static assets (so browsers re-fetch changed JS/CSS), ② the "frontend X" part of commit messages. Sharing one number means the commit message can never disagree with what the browser actually loaded. You do not need to touch the backend version for a frontend change.
• web/VERSION is read once at startup and cached (app/web_version.py), so restart the service after bumping it.
• increment the patch version for fixes and small adjustments, the minor version for new features, and the major version for architectural changes;
• before release, record the software version, data format, and API changes together.

Current versions: backend 1.19.4, frontend 1.12.3.