README.md

---
language:
- en
tags:
- computer-use
pretty_name: Real-World Computer Use Agent Training Data
---

# Pango Sample: Real-World Computer Use Agent Training Data

**Pango** represents **P**roductivity **A**pplications with **N**atural **G**UI **O**bservations and trajectories.

## Dataset Description

This dataset contains authentic computer interaction data collected from users performing real work tasks in productivity applications. The data was collected through [Pango](https://pango.so), a crowdsourced platform where users are compensated for contributing their natural computer interactions during actual work sessions.

![Pango Screenshot](images/pango_1.png)

## Motivation

Current Computer Use Agent (CUA) training datasets face several limitations:
- **Scale constraints**: Existing datasets like Mind2Web (2,350 tasks), GUI-World (12,000 videos), and OSWorld (369 tasks) provide limited coverage
- **Artificial contexts**: Most demonstrations are scripted rather than authentic work sessions
- **Distribution gaps**: Performance drops significantly when agents encounter interfaces outside their training distribution
- **Missing error patterns**: Academic datasets typically exclude "failed" interactions, removing important recovery behaviors

This dataset addresses these limitations by capturing real users performing genuine work tasks, providing natural interaction patterns, error recovery sequences, and diverse problem-solving approaches.

## Data Collection Methodology

Data is collected through a Chrome extension that records user interactions during structured "quests" in target applications:
- **Applications**: Google Sheets, Google Slides, Figma, Canva (more coming soon)
- **User base**: Global contributor network across 180+ countries
- **Task context**: Authentic work sessions (financial analysis, presentation creation, design work, etc.)
- **Compensation**: Users are paid based on session length and data quality

## Dataset Structure

Each record contains:
- `id`: Unique session identifier
- `video_url`: Screen recording of the interaction session
- `input_metadata`: Structured JSON containing granular interaction events
- `input_metadata_preview_url`: Link to interactive data viewer for the `input_metadata`
- `task_description`: User-provided description of what they were doing
- `quest_type`: Application category (Sheets, Slides, Figma, Canva)
- `profession`: User's professional background
- `synthetically_generated_instruction`: Synthetically generated task instruction for training purposes. Represents the context of the full task.
- `synthetically_generated_thought_metadata`: (Beta) Synthetically generated thoughts for each user step. Represents the thought of the current step. Available by request.
- `artifact_history_preview_url`: Link to the full history of the data artifact. Currently only Canva data. Useful for code generation in addition to pure CUA. 

### Input Metadata Schema

The `input_metadata` field contains timestamped interaction events with the following structure:

```json
{
  "relative_timestamp_ms": 1028,
  "type": "click",
  "x": 186.0,
  "y": 62.445,
  "button": "button_left",
  "screenshot_url": "https://...",
  "click_count": 1
}
```


**Key fields:**
- `relative_timestamp_ms`: Milliseconds since session start
- `type`: Event type (click, input, key_press, mouseover_start, mouseover_end, drag_start, drag_end, scroll)
- `x,y`: Screen coordinates (normalized for display resolution)
- `screenshot_url`: URL to corresponding interface screenshot
- `text`: Text content for input events - only available on input events
- `key_codes`: Keyboard key identifier (DOM KeyboardEvent codes) - only available on key_press events

### Previewing the Data

While the `video_url` is provided for completeness, it is often clunky to interact with `webm` files and it's impractical to leverage these for most use cases. 

The `input_metadata` is a much more convenient way to interact with the data, however it is not as easy to visualize. To solve this, we built an interactive preview tool, which takes the `input_metadata` and renders it frame-by-frame with corresponding actions. 

This can be found [here](https://admin.pango.so/actions-preview?url=https://pango-service-production.s3.us-east-1.amazonaws.com/finalized/bc50d07a-d11a-4c9c-9c5a-e5fdb6e85b5f/actions_v2.json) and it takes any valid `input_metadata` URL as input.

![Preview Tool Screenshot](images/preview_tool.png)

## Thought Metadata (Beta)

An additional field, `synthetically_generated_thought_metadata`, is included to provide synthetically generated thoughts for each user step. This field is designed to enhance the dataset's utility for training reasoning VLMs like [UI-TARS 1.5](https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B). It is not to be confused with `synthetically_generated_instruction`, which is the context of the full task. This field is available by request. 

**Step Generation and Aggregation**

To create `thought_metadata`, we begin with the `input_metadata` where each row represents an individual user action. As a first stage, we aggregate actions into steps, where each step represents either a single action or a collection of actions.

**Batch Processing Strategy**

We partition the steps into batches with the following parameters:

$$
\alpha = 7, \quad \beta = 15, \quad \gamma = 15
$$

where:  
- \\(\alpha\\) (**pre_window_size**): Number of steps preceding the target step used for context  
- \\(\beta\\) (**post_window_size**): Number of subsequent steps used for context  
- \\(\gamma\\) (**batch_size**): Interval between target steps for thought generation. i.e. if \\(\gamma = 15\\), then for every 15 steps, we generate a thought.

The \\(\gamma = 15\\) prevents overlapping thoughts and ensures that thoughts generated in earlier batches are considered completed when generating subsequent thoughts.

**LLM Usage**

The processed step batches are fed to GPT-4o with its vision API, using high image detail settings. Each thought generation process consumes approximately 30,000 input tokens.

## Quality Assurance

Data quality is maintained through:
- Automated filtering of invalid interactions and privacy-sensitive content
- Quality scoring based on task coherence and completion patterns
- Compensation algorithms that reward genuine engagement
- Differential privacy techniques to prevent individual behavior reconstruction

## Use Cases

This dataset is designed for:
- Training computer use agents on authentic interaction patterns
- Studying human-computer interaction behaviors across diverse populations
- Developing more robust GUI automation systems
- Research on temporal reasoning and error recovery in sequential decision-making

## Ethical Considerations

- All users provide informed consent for data collection and redistribution
- Privacy-sensitive content is automatically filtered
- Compensation ensures fair value exchange for user contributions
- Data collection follows ethical guidelines for crowdsourced research

## Data Scale and Growth

The dataset is continuously growing through ongoing collection:
- Planned: Scaling to 100,000+ hours over 2025

## Citation

If you use this dataset in your research, please cite:

```bibtex
@dataset{pango2025,
  title={Pango: Real-World Computer Use Agent Training Data},
  author={Chakra Labs},
  year={2025},
  url={https://huggingface.co/datasets/chakra-labs/pango}
}
```

## Contact

For access to the full dataset or collaboration opportunities, please contact [Chakra Labs](mailto:nirmal@chakra-labs.com).