DATA_STRUCTURE.md

# OpenHands Index Data Structure

This document describes the expected data structure for the `openhands-index-results` GitHub repository.

## Repository Structure

The data should be organized in the following structure:

```
openhands-index-results/
├── 1.0.0-dev1/              # Version directory (matches CONFIG_NAME in config.py)
│   ├── test.jsonl            # Test split results
│   ├── validation.jsonl      # Validation split results
│   ├── swe-bench.jsonl       # Individual benchmark results
│   ├── multi-swe-bench.jsonl
│   ├── swe-bench-multimodal.jsonl
│   ├── swt-bench.jsonl
│   ├── commit0.jsonl
│   ├── gaia.jsonl
│   └── agenteval.json        # Configuration file
```

## File Formats

### JSONL Files (test.jsonl, validation.jsonl, etc.)

Each line in a JSONL file should be a JSON object representing one agent's results:

```json
{
  "Agent_Name": "OpenHands CodeAct v2.1",
  "Llm_Base": "claude-3-5-sonnet-20241022",
  "Openness": "closed_api_available",
  "Tool_Usage": "standard",
  "Score": 48.3,
  "Metric": "resolve_rate",
  "Submission_Time": "2025-11-24T19:56:00.092865",
  "Tags": ["swe-bench"],
  "Total_Cost": 34.15,
  "Total_Runtime": 541.5
}
```

### Configuration File (agenteval.json)

The configuration file defines the benchmark structure:

```json
{
  "suite_config": {
    "name": "openhands-index",
    "version": "1.0.0-dev1",
    "splits": [
      {
        "name": "test",
        "tasks": [
          {
            "name": "swe-bench",
            "tags": ["swe-bench"]
          },
          {
            "name": "multi-swe-bench",
            "tags": ["multi-swe-bench"]
          },
          {
            "name": "swe-bench-multimodal",
            "tags": ["swe-bench-multimodal"]
          },
          {
            "name": "swt-bench",
            "tags": ["swt-bench"]
          },
          {
            "name": "commit0",
            "tags": ["commit0"]
          },
          {
            "name": "gaia",
            "tags": ["gaia"]
          }
        ]
      },
      {
        "name": "validation",
        "tasks": [
          {
            "name": "swe-bench",
            "tags": ["swe-bench"]
          },
          {
            "name": "multi-swe-bench",
            "tags": ["multi-swe-bench"]
          },
          {
            "name": "swe-bench-multimodal",
            "tags": ["swe-bench-multimodal"]
          },
          {
            "name": "swt-bench",
            "tags": ["swt-bench"]
          },
          {
            "name": "commit0",
            "tags": ["commit0"]
          },
          {
            "name": "gaia",
            "tags": ["gaia"]
          }
        ]
      }
    ]
  }
}
```

## Data Loading Process

1. **GitHub Repository Check**: The app first attempts to clone the `openhands-index-results` repository
2. **Version Directory**: Looks for a directory matching `CONFIG_NAME` (currently "1.0.0-dev1")
3. **Fallback to Mock Data**: If GitHub data is unavailable, falls back to local mock data in `mock_results/`
4. **Data Extraction**: Copies data to `/tmp/oh_index/data/{version}/extracted/{version}/`

## Updating Data

To update the leaderboard data:

1. Push new JSONL files to the `openhands-index-results` repository
2. Ensure the version directory matches `CONFIG_NAME` in `config.py`
3. The app will automatically fetch the latest data on restart

## Mock Data

Mock data is stored in `mock_results/1.0.0-dev1/` and is used:
- During development and testing
- When the GitHub repository is unavailable
- As a template for the expected data format