Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -27,88 +27,6 @@ The public subset contains 120 tasks across Global (English) and Chinese markets
|
|
| 27 |
- π» Code: <https://github.com/ICBCBench/ICBCBench>
|
| 28 |
- π Dataset: <https://huggingface.co/datasets/DeepFin-Intelligence/ICBCBench>
|
| 29 |
|
| 30 |
-
## Repository Layout
|
| 31 |
-
|
| 32 |
-
```
|
| 33 |
-
βββ app.py # Gradio entry point
|
| 34 |
-
βββ create_leaderboard.py # Main Gradio Blocks UI
|
| 35 |
-
βββ tabs/
|
| 36 |
-
β βββ leaderboard_tab.py # Leaderboard rendering
|
| 37 |
-
β βββ data_viewer_tab.py # Single-model report viewer
|
| 38 |
-
β βββ data_viewer_side_by_side_tab.py # Side-by-side comparison
|
| 39 |
-
β βββ shared_data.py # Lazy data loading
|
| 40 |
-
βββ utils/
|
| 41 |
-
β βββ rank_leaderboard.py # Aggregate raw results -> leaderboard.csv
|
| 42 |
-
β βββ merge_raw_data.py # Merge raw data + scores -> data_viewer.jsonl
|
| 43 |
-
βββ data/
|
| 44 |
-
βββ leaderboard.csv # Aggregated leaderboard (generated)
|
| 45 |
-
βββ data_viewer.jsonl # Per-sample viewer data (generated)
|
| 46 |
-
βββ raw_data/ # Per-model prompts + articles
|
| 47 |
-
βββ raw_results/ # Per-model evaluation scores
|
| 48 |
-
```
|
| 49 |
-
|
| 50 |
-
## Data Format
|
| 51 |
-
|
| 52 |
-
### `data/leaderboard.csv`
|
| 53 |
-
|
| 54 |
-
| Column | Description |
|
| 55 |
-
|--------|-------------|
|
| 56 |
-
| `model` | Model / system name |
|
| 57 |
-
| `overall` | Weighted overall score across EN & ZH, Objective & Subjective |
|
| 58 |
-
| `objective_en` | Objective score on Global (English) tasks |
|
| 59 |
-
| `objective_zh` | Objective score on Chinese tasks |
|
| 60 |
-
| `objective_avg` | Average objective score across both language tracks |
|
| 61 |
-
| `subjective_en` | Subjective report score on Global (English) tasks |
|
| 62 |
-
| `subjective_zh` | Subjective report score on Chinese tasks |
|
| 63 |
-
| `subjective_avg` | Average subjective score across both language tracks |
|
| 64 |
-
| `citation_score` | Citation consistency score (0-100) |
|
| 65 |
-
| `source_quality` | Source authority & timeliness score (0-100) |
|
| 66 |
-
| `rmsce` | Root Mean Square Calibration Error for objective confidence (lower is better) |
|
| 67 |
-
|
| 68 |
-
### `data/raw_results/<model>/results.jsonl`
|
| 69 |
-
|
| 70 |
-
Each line should be a JSON object:
|
| 71 |
-
|
| 72 |
-
```json
|
| 73 |
-
{
|
| 74 |
-
"id": "1",
|
| 75 |
-
"language": "en",
|
| 76 |
-
"track": "objective",
|
| 77 |
-
"score": 0.85,
|
| 78 |
-
"correct": true,
|
| 79 |
-
"confidence": 0.90
|
| 80 |
-
}
|
| 81 |
-
```
|
| 82 |
-
|
| 83 |
-
or for subjective tasks:
|
| 84 |
-
|
| 85 |
-
```json
|
| 86 |
-
{
|
| 87 |
-
"id": "41",
|
| 88 |
-
"language": "zh",
|
| 89 |
-
"track": "subjective",
|
| 90 |
-
"score": 0.72,
|
| 91 |
-
"expert_score": 0.74,
|
| 92 |
-
"citation_score": 0.68,
|
| 93 |
-
"source_quality_score": 0.70
|
| 94 |
-
}
|
| 95 |
-
```
|
| 96 |
-
|
| 97 |
-
Scores can be either 0-1 ratios or 0-100 percentages.
|
| 98 |
-
|
| 99 |
-
## Local Development
|
| 100 |
-
|
| 101 |
-
```bash
|
| 102 |
-
pip install -r requirements.txt
|
| 103 |
-
python utils/rank_leaderboard.py # generate data/leaderboard.csv
|
| 104 |
-
python utils/merge_raw_data.py # generate data/data_viewer.jsonl
|
| 105 |
-
python app.py
|
| 106 |
-
```
|
| 107 |
-
|
| 108 |
-
> **Note**: The `data/raw_results/` directory in this repository currently contains legacy DeepResearch-Bench-format `race_result.txt` files. Before running `rank_leaderboard.py` for ICBCBench, replace these with ICBCBench `results.jsonl` files as described above. The checked-in `data/leaderboard.csv` is a sample ICBCBench leaderboard and will be overwritten if you rerun the script without updating `raw_results/`.
|
| 109 |
-
|
| 110 |
-
Then open <http://localhost:7860>.
|
| 111 |
-
|
| 112 |
## Citation
|
| 113 |
|
| 114 |
```bibtex
|
|
|
|
| 27 |
- π» Code: <https://github.com/ICBCBench/ICBCBench>
|
| 28 |
- π Dataset: <https://huggingface.co/datasets/DeepFin-Intelligence/ICBCBench>
|
| 29 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
## Citation
|
| 31 |
|
| 32 |
```bibtex
|