Spaces:
Running
Running
Create README.md
Browse files
README.md
CHANGED
|
@@ -1,13 +1,44 @@
|
|
| 1 |
---
|
| 2 |
-
title: EcoEval
|
| 3 |
-
emoji:
|
| 4 |
-
colorFrom:
|
| 5 |
-
colorTo:
|
| 6 |
sdk: gradio
|
| 7 |
-
sdk_version:
|
| 8 |
app_file: app.py
|
| 9 |
-
pinned:
|
| 10 |
-
short_description: 'Framework that benchmarks models for energy usage '
|
| 11 |
---
|
| 12 |
|
| 13 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
title: EcoEval-LLM
|
| 3 |
+
emoji: 🌱
|
| 4 |
+
colorFrom: green
|
| 5 |
+
colorTo: blue
|
| 6 |
sdk: gradio
|
| 7 |
+
sdk_version: 4.0.0
|
| 8 |
app_file: app.py
|
| 9 |
+
pinned: true
|
|
|
|
| 10 |
---
|
| 11 |
|
| 12 |
+
# 🌱 EcoEval-LLM: Energy & Carbon Benchmarking for LLM Code Generation
|
| 13 |
+
|
| 14 |
+
**EcoEval-LLM** benchmarks code generation models on:
|
| 15 |
+
|
| 16 |
+
- ✅ Task correctness (unit-test based pass rate)
|
| 17 |
+
- ⏱ Runtime
|
| 18 |
+
- ⚡ Energy consumption (kWh)
|
| 19 |
+
- 🌍 CO₂ emissions (kg) via [CodeCarbon](https://github.com/mlco2/codecarbon)
|
| 20 |
+
|
| 21 |
+
It runs a small benchmark of Python programming tasks, executes the generated code against unit tests, and measures the environmental footprint of the run.
|
| 22 |
+
|
| 23 |
+
## How it works
|
| 24 |
+
|
| 25 |
+
1. You choose:
|
| 26 |
+
- A Hugging Face Hub model ID (e.g. `Salesforce/codegen-350M-multi`)
|
| 27 |
+
- A built-in Python benchmark dataset
|
| 28 |
+
2. The app:
|
| 29 |
+
- Loads the model and tokenizer via `transformers`
|
| 30 |
+
- Generates code for each task
|
| 31 |
+
- Executes unit tests to check correctness
|
| 32 |
+
- Wraps the whole process in a `CodeCarbon.EmissionsTracker` to measure energy and CO₂
|
| 33 |
+
3. Results:
|
| 34 |
+
- Run-level summary (accuracy, runtime, energy, CO₂, energy per task, CO₂ per passed task)
|
| 35 |
+
- Per-task pass/fail and runtime
|
| 36 |
+
- Persistent leaderboard (`runs.csv`) across Space sessions
|
| 37 |
+
|
| 38 |
+
## Run locally
|
| 39 |
+
|
| 40 |
+
```bash
|
| 41 |
+
git clone <this-repo-url>
|
| 42 |
+
cd EcoEval-LLM
|
| 43 |
+
pip install -r requirements.txt
|
| 44 |
+
python app.py
|