singhalamaan116 commited on
Commit
4751408
·
verified ·
1 Parent(s): 8c35076

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +39 -8
README.md CHANGED
@@ -1,13 +1,44 @@
1
  ---
2
- title: EcoEval LLM
3
- emoji: 🏆
4
- colorFrom: pink
5
- colorTo: green
6
  sdk: gradio
7
- sdk_version: 6.0.1
8
  app_file: app.py
9
- pinned: false
10
- short_description: 'Framework that benchmarks models for energy usage '
11
  ---
12
 
13
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ title: EcoEval-LLM
3
+ emoji: 🌱
4
+ colorFrom: green
5
+ colorTo: blue
6
  sdk: gradio
7
+ sdk_version: 4.0.0
8
  app_file: app.py
9
+ pinned: true
 
10
  ---
11
 
12
+ # 🌱 EcoEval-LLM: Energy & Carbon Benchmarking for LLM Code Generation
13
+
14
+ **EcoEval-LLM** benchmarks code generation models on:
15
+
16
+ - ✅ Task correctness (unit-test based pass rate)
17
+ - ⏱ Runtime
18
+ - ⚡ Energy consumption (kWh)
19
+ - 🌍 CO₂ emissions (kg) via [CodeCarbon](https://github.com/mlco2/codecarbon)
20
+
21
+ It runs a small benchmark of Python programming tasks, executes the generated code against unit tests, and measures the environmental footprint of the run.
22
+
23
+ ## How it works
24
+
25
+ 1. You choose:
26
+ - A Hugging Face Hub model ID (e.g. `Salesforce/codegen-350M-multi`)
27
+ - A built-in Python benchmark dataset
28
+ 2. The app:
29
+ - Loads the model and tokenizer via `transformers`
30
+ - Generates code for each task
31
+ - Executes unit tests to check correctness
32
+ - Wraps the whole process in a `CodeCarbon.EmissionsTracker` to measure energy and CO₂
33
+ 3. Results:
34
+ - Run-level summary (accuracy, runtime, energy, CO₂, energy per task, CO₂ per passed task)
35
+ - Per-task pass/fail and runtime
36
+ - Persistent leaderboard (`runs.csv`) across Space sessions
37
+
38
+ ## Run locally
39
+
40
+ ```bash
41
+ git clone <this-repo-url>
42
+ cd EcoEval-LLM
43
+ pip install -r requirements.txt
44
+ python app.py