Spaces:
Running
Running
File size: 2,703 Bytes
47ccf08 68ef960 c563cc0 68ef960 c563cc0 68ef960 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 | ---
title: README
emoji: 💻
colorFrom: pink
colorTo: blue
sdk: static
pinned: false
---
# FPEval: A Holistic Evaluation Framework for Functional Programming
Built upon **FPBench**, the framework includes **721 programming tasks** across three difficulty levels. It focuses on three mainstream FP languages (**Haskell, OCaml, and Scala**) and utilizes **Java** as an imperative baseline for comparative analysis.
---
## 📊 Dataset Structure
The dataset is organized into two primary components:
### 1. `LeetCodeProblem/`
This directory contains the core problem specifications and testing infrastructure.
* **Data Collection Period:** 2021 – March 2025.
* **`meta.json`**: A comprehensive file for each problem containing:
* Problem descriptions and I/O constraints.
* Public test cases.
* **Private Test Cases**: High-coverage inputs/outputs synthetically generated using **GPT-4o**.
* **Test Templates**: Pre-configured templates to ensure seamless evaluation:
* `Main.hs` (Haskell)
* `main.ml` (OCaml)
* `MySuite.scala` (Scala)
### 2. `LLMsGeneratedCode/`
This directory stores outputs from **GPT-3.5-turbo**, **GPT-4o**, and **GPT-5**, categorized into three distinct refinement stages:
| Stage | Description |
| :--- | :--- |
| **`CodeGenerated`** | Initial zero-shot code outputs. |
| **`BaselineRepair`** | Code refined based on old code. |
| **`InstructionRepair`** | Code optimized using static analysis feedback and hand-crafted idiomatic instructions. |
---
## 🛠 Static Analysis & Repair Tools
FPEval utilizes industry-standard tools to provide feedback during the `InstructionRepair` stage, ensuring the generated code adheres to idiomatic patterns:
* **Haskell**: `HLint` (style/idioms) and `GHC` warnings (correctness).
* **OCaml**: `dune` (build validation) and `ocamlformat` (formatting).
* **Scala**: `Scalastyle` (functional style enforcement).
* **Java**: `Checkstyle` and `PMD` (standard imperative quality checks).
---
## 📂 Directory Layout
```text
.
├── LeetCodeProblem/
│ ├── [Problem_NAME]/
│ │ ├── meta.json # Core problem data & private tests
│ │ ├── Main.hs # Haskell test template
│ │ ├── main.ml # OCaml test template
│ │ └── MySuite.scala # Scala test template
├── LLMsGeneratedCode/
│ ├── [Model_Name]/ # gpt-3.5-turbo, gpt-4o, gpt-5
│ │ ├── CodeGenerated/ # Initial raw outputs
│ │ ├── BaselineRepair/ # Post-compiler feedback attempts
│ │ └── InstructionRepair/ # Post-static analysis refinement
└── README.md
|