Spaces:
Running
Running
| title: README | |
| emoji: 💻 | |
| colorFrom: pink | |
| colorTo: blue | |
| sdk: static | |
| pinned: false | |
| # FPEval: A Holistic Evaluation Framework for Functional Programming | |
| Built upon **FPBench**, the framework includes **721 programming tasks** across three difficulty levels. It focuses on three mainstream FP languages (**Haskell, OCaml, and Scala**) and utilizes **Java** as an imperative baseline for comparative analysis. | |
| --- | |
| ## 📊 Dataset Structure | |
| The dataset is organized into two primary components: | |
| ### 1. `LeetCodeProblem/` | |
| This directory contains the core problem specifications and testing infrastructure. | |
| * **Data Collection Period:** 2021 – March 2025. | |
| * **`meta.json`**: A comprehensive file for each problem containing: | |
| * Problem descriptions and I/O constraints. | |
| * Public test cases. | |
| * **Private Test Cases**: High-coverage inputs/outputs synthetically generated using **GPT-4o**. | |
| * **Test Templates**: Pre-configured templates to ensure seamless evaluation: | |
| * `Main.hs` (Haskell) | |
| * `main.ml` (OCaml) | |
| * `MySuite.scala` (Scala) | |
| ### 2. `LLMsGeneratedCode/` | |
| This directory stores outputs from **GPT-3.5-turbo**, **GPT-4o**, and **GPT-5**, categorized into three distinct refinement stages: | |
| | Stage | Description | | |
| | :--- | :--- | | |
| | **`CodeGenerated`** | Initial zero-shot code outputs. | | |
| | **`BaselineRepair`** | Code refined based on old code. | | |
| | **`InstructionRepair`** | Code optimized using static analysis feedback and hand-crafted idiomatic instructions. | | |
| --- | |
| ## 🛠 Static Analysis & Repair Tools | |
| FPEval utilizes industry-standard tools to provide feedback during the `InstructionRepair` stage, ensuring the generated code adheres to idiomatic patterns: | |
| * **Haskell**: `HLint` (style/idioms) and `GHC` warnings (correctness). | |
| * **OCaml**: `dune` (build validation) and `ocamlformat` (formatting). | |
| * **Scala**: `Scalastyle` (functional style enforcement). | |
| * **Java**: `Checkstyle` and `PMD` (standard imperative quality checks). | |
| --- | |
| ## 📂 Directory Layout | |
| ```text | |
| . | |
| ├── LeetCodeProblem/ | |
| │ ├── [Problem_NAME]/ | |
| │ │ ├── meta.json # Core problem data & private tests | |
| │ │ ├── Main.hs # Haskell test template | |
| │ │ ├── main.ml # OCaml test template | |
| │ │ └── MySuite.scala # Scala test template | |
| ├── LLMsGeneratedCode/ | |
| │ ├── [Model_Name]/ # gpt-3.5-turbo, gpt-4o, gpt-5 | |
| │ │ ├── CodeGenerated/ # Initial raw outputs | |
| │ │ ├── BaselineRepair/ # Post-compiler feedback attempts | |
| │ │ └── InstructionRepair/ # Post-static analysis refinement | |
| └── README.md | |