README.md

metadata

title: README
emoji: 💻
colorFrom: pink
colorTo: blue
sdk: static
pinned: false

FPEval: A Holistic Evaluation Framework for Functional Programming

Built upon FPBench, the framework includes 721 programming tasks across three difficulty levels. It focuses on three mainstream FP languages (Haskell, OCaml, and Scala) and utilizes Java as an imperative baseline for comparative analysis.

---

📊 Dataset Structure

The dataset is organized into two primary components:

1. LeetCodeProblem/

This directory contains the core problem specifications and testing infrastructure.

• Data Collection Period: 2021 – March 2025.
• meta.json: A comprehensive file for each problem containing:
  • Problem descriptions and I/O constraints.
  • Public test cases.
  • Private Test Cases: High-coverage inputs/outputs synthetically generated using GPT-4o.
• Test Templates: Pre-configured templates to ensure seamless evaluation:
  • Main.hs (Haskell)
  • main.ml (OCaml)
  • MySuite.scala (Scala)

2. LLMsGeneratedCode/

This directory stores outputs from GPT-3.5-turbo, GPT-4o, and GPT-5, categorized into three distinct refinement stages:

Stage	Description
CodeGenerated	Initial zero-shot code outputs.
BaselineRepair	Code refined based on old code.
InstructionRepair	Code optimized using static analysis feedback and hand-crafted idiomatic instructions.

---

🛠 Static Analysis & Repair Tools

FPEval utilizes industry-standard tools to provide feedback during the InstructionRepair stage, ensuring the generated code adheres to idiomatic patterns:

• Haskell: HLint (style/idioms) and GHC warnings (correctness).
• OCaml: dune (build validation) and ocamlformat (formatting).
• Scala: Scalastyle (functional style enforcement).
• Java: Checkstyle and PMD (standard imperative quality checks).

---

📂 Directory Layout

.
├── LeetCodeProblem/
│   ├── [Problem_NAME]/
│   │   ├── meta.json             # Core problem data & private tests
│   │   ├── Main.hs               # Haskell test template
│   │   ├── main.ml               # OCaml test template
│   │   └── MySuite.scala         # Scala test template
├── LLMsGeneratedCode/
│   ├── [Model_Name]/             # gpt-3.5-turbo, gpt-4o, gpt-5
│   │   ├── CodeGenerated/        # Initial raw outputs
│   │   ├── BaselineRepair/       # Post-compiler feedback attempts
│   │   └── InstructionRepair/    # Post-static analysis refinement
└── README.md