README.md

metadata

title: Evaluation Summarization
emoji: ⚡
colorFrom: gray
colorTo: yellow
sdk: docker
app_port: 7860
pinned: false
short_description: Evaluation Zusammenfassung Rundfunktranskripte

Evaluation Summarization Space

This repository contains a Docker-based Hugging Face Space for evaluating generated summaries against source transcripts. The app serves a lightweight annotation interface backed by Flask and stores submitted judgements in SQLite.

What the Space does

• Presents evaluation items from a JSONL dataset.
• Lets annotators score summary quality across multiple criteria.
• Persists annotations in a SQLite database.
• Exports collected annotations as JSONL.
• Supports optional password protection through a Space secret.

Runtime model

This Space uses the docker SDK and starts the Flask app defined in app.py. The container exposes port 7860, which is declared in the YAML front matter above and in Dockerfile.

At runtime, the app reads:

• DATASET_PATH for the evaluation dataset JSONL
• DB_PATH for the SQLite annotations database
• APP_PASSWORD for optional login protection
• SECRET_KEY for stable HMAC token generation

By default, the Docker image is configured for Hugging Face persistent storage mounted at /data:

• dataset: /data/2026-04-23_prompt_evaluation_dataset.jsonl
• annotations DB: /data/annotations.db

Deploying on Hugging Face Spaces

1. Create a new Space using the Docker SDK.
2. Add persistent storage mounted at /data.
3. Set the secret APP_PASSWORD if the UI should require a login.
4. Set the secret SECRET_KEY if you want authentication tokens to remain valid across container restarts.
5. Push this repository to the Space.
6. Upload the dataset JSONL into /data/ in the Space storage browser.
7. Confirm that the uploaded dataset filename matches DATASET_PATH in Dockerfile.

Without persistent storage, annotations stored in SQLite will be lost when the container filesystem is replaced.

Local development

The app can also run locally. In local development, it defaults to the dataset file in the project root when DATASET_PATH is unset.

Example environment variables:

$env:DATASET_PATH = "2026-04-23_prompt_evaluation_dataset.jsonl"
$env:DB_PATH = "annotations.db"
$env:APP_PASSWORD = "your-password"
python app.py

The Python dependency is defined in pyproject.toml. The container build uses uv as configured in Dockerfile.

Data expectations

The dataset is expected to be a JSONL file with one evaluation item per line. The application relies on stable item IDs and the text fields required to render the transcript, summary, and metadata shown in the UI.

Submitted annotations are stored by eval_id and include these fields:

• bewertung
• korrekt
• relevant
• vollstaendig
• kohaerenz
• anmerkungen

If you replace the dataset with a new file that uses different eval_id values, existing annotation rows in the SQLite database will no longer line up with the new items.

Notes for duplication

If someone duplicates this Space, the most important setup steps are:

1. mount persistent storage at /data
2. upload a dataset JSONL file to /data
3. keep DATASET_PATH aligned with the uploaded filename
4. configure APP_PASSWORD if access should be restricted

For Hugging Face Space metadata options, see the Spaces configuration reference:

https://huggingface.co/docs/hub/spaces-config-reference