| --- |
| title: Evaluation Summarization |
| emoji: ⚡ |
| colorFrom: gray |
| colorTo: yellow |
| sdk: docker |
| app_port: 7860 |
| pinned: false |
| short_description: Evaluation Zusammenfassung Rundfunktranskripte |
| --- |
| |
| # Evaluation Summarization Space |
|
|
| This repository contains a Docker-based Hugging Face Space for evaluating generated summaries against source transcripts. The app serves a lightweight annotation interface backed by Flask and stores submitted judgements in SQLite. |
|
|
| ## What the Space does |
|
|
| - Presents evaluation items from a JSONL dataset. |
| - Lets annotators score summary quality across multiple criteria. |
| - Persists annotations in a SQLite database. |
| - Exports collected annotations as JSONL. |
| - Supports optional password protection through a Space secret. |
|
|
| ## Runtime model |
|
|
| This Space uses the `docker` SDK and starts the Flask app defined in [app.py](app.py). The container exposes port `7860`, which is declared in the YAML front matter above and in [Dockerfile](Dockerfile). |
|
|
| At runtime, the app reads: |
|
|
| - `DATASET_PATH` for the evaluation dataset JSONL |
| - `DB_PATH` for the SQLite annotations database |
| - `APP_PASSWORD` for optional login protection |
| - `SECRET_KEY` for stable HMAC token generation |
|
|
| By default, the Docker image is configured for Hugging Face persistent storage mounted at `/data`: |
|
|
| - dataset: `/data/2026-04-23_prompt_evaluation_dataset.jsonl` |
| - annotations DB: `/data/annotations.db` |
|
|
| ## Deploying on Hugging Face Spaces |
|
|
| 1. Create a new Space using the `Docker` SDK. |
| 2. Add persistent storage mounted at `/data`. |
| 3. Set the secret `APP_PASSWORD` if the UI should require a login. |
| 4. Set the secret `SECRET_KEY` if you want authentication tokens to remain valid across container restarts. |
| 5. Push this repository to the Space. |
| 6. Upload the dataset JSONL into `/data/` in the Space storage browser. |
| 7. Confirm that the uploaded dataset filename matches `DATASET_PATH` in [Dockerfile](Dockerfile). |
|
|
| Without persistent storage, annotations stored in SQLite will be lost when the container filesystem is replaced. |
|
|
| ## Local development |
|
|
| The app can also run locally. In local development, it defaults to the dataset file in the project root when `DATASET_PATH` is unset. |
|
|
| Example environment variables: |
|
|
| ```powershell |
| $env:DATASET_PATH = "2026-04-23_prompt_evaluation_dataset.jsonl" |
| $env:DB_PATH = "annotations.db" |
| $env:APP_PASSWORD = "your-password" |
| python app.py |
| ``` |
|
|
| The Python dependency is defined in [pyproject.toml](pyproject.toml). The container build uses `uv` as configured in [Dockerfile](Dockerfile). |
|
|
| ## Data expectations |
|
|
| The dataset is expected to be a JSONL file with one evaluation item per line. The application relies on stable item IDs and the text fields required to render the transcript, summary, and metadata shown in the UI. |
|
|
| Submitted annotations are stored by `eval_id` and include these fields: |
|
|
| - `bewertung` |
| - `korrekt` |
| - `relevant` |
| - `vollstaendig` |
| - `kohaerenz` |
| - `anmerkungen` |
|
|
| If you replace the dataset with a new file that uses different `eval_id` values, existing annotation rows in the SQLite database will no longer line up with the new items. |
|
|
| ## Notes for duplication |
|
|
| If someone duplicates this Space, the most important setup steps are: |
|
|
| 1. mount persistent storage at `/data` |
| 2. upload a dataset JSONL file to `/data` |
| 3. keep `DATASET_PATH` aligned with the uploaded filename |
| 4. configure `APP_PASSWORD` if access should be restricted |
|
|
| For Hugging Face Space metadata options, see the Spaces configuration reference: |
|
|
| https://huggingface.co/docs/hub/spaces-config-reference |
|
|
