README.md

metadata

title: A11yn UI Annotation
emoji: 🎨
colorFrom: indigo
colorTo: blue
sdk: docker
app_port: 7860
pinned: false

A11yn UI Annotation Tool

A pairwise comparison annotation tool for evaluating generated UIs.

• Test 1: Base_UI vs A11yn
• Test 2: Base_UI vs feeda11y

For each of 300 pairs the annotator sees the original user request and the two generated UIs, then picks one of five preferences:

strong_left | weak_left | similar | weak_right | strong_right

Left/right assignment is randomized per pair (seeded by annotator+test+index, so the layout is stable when resuming).

Persistence

Annotations are written to results/annotations_test{1,2}.csv. If the Space has the environment variables HF_TOKEN (write) and HF_RESULTS_DATASET (e.g. your-username/a11yn-annotation-results) set as Secrets, every submit is also synced to that Dataset repo and re-pulled on startup, so data survives Space restarts.

You can also grab the current CSVs at any time:

• /results/1.csv
• /results/2.csv