---
title: Distractor Annotation Tool
emoji: 🎯
colorFrom: purple
colorTo: indigo
sdk: streamlit
sdk_version: 1.36.0
app_file: app.py
pinned: false
---

# 🎯 Distractor Annotation Tool

Collaborative annotation GUI for the MSc NLP research project **"Keeping LLMs on Track in Task-Oriented Dialogue"**.

## One-time Setup

### 1. Create a private HF Dataset repo for shared annotations
Go to [huggingface.co/new-dataset](https://huggingface.co/new-dataset), make it **private**, and note the repo ID (e.g. `yourgroup/distractor-annotations`).

### 2. Set secrets in your HF Space
In your Space → Settings → Repository secrets, add:
| Secret | Value |
|---|---|
| `HF_TOKEN` | Your HF token with **write** access |
| `ANNOTATIONS_REPO_ID` | e.g. `yourgroup/distractor-annotations` |

### 3. Set secrets in your GitHub repo
In GitHub → Settings → Secrets and variables → Actions, add the same `HF_TOKEN`.

### 4. Update the sync workflow
In `.github/workflows/sync_to_hf.yml`, replace `YOUR_HF_USERNAME` and `YOUR_SPACE_NAME` with your actual values.

### 5. Import seed data
On first run, go to the **Dashboard** and click **Import Seed Data** to populate the shared repo with the group's initial entries.

## Workflow

| Page | Purpose |
|---|---|
| 🏠 Dashboard | Stats overview, seed import, config check |
| 📚 Browse | Explore the base nvidia dataset and seed entries |
| ✏️ Annotate | Create multi-turn distractor entries |
| 👥 Annotations | View, edit, review all group work |
| 💬 Test LLM | Send distractors to a live LLM, judge if it gets distracted |

## Annotation Schema

Each annotation follows the `nvidia/CantTalkAboutThis` schema, extended with:
- `distractors_multiturn`: rich multi-turn distractor sequences
- `_id`, `_annotator`, `_review_status`, `_created_at`, `_updated_at`
- `_llm_test_results`: logged results from the Test LLM page

## Base Dataset
[nvidia/CantTalkAboutThis-Topic-Control-Dataset](https://huggingface.co/datasets/nvidia/CantTalkAboutThis-Topic-Control-Dataset)

## Related Papers
- [2024.findings-emnlp.713](https://aclanthology.org/2024.findings-emnlp.713)
- [arXiv:2511.05018](https://arxiv.org/abs/2511.05018)