gemma/diffusion/hackable_diffusion_adapter/data/pubmedqa/README.md

PubMedQA Onboarding and Evaluation on HD DiffusionGemma

This folder contains the dataset ingestion and preprocessing for onboarding the PubMedQA dataset.

PubMedQA is a medical question-answering dataset where the task is to read a medical research abstract context and a question, and then:

1. Output a short categorical answer: "yes", "no", or "maybe".
2. Generate a detailed paragraph-length explanation (LONG_ANSWER).

Our dataset tasks the model with generating both the categorical answer and the detailed explanation.

Directory Structure

• prepare_pubmedqa_dataset.sh: Clones the dataset from the GitHub repo and calls the preprocessing script.
• convert_pubmedqa.py: Script that formats the raw datasets with instruction-tuning turn tokens, and outputs them as line-delimited JSONL files.
• pubmedqa_data.py: Implements JsonlDataSource (an in-memory JSONL loader compatible with the Grain RandomAccessDataSource Protocol) with slicing support to split training data.

Expected Output

• Train split: ori_pqal.json (1,000 expert-labeled examples) minus test_set.json (500 test IDs) → pubmedqa_train.jsonl (500 clean training examples).
• Test split: test_set.json (500 official test examples) → pubmedqa_test.jsonl.