{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3c111aa7",
   "metadata": {},
   "source": [
    "# Build TEST-100 — disease + VQA held-out test set (Colab)\n",
    "\n",
    "Tải **100 ảnh test có bệnh + có VQA** từ PhysioNet rồi đóng gói + push lên\n",
    "`hieu3636/cxr-vlm-data/MIMIC-CXR_test100/` để dùng cho inference / evaluate.\n",
    "\n",
    "**100 studies đã được lọc sẵn ở local** (`data/test100_manifest.json`, kèm trong repo\n",
    "`hieu3636/cxr-vlm-code`):\n",
    "- Lấy từ **test split** của `MIMIC-CXR_resized` → model **chưa thấy** khi train.\n",
    "- Mỗi study: ít nhất 1 pathology dương tính (trừ *No Finding*/*Support Devices*),\n",
    "  có **VQA**, có **FINDINGS + IMPRESSION** ground-truth, kèm **PNU** (chex 3-lớp).\n",
    "- 100 studies · 260 câu VQA (137 yes/no · 123 open) · phủ 12 pathology.\n",
    "\n",
    "**Notebook chỉ làm 2 việc:** (1) tải 100 ảnh từ `jpg_url` (PhysioNet, wget),\n",
    "(2) đóng gói + push HF. Toàn bộ logic lọc/parse report/PNU đã làm sẵn ở local.\n",
    "\n",
    "**Output trên HF** (`MIMIC-CXR_test100/`), drop-in như mini `MIMIC-CXR_resized`:\n",
    "```\n",
    "files/pXX/pSUBJ/sSTUDY/<dicom>.jpg     ← 100 ảnh (giữ path gốc PhysioNet)\n",
    "manifest_test.csv                       ← 100 rows, cột resized chuẩn (chex_*, image_relpath…)\n",
    "vqa/vqa_test.json                       ← 260 câu VQA của 100 studies\n",
    "test100.json                            ← GT gom theo study (findings/impression/vqa/PNU) — tiện inference\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "752849e7",
   "metadata": {},
   "source": [
    "## 0. Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "591eb597",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys, os\n",
    "IN_COLAB = \"google.colab\" in sys.modules\n",
    "if IN_COLAB:\n",
    "    !pip -q install huggingface_hub tqdm\n",
    "print(\"IN_COLAB =\", IN_COLAB)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d57c0f86",
   "metadata": {},
   "source": [
    "## 1. Config + credentials\n",
    "\n",
    "Creds qua **Colab Secrets** (icon 🔑 cột trái) — tạo `PHYSIONET_USER`, `PHYSIONET_PASS`,\n",
    "`HF_TOKEN` (bật *Notebook access*). Hoặc gõ thẳng vào `_HARDCODE_*` (đừng commit)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca1b73e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import os, getpass, json, time, shutil, csv, subprocess, threading\n",
    "from concurrent.futures import ThreadPoolExecutor, as_completed\n",
    "from collections import Counter\n",
    "\n",
    "# ── HF source/target ────────────────────────────────────────────────────────\n",
    "HF_CODE_REPO = \"hieu3636/cxr-vlm-code\"      # chứa data/test100_manifest.json\n",
    "HF_DATA_REPO = \"hieu3636/cxr-vlm-data\"      # đích upload (dataset)\n",
    "HF_PATH      = \"MIMIC-CXR_test100\"          # thư mục con trong data repo\n",
    "\n",
    "WORK = Path(\"/content/test100\") if IN_COLAB else Path(\"./test100\")\n",
    "WORK.mkdir(parents=True, exist_ok=True)\n",
    "\n",
    "# ── credentials ─────────────────────────────────────────────────────────────\n",
    "_HARDCODE_USER  = \"\"\n",
    "_HARDCODE_PASS  = \"\"\n",
    "_HARDCODE_HFTOK = \"\"\n",
    "def _get(name, hard):\n",
    "    if hard: return hard\n",
    "    try:\n",
    "        from google.colab import userdata\n",
    "        v = userdata.get(name)\n",
    "        if v: return v\n",
    "    except Exception: pass\n",
    "    return os.environ.get(name)\n",
    "PHYSIONET_USER = _get(\"PHYSIONET_USER\", _HARDCODE_USER) or input(\"PhysioNet username: \")\n",
    "PHYSIONET_PASS = _get(\"PHYSIONET_PASS\", _HARDCODE_PASS) or getpass.getpass(\"PhysioNet password: \")\n",
    "HF_TOKEN       = _get(\"HF_TOKEN\",       _HARDCODE_HFTOK) or getpass.getpass(\"HF write token: \")\n",
    "print(\"creds OK | work dir:\", WORK)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98ed8d7f",
   "metadata": {},
   "source": [
    "## 2. Lấy danh sách 100 studies đã lọc sẵn\n",
    "\n",
    "`data/test100_manifest.json` nằm trong repo code. Nếu chạy local trong repo thì đọc\n",
    "thẳng file; trên Colab thì tải từ HF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c1d882f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from huggingface_hub import hf_hub_download\n",
    "\n",
    "local_manifest = Path(\"data/test100_manifest.json\")\n",
    "if local_manifest.is_file():\n",
    "    MANIFEST_PATH = local_manifest\n",
    "    print(\"dùng manifest local:\", MANIFEST_PATH)\n",
    "else:\n",
    "    MANIFEST_PATH = Path(hf_hub_download(\n",
    "        repo_id=HF_CODE_REPO, repo_type=\"model\",\n",
    "        filename=\"data/test100_manifest.json\", token=HF_TOKEN))\n",
    "    print(\"tải manifest từ HF:\", MANIFEST_PATH)\n",
    "\n",
    "studies = json.load(open(MANIFEST_PATH, encoding=\"utf-8\"))\n",
    "print(f\"studies: {len(studies)} | vqa: {sum(len(s['vqa']) for s in studies)}\")\n",
    "cov = Counter(l for s in studies for l in s[\"positive_labels\"])\n",
    "print(\"pathology coverage:\", dict(cov.most_common()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a171ac4",
   "metadata": {},
   "source": [
    "## 3. Tải 100 ảnh từ PhysioNet (wget, resume-safe)\n",
    "\n",
    "PhysioNet từ chối `requests` basic-auth nhưng OK với `wget --user/--password`.\n",
    "12 luồng; bỏ qua ảnh đã có."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "723abc9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def dl(s):\n",
    "    rel = s[\"image_relpath\"]                      # files/pXX/.../<dicom>.jpg\n",
    "    out = WORK / rel\n",
    "    if out.exists() and out.stat().st_size > 10_000:\n",
    "        return \"skip\"\n",
    "    out.parent.mkdir(parents=True, exist_ok=True)\n",
    "    tmp = out.with_suffix(\".part\")\n",
    "    cmd = [\"wget\", \"-q\", \"-T\", \"60\", \"-t\", \"3\", \"-O\", str(tmp),\n",
    "           \"--user\", PHYSIONET_USER, \"--password\", PHYSIONET_PASS, s[\"jpg_url\"]]\n",
    "    rc = subprocess.run(cmd).returncode\n",
    "    if rc == 0 and tmp.exists() and tmp.stat().st_size > 10_000:\n",
    "        tmp.replace(out); return \"ok\"\n",
    "    if tmp.exists(): tmp.unlink()\n",
    "    return f\"fail(rc={rc})\"\n",
    "\n",
    "res = Counter()\n",
    "with ThreadPoolExecutor(max_workers=12) as ex:\n",
    "    futs = {ex.submit(dl, s): s for s in studies}\n",
    "    for f in as_completed(futs):\n",
    "        res[f.result().split(\"(\")[0]] += 1\n",
    "print(dict(res))\n",
    "missing = [s for s in studies if not (WORK / s[\"image_relpath\"]).exists()]\n",
    "print(f\"ảnh thiếu: {len(missing)}\", [s['study_name'] for s in missing][:10])\n",
    "assert not missing, \"Còn ảnh thiếu — chạy lại cell này (chỉ tải phần thiếu).\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61eccca9",
   "metadata": {},
   "source": [
    "## 4. Đóng gói: manifest_test.csv + vqa_test.json + test100.json\n",
    "\n",
    "Giữ cấu trúc giống `MIMIC-CXR_resized` để dùng lại được builder/evaluate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84ac0476",
   "metadata": {},
   "outputs": [],
   "source": [
    "LABELS = [\"Atelectasis\",\"Cardiomegaly\",\"Consolidation\",\"Edema\",\"Enlarged Cardiomediastinum\",\n",
    "  \"Fracture\",\"Lung Lesion\",\"Lung Opacity\",\"No Finding\",\"Pleural Effusion\",\"Pleural Other\",\n",
    "  \"Pneumonia\",\"Pneumothorax\",\"Support Devices\"]\n",
    "\n",
    "# ── (a) manifest_test.csv (cột resized chuẩn) ───────────────────────────────\n",
    "man_cols = ([\"study_name\",\"split\",\"subject_id\",\"study_id\",\"subset\",\"dicom_id\",\n",
    "             \"image_filename\",\"view\",\"image_relpath\",\"report_relpath\",\"jpg_url\",\"has_vqa\"]\n",
    "            + [f\"chex_{l}\" for l in LABELS])\n",
    "with open(WORK/\"manifest_test.csv\",\"w\",newline=\"\",encoding=\"utf-8\") as f:\n",
    "    w = csv.DictWriter(f, fieldnames=man_cols); w.writeheader()\n",
    "    for s in studies:\n",
    "        row = {\"study_name\":s[\"study_name\"],\"split\":\"test\",\"subject_id\":s[\"subject_id\"],\n",
    "               \"study_id\":s[\"study_id\"],\"subset\":s[\"subset\"],\"dicom_id\":s[\"dicom_id\"],\n",
    "               \"image_filename\":f\"{s['dicom_id']}.jpg\",\"view\":s[\"view\"],\n",
    "               \"image_relpath\":s[\"image_relpath\"],\"report_relpath\":s[\"report_relpath\"],\n",
    "               \"jpg_url\":s[\"jpg_url\"],\"has_vqa\":True}\n",
    "        row.update({f\"chex_{l}\": s[\"chex\"].get(l,\"\") for l in LABELS})\n",
    "        w.writerow(row)\n",
    "\n",
    "# ── (b) vqa/vqa_test.json (260 câu, image_path = image_relpath) ──────────────\n",
    "(WORK/\"vqa\").mkdir(exist_ok=True)\n",
    "vqa_rows = []\n",
    "for s in studies:\n",
    "    for q in s[\"vqa\"]:\n",
    "        vqa_rows.append({\"study_name\":s[\"study_name\"],\"image_path\":s[\"image_relpath\"],\n",
    "            \"question\":q[\"question\"],\"answer\":[a.strip() for a in q[\"answer\"].split(\",\")],\n",
    "            \"semantic_type\":q.get(\"semantic_type\"),\"content_type\":q.get(\"content_type\"),\n",
    "            \"subject_id\":s[\"subject_id\"],\"study_id\":s[\"study_id\"],\"image_id\":q.get(\"image_id\")})\n",
    "json.dump(vqa_rows, open(WORK/\"vqa\"/\"vqa_test.json\",\"w\",encoding=\"utf-8\"), indent=1)\n",
    "\n",
    "# ── (c) test100.json — GT gom theo study (tiện inference/so sánh) ────────────\n",
    "preview = [{\"study_name\":s[\"study_name\"],\"study_id\":s[\"study_id\"],\n",
    "            \"image_path\":s[\"image_relpath\"],\"positive_labels\":s[\"positive_labels\"],\n",
    "            \"structured_findings\":s[\"structured_findings\"],\n",
    "            \"findings\":s[\"findings\"],\"impression\":s[\"impression\"],\n",
    "            \"vqa\":[{\"question\":q[\"question\"],\"answer\":q[\"answer\"]} for q in s[\"vqa\"]]}\n",
    "           for s in studies]\n",
    "json.dump(preview, open(WORK/\"test100.json\",\"w\",encoding=\"utf-8\"), ensure_ascii=False, indent=1)\n",
    "\n",
    "# ── reports: ghi findings/impression đã parse ra files/.../sSTUDY.txt ────────\n",
    "# (đủ cho evaluate; nội dung gốc full report không cần cho test)\n",
    "for s in studies:\n",
    "    rp = WORK / s[\"report_relpath\"]\n",
    "    rp.parent.mkdir(parents=True, exist_ok=True)\n",
    "    rp.write_text(f\"FINDINGS: {s['findings']}\\n\\nIMPRESSION: {s['impression']}\\n\",\n",
    "                  encoding=\"utf-8\")\n",
    "\n",
    "print(\"đóng gói xong:\")\n",
    "print(\"  manifest_test.csv :\", len(studies), \"rows\")\n",
    "print(\"  vqa_test.json     :\", len(vqa_rows), \"câu\")\n",
    "print(\"  test100.json      :\", len(preview), \"studies\")\n",
    "print(\"  ảnh + report       :\", sum(1 for s in studies if (WORK/s['image_relpath']).exists()), \"ảnh\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5d43628",
   "metadata": {},
   "source": [
    "### Xem thử 1 study (GT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37cf8bef",
   "metadata": {},
   "outputs": [],
   "source": [
    "import textwrap\n",
    "s = preview[0]\n",
    "print(\"STUDY:\", s[\"study_name\"], \"| labels:\", s[\"positive_labels\"])\n",
    "print(\"\\n[FINDINGS]\\n\", textwrap.fill(s[\"findings\"], 100))\n",
    "print(\"\\n[IMPRESSION]\\n\", textwrap.fill(s[\"impression\"], 100))\n",
    "print(\"\\n[PNU]\\n\", s[\"structured_findings\"])\n",
    "print(\"\\n[VQA]\")\n",
    "for q in s[\"vqa\"][:5]:\n",
    "    print(f\"  Q: {q['question']}\\n  A: {q['answer']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "772c37e2",
   "metadata": {},
   "source": [
    "## 5. Push lên Hugging Face\n",
    "\n",
    "`RUN_HF=True` để upload `MIMIC-CXR_test100/` (100 ảnh + report + manifest + vqa + test100.json).\n",
    "100 file lẻ là nhẹ → upload thẳng cả thư mục."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d6b5800",
   "metadata": {},
   "outputs": [],
   "source": [
    "RUN_HF = False     # ← bật True khi sẵn sàng\n",
    "\n",
    "if RUN_HF:\n",
    "    from huggingface_hub import HfApi\n",
    "    api = HfApi(token=HF_TOKEN)\n",
    "    api.create_repo(HF_DATA_REPO, repo_type=\"dataset\", exist_ok=True)\n",
    "    api.upload_folder(\n",
    "        folder_path=str(WORK),\n",
    "        path_in_repo=HF_PATH,\n",
    "        repo_id=HF_DATA_REPO, repo_type=\"dataset\",\n",
    "        commit_message=\"add MIMIC-CXR_test100 (100 disease+VQA held-out studies)\",\n",
    "        ignore_patterns=[\"*.part\"],\n",
    "    )\n",
    "    print(\"done →\",\n",
    "          f\"https://huggingface.co/datasets/{HF_DATA_REPO}/tree/main/{HF_PATH}\")\n",
    "else:\n",
    "    print(\"RUN_HF=False — bật True để push.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdc17517",
   "metadata": {},
   "source": [
    "## 6. Summary + cách dùng\n",
    "\n",
    "**Inference nhanh** (mỗi study chạy findings/impression/vqa, so với `test100.json`):\n",
    "```python\n",
    "import json\n",
    "from huggingface_hub import snapshot_download\n",
    "d = snapshot_download(\"hieu3636/cxr-vlm-data\", repo_type=\"dataset\",\n",
    "                      allow_patterns=\"MIMIC-CXR_test100/*\")\n",
    "test = json.load(open(f\"{d}/MIMIC-CXR_test100/test100.json\"))\n",
    "for s in test:\n",
    "    img = f\"{d}/MIMIC-CXR_test100/{s['image_path']}\"\n",
    "    # pred_f = model.generate(img, task=\"findings\"); so với s[\"findings\"]\n",
    "    # pred_i = model.generate(img, task=\"impression\"); so với s[\"impression\"]\n",
    "    # for q in s[\"vqa\"]: model.answer(img, q[\"question\"]) vs q[\"answer\"]\n",
    "```\n",
    "\n",
    "**Evaluate pipeline** (drop-in như `MIMIC-CXR_resized`, chỉ có test split):\n",
    "```bash\n",
    "python -m data.mimic_cxr_resized_builder \\\n",
    "    --root <dir>/MIMIC-CXR_test100 --vqa_dir <dir>/MIMIC-CXR_test100/vqa \\\n",
    "    --output test100_instruct.json --report_mode split --image_mode all_views_split\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "299aea58",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"=\"*54); print(\"  BUILD TEST-100 DONE\"); print(\"=\"*54)\n",
    "print(f\"  studies : {len(studies)}\")\n",
    "print(f\"  vqa     : {sum(len(s['vqa']) for s in studies)}\")\n",
    "print(f\"  package : {WORK}\")\n",
    "print(f\"  HF      : {HF_DATA_REPO}/{HF_PATH}  (RUN_HF={('done' if False else 'set True to push')})\")\n",
    "print(\"=\"*54)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}