f

configs/train_config.yaml

@@ -238,7 +238,7 @@ stage1:
   # handled entirely by Stage 2. When disabled (default), Stage 1 trains
   # the projection with the causal-LM loss through Vicuna (original path).
   itc:
-    enabled: false
+    enabled: true
     # Path to the {study_id: tensor[proj_dim]} cache produced by
     # scripts/precompute_cxrbert_embeddings.ipynb. Required when enabled.
     text_embed_cache: null

data/eda_reports.ipynb

@@ -1030,6 +1030,54 @@
     "    print(pct_data.to_string())"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "f77592d6",
+   "metadata": {},
+   "source": "## 6b. Ví dụ báo cáo bất thường — outlier dài, quá ngắn, section lạ"
+  },
+  {
+   "cell_type": "markdown",
+   "id": "997c36b4",
+   "metadata": {},
+   "source": "### 1. Outlier dài — vượt Q3 + 1.5·IQR\n\nNgưỡng theo IQR fence (tính từ toàn bộ dataset):\n- **Findings** : Q3 = 61, IQR = 28 → upper fence = **103 từ**\n- **Impression**: Q3 = 40, IQR = 34 → upper fence = **91 từ**"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d3fa9af1",
+   "metadata": {},
+   "outputs": [],
+   "source": "q1_f, q3_f = df[\"findings_len\"].quantile(0.25), df[\"findings_len\"].quantile(0.75)\nq1_i, q3_i = df[\"impression_len\"].quantile(0.25), df[\"impression_len\"].quantile(0.75)\nthresh_f = q3_f + 1.5 * (q3_f - q1_f)\nthresh_i = q3_i + 1.5 * (q3_i - q1_i)\nprint(f\"Outlier fence - Findings   > {thresh_f:.0f} tu  | count: {(df['findings_len']  > thresh_f).sum():,}\")\nprint(f\"Outlier fence - Impression > {thresh_i:.0f} tu  | count: {(df['impression_len'] > thresh_i).sum():,}\")\n\n\ndef report_path(row):\n    return f\"files/{row['subset']}/p{row['subject_id']}/s{row['study_id']}.txt\"\n\n\nN = 10  # so vi du muon lay moi loai\n\nprint(f\"\\n=== Findings outlier (top {N} dai nhat) ===\")\nfor _, row in df[df[\"findings_len\"] > thresh_f].nlargest(N, \"findings_len\").iterrows():\n    print(f\"  {int(row['findings_len']):>4} tu   {report_path(row)}\")\n\nprint(f\"\\n=== Impression outlier (top {N} dai nhat) ===\")\nfor _, row in df[df[\"impression_len\"] > thresh_i].nlargest(N, \"impression_len\").iterrows():\n    print(f\"  {int(row['impression_len']):>4} tu   {report_path(row)}\")"
+  },
+  {
+   "cell_type": "markdown",
+   "id": "883f744d",
+   "metadata": {},
+   "source": "### 2. Báo cáo có findings < 3 từ hoặc impression < 2 từ\n\nNhững report ngắn bất thường — thường là lỗi parse hoặc nội dung quá sơ lược,\nkhông đủ thông tin để huấn luyện mô hình."
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6f5ba746",
+   "metadata": {},
+   "outputs": [],
+   "source": "short_f = df[df[\"findings_len\"] < 3].copy()\nshort_i = df[df[\"impression_len\"] < 2].copy()\nprint(f\"Findings  < 3 tu : {len(short_f):,}  ({len(short_f)/total*100:.2f}%)\")\nprint(f\"Impression < 2 tu: {len(short_i):,}  ({len(short_i)/total*100:.2f}%)\")\n\nN = 10  # so vi du sample moi loai\n\nprint(f\"\\n=== Findings < 3 tu (sample {N}) ===\")\nfor _, row in short_f.sample(min(N, len(short_f)), random_state=42).iterrows():\n    print(f\"  findings={int(row['findings_len'])} tu   {report_path(row)}\")\n\nprint(f\"\\n=== Impression < 2 tu (sample {N}) ===\")\nfor _, row in short_i.sample(min(N, len(short_i)), random_state=42).iterrows():\n    print(f\"  impression={int(row['impression_len'])} tu   {report_path(row)}\")"
+  },
+  {
+   "cell_type": "markdown",
+   "id": "53a3ff19",
+   "metadata": {},
+   "source": "### 3. Báo cáo có section lạ — không có FINDINGS và IMPRESSION\n\nNhững report này có header không thuộc nhóm `findings` / `impression`.\nThường là admin-only hoặc các header category `other` / `view_desc` mà parser hiện tại bỏ qua."
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "47f751e4",
+   "metadata": {},
+   "outputs": [],
+   "source": "from collections import Counter as _Counter\n\n# Reports khong co findings/impression nhung co it nhat 1 section\nno_fi = df[(df[\"findings\"].isna()) & (df[\"impression\"].isna()) & (df[\"n_sections\"] > 0)].copy()\n\nno_fi_cat = _Counter()\nfor headers_str in no_fi[\"section_headers\"].dropna():\n    for h in headers_str.split(\"|\"):\n        h = h.strip()\n        if h:\n            no_fi_cat[classify_header(h)] += 1\n\nprint(f\"Reports khong co findings/impression nhung co sections: {len(no_fi):,}\")\nprint(\"\\nPhan bo category header trong nhom nay:\")\nfor cat, cnt in sorted(no_fi_cat.items(), key=lambda x: -x[1]):\n    print(f\"  {cat:<12}: {cnt:>6,}\")\n\n# Loc nhung report co header 'other' (that su la)\nother_set = set(header_df[header_df[\"category\"] == \"other\"][\"header\"].str.upper())\nhas_other = no_fi[no_fi[\"section_headers\"].apply(\n    lambda s: bool(set(h.strip().upper() for h in str(s).split(\"|\")) & other_set)\n)]\nprint(f\"\\nTrong do co header 'other' (la nhat): {len(has_other):,}\")\n\nN = 10  # so vi du\nsample_src = has_other if len(has_other) >= N else no_fi\nsample = sample_src.sample(min(N, len(sample_src)), random_state=7)\n\nprint(f\"\\n=== Vi du bao cao co section la (khong FINDINGS / IMPRESSION) ===\")\nfor _, row in sample.iterrows():\n    print(f\"  {report_path(row)}   sections: {row['section_headers']}\")"
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -1285,4 +1333,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 5
-}
+}

scripts/vertex_job.yaml

@@ -44,7 +44,7 @@ workerPoolSpecs:
       exec python scripts/gcp_entrypoint.py
     env:
     - name: HF_TOKEN
-      value: 
+      value: _tSxYqkUSvesmWQOtoyrNpsHmfZlNFVUDRb
     - name: DATASET_NAME
       value: MIMIC-CXR_resized
     - name: REPORT_MODE