{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "642e9523",
      "metadata": {},
      "source": [
        "# experiments/explore.ipynb — 評価結果を対話的に掘るスクラッチ\n",
        "\n",
        "前提:\n",
        "- dev 依存に ipykernel / pandas / umap-learn / matplotlib / scikit-learn（`uv add --dev` 済み）\n",
        "- **repo ルートから実行する**（config.DATA_DIR が cwd 基準の ./data のため）\n",
        "- Qdrant が起動していること（埋め込み可視化セルで scroll する）\n",
        "- カーネルは `music-rag`（venv の Python）を選択する"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "a7d99a6c",
      "metadata": {},
      "outputs": [
        {
          "ename": "ModuleNotFoundError",
          "evalue": "No module named 'matplotlib'",
          "output_type": "error",
          "traceback": [
            "\u001b[31m---------------------------------------------------------------------------\u001b[39m",
            "\u001b[31mModuleNotFoundError\u001b[39m                       Traceback (most recent call last)",
            "\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[1]\u001b[39m\u001b[32m, line 4\u001b[39m\n\u001b[32m      1\u001b[39m \u001b[38;5;66;03m# ── セットアップ ──\u001b[39;00m\n\u001b[32m      2\u001b[39m \u001b[38;5;28;01mimport\u001b[39;00m json\n\u001b[32m      3\u001b[39m \n\u001b[32m----> \u001b[39m\u001b[32m4\u001b[39m \u001b[38;5;28;01mimport\u001b[39;00m matplotlib.pyplot \u001b[38;5;28;01mas\u001b[39;00m plt\n\u001b[32m      5\u001b[39m \u001b[38;5;28;01mimport\u001b[39;00m numpy \u001b[38;5;28;01mas\u001b[39;00m np\n\u001b[32m      6\u001b[39m \u001b[38;5;28;01mimport\u001b[39;00m pandas \u001b[38;5;28;01mas\u001b[39;00m pd\n\u001b[32m      7\u001b[39m \n",
            "\u001b[31mModuleNotFoundError\u001b[39m: No module named 'matplotlib'"
          ]
        }
      ],
      "source": [
        "# ── セットアップ ──\n",
        "import json\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "from music_rag import config, embedder, retriever\n",
        "\n",
        "plt.rcParams[\"font.family\"] = \"Hiragino Sans\"  # 日本語タイトルの豆腐化防止(macOS)\n",
        "pd.set_option(\"display.max_colwidth\", 60)\n",
        "pd.set_option(\"display.width\", 160)\n",
        "\n",
        "EVAL = config.EVAL_DIR\n",
        "EVAL_SET = json.loads((EVAL / \"eval_set_merged.json\").read_text(encoding=\"utf-8\"))\n",
        "BY_ID = {r[\"id\"]: r for r in EVAL_SET}\n",
        "print(f\"eval set: {len(EVAL_SET)} 問  ({EVAL/'eval_set_merged.json'})\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "efb9fa26",
      "metadata": {},
      "source": [
        "## 1. 検索スコア（scores_*.json）を DataFrame で眺める\n",
        "\n",
        "`uv run python experiments/evaluation.py` が吐く per-question を読む。\n",
        "ここにあるのは retrieval 層の指標（recall@k / strict_hit / MRR）で、**RAGAS の管轄外**。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "06e4a056",
      "metadata": {},
      "outputs": [],
      "source": [
        "SCORES_PATH = sorted(EVAL.glob(\"scores_2*.json\"))[-1]  # 最新の日付を採用\n",
        "scores = json.loads(SCORES_PATH.read_text(encoding=\"utf-8\"))\n",
        "print(f\"loaded {SCORES_PATH.name}  strategies={list(scores)}\")\n",
        "\n",
        "\n",
        "def scores_df(strategy: str) -> pd.DataFrame:\n",
        "    return pd.DataFrame(scores[strategy][\"per_question\"])\n",
        "\n",
        "\n",
        "df = scores_df(\"structure\")\n",
        "df[[\"id\", \"match_type\", \"source\", \"difficulty\", \"recall\", \"strict_hit\", \"reciprocal_rank\"]]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "49d51ac6",
      "metadata": {},
      "source": [
        "### 層別集計 — 全体平均は AND質問（多ソース比較）の弱点を隠すので match_type 別に割る"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "cff35a0c",
      "metadata": {},
      "outputs": [],
      "source": [
        "df.groupby(\"match_type\")[[\"recall\", \"strict_hit\", \"reciprocal_rank\"]].agg([\"mean\", \"count\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c3a6dee3",
      "metadata": {},
      "source": [
        "### fixed vs structure を 1 問ずつ突き合わせ（paired diff の素）\n",
        "\n",
        "差 `d_recall` が負の行 = structure で悪化した問題。どの match_type で動いたかを見る。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2a9bbfa9",
      "metadata": {},
      "outputs": [],
      "source": [
        "f = scores_df(\"fixed\").set_index(\"id\")\n",
        "s = scores_df(\"structure\").set_index(\"id\")\n",
        "cmp = pd.DataFrame({\n",
        "    \"match_type\": s[\"match_type\"],\n",
        "    \"recall_fixed\": f[\"recall\"],\n",
        "    \"recall_struct\": s[\"recall\"],\n",
        "    \"d_recall\": (s[\"recall\"] - f[\"recall\"]).round(4),\n",
        "    \"d_mrr\": (s[\"reciprocal_rank\"] - f[\"reciprocal_rank\"]).round(4),\n",
        "})\n",
        "cmp.sort_values(\"d_recall\")  # 悪化が上、改善が下"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "711ac186",
      "metadata": {},
      "source": [
        "## 2. 「拾ってるのに捨てている」Qdrant の生 cos 類似度を実際に見る\n",
        "\n",
        "`evaluation.retrieve_with_text` は `{\"text\",\"source\",\"score\"}` を返すが、\n",
        "`evaluate_generation` は text だけ使い `score` を捨てている。ここでは score を残して並べる。\n",
        "これは BGE-M3 × Qdrant の cos で、RAGAS が内部で使う Gemini embedding の cos とは別物。\n",
        "\n",
        "`QID` を変えれば任意の質問を掘れる（AND質問の例を初期値に）。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5096ab9c",
      "metadata": {},
      "outputs": [],
      "source": [
        "QID = \"forum_76963\"  # ← 見たい質問idに変える\n",
        "row = BY_ID[QID]\n",
        "expected = set(row[\"expected_source\"])\n",
        "print(f\"Q: {row['question']}\")\n",
        "print(f\"expected ({row['match_type']}): {sorted(expected)}\")\n",
        "\n",
        "qvec = embedder.embed_query(row[\"question\"])\n",
        "hits = retriever.search(qvec, top_k=config.TOP_K, collection=config.COLLECTION_NAME)\n",
        "hits_df = pd.DataFrame([{\n",
        "    \"rank\": i + 1,\n",
        "    \"source_tail\": h[\"source\"].split(\"_\")[-1],\n",
        "    \"cos_score\": round(h[\"score\"], 4),\n",
        "    \"is_correct\": h[\"source\"] in expected,\n",
        "    \"chunk_index\": h[\"chunk_index\"],\n",
        "} for i, h in enumerate(hits)])\n",
        "hits_df"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d5dda8a9",
      "metadata": {},
      "source": [
        "## 3. RAGAS（生成層）の 5 指標を 1 問ずつ見る\n",
        "\n",
        "`run_ragas` が吐いた json。faithfulness / answer_relevancy / answer_correctness /\n",
        "context_precision / context_recall。まだ 66 問版が無ければ過去の structure 版を読む。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5232c53f",
      "metadata": {},
      "outputs": [],
      "source": [
        "metric_cols = [\n",
        "    \"faithfulness\", \"answer_relevancy\", \"answer_correctness\",\n",
        "    \"context_precision\", \"context_recall\",\n",
        "]\n",
        "ragas_files = sorted(EVAL.glob(\"ragas_*structure*.json\")) or sorted(EVAL.glob(\"ragas_*.json\"))\n",
        "if ragas_files:\n",
        "    RAGAS_PATH = ragas_files[-1]\n",
        "    ragas = json.loads(RAGAS_PATH.read_text(encoding=\"utf-8\"))\n",
        "    print(f\"loaded {RAGAS_PATH.name}  n={ragas['n']}\")\n",
        "    rdf = pd.DataFrame(ragas[\"per_question\"])\n",
        "    display_cols = [\"question\", *[c for c in metric_cols if c in rdf.columns]]\n",
        "    rdf_view = rdf[display_cols]\n",
        "else:\n",
        "    print(\"ragas_*.json が無い。run_ragas 実行後にこのセルを回す\")\n",
        "    rdf, rdf_view = pd.DataFrame(), pd.DataFrame()\n",
        "rdf_view"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "87593502",
      "metadata": {},
      "source": [
        "### RAGAS 5 指標の分布（箱ひげ）— 平均だけでなくばらつきを見る"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "cee8b247",
      "metadata": {},
      "outputs": [],
      "source": [
        "if not rdf.empty:\n",
        "    present = [c for c in metric_cols if c in rdf.columns]\n",
        "    rdf[present].plot(kind=\"box\", figsize=(9, 4), rot=20,\n",
        "                      title=f\"RAGAS 5指標の分布 ({RAGAS_PATH.name})\")\n",
        "    plt.tight_layout()\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "182fbc20",
      "metadata": {},
      "source": [
        "## 4. 埋め込み空間を UMAP で 2D に落として 1 問を可視化\n",
        "\n",
        "灰 = 全チャンク / 緑 = 正解記事のチャンク / 赤縁 = top-k ヒット / 星 = クエリ。\n",
        "→ まさに「リトリーブした top-K を色で / 正解ラベルを別色で」。\n",
        "\n",
        "UMAP は t-SNE と違い `transform` を持つ。**コーパスに一度 fit すれば、後から来た\n",
        "クエリ点を同じ 2D 空間へ写せる**（クエリ拡張の前後比較＝次セルに必須）。\n",
        "`scroll_all` は Qdrant 全ポイントをベクトル付きで取る（viz_retrieval.py と同じ発想）。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c209944d",
      "metadata": {},
      "outputs": [],
      "source": [
        "import umap\n",
        "\n",
        "\n",
        "def scroll_all(collection: str):\n",
        "    from qdrant_client import QdrantClient\n",
        "    client = QdrantClient(url=config.QDRANT_URL, api_key=config.QDRANT_API_KEY)\n",
        "    vecs, payloads, offset = [], [], None\n",
        "    while True:\n",
        "        pts, offset = client.scroll(\n",
        "            collection_name=collection, limit=512, offset=offset,\n",
        "            with_payload=True, with_vectors=True,\n",
        "        )\n",
        "        for p in pts:\n",
        "            vecs.append(p.vector)\n",
        "            payloads.append(p.payload)\n",
        "        if offset is None:\n",
        "            break\n",
        "    return np.asarray(vecs, dtype=np.float32), payloads\n",
        "\n",
        "\n",
        "COLLECTION = config.COLLECTION_NAME\n",
        "corpus_vecs, payloads = scroll_all(COLLECTION)\n",
        "print(f\"corpus: {len(corpus_vecs)} points @ {COLLECTION}\")\n",
        "\n",
        "# cos 空間で作った BGE-M3 ベクトルなので UMAP も metric=\"cosine\"。\n",
        "reducer = umap.UMAP(n_neighbors=15, min_dist=0.1, metric=\"cosine\", random_state=0)\n",
        "xy = reducer.fit_transform(corpus_vecs)  # この 2D 空間を以降のセルで再利用する"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "60db6679",
      "metadata": {},
      "outputs": [],
      "source": [
        "# ── プロット（QID / hits は §2 のセルで計算済みのものを使う）──\n",
        "q_xy = reducer.transform(np.asarray(qvec, dtype=np.float32)[None, :])[0]\n",
        "hit_keys = {(h[\"source\"], h[\"chunk_index\"]) for h in hits}\n",
        "is_expected = np.array([p.get(\"source\") in expected for p in payloads])\n",
        "is_hit = np.array([(p.get(\"source\"), p.get(\"chunk_index\")) in hit_keys for p in payloads])\n",
        "\n",
        "fig, ax = plt.subplots(figsize=(10, 8))\n",
        "ax.scatter(xy[:, 0], xy[:, 1], s=6, c=\"#dddddd\", linewidths=0, label=\"全チャンク\")\n",
        "ax.scatter(xy[is_expected, 0], xy[is_expected, 1], s=30, c=\"#2e8b57\",\n",
        "           linewidths=0, label=\"正解記事のチャンク\")\n",
        "ax.scatter(xy[is_hit, 0], xy[is_hit, 1], s=95, facecolors=\"none\",\n",
        "           edgecolors=\"#d62728\", linewidths=1.8, label=f\"top-{config.TOP_K} ヒット\")\n",
        "ax.scatter(q_xy[0], q_xy[1], s=280, marker=\"*\", c=\"#111111\", zorder=5, label=\"クエリ\")\n",
        "\n",
        "# 正解記事ごとに重心へ記事名（slug末尾）を添える\n",
        "for src in sorted(expected):\n",
        "    m = np.array([p.get(\"source\") == src for p in payloads])\n",
        "    if m.any():\n",
        "        ax.annotate(src.split(\"_\")[-1], (xy[m, 0].mean(), xy[m, 1].mean()),\n",
        "                    fontsize=9, color=\"#1a5e38\", weight=\"bold\")\n",
        "\n",
        "ax.set_title(f\"[{row['id']}] {row['question'][:46]}…  match_type={row['match_type']}\")\n",
        "ax.legend(loc=\"best\")\n",
        "ax.set_xticks([])\n",
        "ax.set_yticks([])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "17972f66",
      "metadata": {},
      "source": [
        "## 5. 拡張クエリ（HyDE）を可視化 — 元クエリ vs 仮の回答で膨らませたクエリ\n",
        "\n",
        "参考資料の図（赤 X = 元クエリ / オレンジ X = 拡張クエリ / 緑 = 検索された doc）と同じ発想。\n",
        "LLM に**資料なしで仮の回答（hypothetical answer）を書かせ**、元質問と連結して埋め込む。\n",
        "拡張後のクエリ点（オレンジ）が正解記事（緑）へ寄れば、クエリ拡張が効く見込み。\n",
        "AND質問（structure recall 0.35）に効くかの事前診断に使う。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6a61836f",
      "metadata": {},
      "outputs": [],
      "source": [
        "from music_rag import llm as llm_module\n",
        "\n",
        "hypo = llm_module.explain(row[\"question\"], [], None)  # 資料なし＝仮の回答を書かせる(HyDE)\n",
        "augmented = row[\"question\"] + \"\\n\" + hypo\n",
        "aug_vec = embedder.embed_query(augmented)\n",
        "\n",
        "orig_xy = reducer.transform(np.asarray(qvec, dtype=np.float32)[None, :])[0]\n",
        "aug_xy = reducer.transform(np.asarray(aug_vec, dtype=np.float32)[None, :])[0]\n",
        "\n",
        "fig, ax = plt.subplots(figsize=(10, 8))\n",
        "ax.scatter(xy[:, 0], xy[:, 1], s=6, c=\"#dddddd\", linewidths=0, label=\"全チャンク\")\n",
        "ax.scatter(xy[is_expected, 0], xy[is_expected, 1], s=30, c=\"#2e8b57\",\n",
        "           linewidths=0, label=\"正解記事のチャンク\")\n",
        "ax.scatter(orig_xy[0], orig_xy[1], s=260, marker=\"X\", c=\"#d62728\",\n",
        "           zorder=5, label=\"元クエリ\")\n",
        "ax.scatter(aug_xy[0], aug_xy[1], s=260, marker=\"X\", c=\"#ff8c00\",\n",
        "           zorder=5, label=\"拡張クエリ(HyDE)\")\n",
        "for src in sorted(expected):\n",
        "    m = np.array([p.get(\"source\") == src for p in payloads])\n",
        "    if m.any():\n",
        "        ax.annotate(src.split(\"_\")[-1], (xy[m, 0].mean(), xy[m, 1].mean()),\n",
        "                    fontsize=9, color=\"#1a5e38\", weight=\"bold\")\n",
        "\n",
        "ax.set_title(f\"[{row['id']}] 元クエリ vs 拡張クエリ — 正解記事(緑)に近づくか\")\n",
        "ax.legend(loc=\"best\")\n",
        "ax.set_xticks([])\n",
        "ax.set_yticks([])\n",
        "plt.show()\n",
        "print(\"hypothetical answer (先頭200字):\\n\", hypo[:200])"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "rag_tim",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.15"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}