{ "cells": [ { "cell_type": "markdown", "id": "title-cell", "metadata": {}, "source": [ "# 02 — Embeddings & ChromaDB\n", "## Morningstar RAG Pipeline\n", "\n", "This notebook covers **Phase 4** — converting document chunks into vector embeddings and storing them in a persistent ChromaDB collection ready for retrieval.\n", "\n", "### What happens in this phase\n", "```\n", "Chunks JSON (text + tables + metadata)\n", " ↓\n", "all-MiniLM-L6-v2 ← encodes each chunk as a 384-dim vector\n", " ↓\n", "ChromaDB ← persists vectors + original text + metadata on disk\n", " ↓\n", "Similarity search ← query string → top-k matching chunks\n", "```\n", "\n", "### Steps in this notebook\n", "```\n", "── MORNINGSTAR ──────────────────────────────────────────────\n", "STEP 1 — Imports & Paths\n", "STEP 2 — Why all-MiniLM-L6-v2 (model choice)\n", "STEP 3 — Load & Inspect Chunks\n", "STEP 4 — Set Up ChromaDB Collection\n", "STEP 5 — Embed One Document (demo)\n", "STEP 6 — Inspect the Collection\n", "STEP 7 — Test Similarity Queries\n", "STEP 8 — Batch Embed All Morningstar Documents\n", "\n", "── SEC FILINGS ──────────────────────────────────────────────\n", "STEP 9 — Chunk SEC Filings (prerequisite: run sec_processor.ipynb first)\n", "STEP 10 — Embed SEC Chunks into ChromaDB\n", "STEP 11 — Verify SEC-Specific Retrieval\n", "```" ] }, { "cell_type": "markdown", "id": "step1-header", "metadata": {}, "source": [ "## STEP 1 — Imports & Paths" ] }, { "cell_type": "code", "execution_count": 1, "id": "step1-code", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Project root : /home/pushkardeshpand/Documents/Morningstar RAG Pipeline\n", "Chunks dir : /home/pushkardeshpand/Documents/Morningstar RAG Pipeline/data/chunks\n", "Vectorstore dir : /home/pushkardeshpand/Documents/Morningstar RAG Pipeline/data/vectorstore\n", "\n", "Chunk files found: 2\n", " morningstar/a-wide-moat-focus-provides-differentiation_chunks.json (53 KB)\n", " morningstar/ptc01302411420_chunks.json (100 KB)\n" ] } ], "source": [ "import json\n", "import sys\n", "from pathlib import Path\n", "\n", "# ── Project paths ──────────────────────────────────────────────────────────────\n", "NOTEBOOK_DIR = Path().resolve()\n", "PROJECT_ROOT = NOTEBOOK_DIR.parent\n", "SRC_DIR = PROJECT_ROOT / \"src\"\n", "CHUNKS_DIR = PROJECT_ROOT / \"data\" / \"chunks\"\n", "VECTORSTORE_DIR = PROJECT_ROOT / \"data\" / \"vectorstore\"\n", "\n", "sys.path.insert(0, str(SRC_DIR))\n", "\n", "print(f\"Project root : {PROJECT_ROOT}\")\n", "print(f\"Chunks dir : {CHUNKS_DIR}\")\n", "print(f\"Vectorstore dir : {VECTORSTORE_DIR}\")\n", "print()\n", "\n", "# List available chunk files\n", "chunk_files = sorted(CHUNKS_DIR.rglob(\"*_chunks.json\"))\n", "print(f\"Chunk files found: {len(chunk_files)}\")\n", "for cf in chunk_files:\n", " size_kb = cf.stat().st_size / 1024\n", " print(f\" {cf.relative_to(CHUNKS_DIR)} ({size_kb:.0f} KB)\")" ] }, { "cell_type": "markdown", "id": "step2-header", "metadata": {}, "source": [ "## STEP 2 — Why `all-MiniLM-L6-v2`\n", "\n", "The embedding model converts text into a dense vector. Every design choice here has a direct impact on retrieval quality.\n", "\n", "### Model comparison\n", "\n", "| Model | Dims | Tokens | Speed | Financial quality |\n", "|---|---|---|---|---|\n", "| `all-MiniLM-L6-v2` | 384 | 256 | Fast | Good (our choice) |\n", "| `all-mpnet-base-v2` | 768 | 384 | Medium | Better |\n", "| `ProsusAI/finbert` | 768 | 512 | Slow | Best for finance |\n", "\n", "### Why `all-MiniLM-L6-v2` for now\n", "- **Fast**: ~14k tokens/sec on CPU — suitable for batch embedding 1000s of chunks\n", "- **256 token window**: matches our `max_tokens=256` in HybridChunker exactly — no truncation\n", "- **Good baseline**: trained on 1B sentence pairs — handles financial terminology well\n", "- **Upgrade path**: swap for `ProsusAI/finbert` in production for better precision\n", "\n", "### The critical rule: same model for indexing AND retrieval\n", "```\n", "Indexing time: chunk text → all-MiniLM-L6-v2 → 384-dim vector → ChromaDB\n", "Retrieval time: query text → all-MiniLM-L6-v2 → 384-dim vector → cosine search\n", "```\n", "If you use a different model at retrieval time, the query vector lives in a different\n", "vector space — similarity scores become meaningless." ] }, { "cell_type": "markdown", "id": "step3-header", "metadata": {}, "source": [ "## STEP 3 — Load & Inspect Chunks\n", "\n", "Before embedding, let's see what we are working with — chunk counts, types, and sizes." ] }, { "cell_type": "code", "execution_count": 2, "id": "step3-code", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "File : a-wide-moat-focus-provides-differentiation_chunks.json\n", " doc_id : a-wide-moat-focus-provides-differentiation\n", " total chunks : 32 (22 text + 10 table)\n", " chunk size : min=45 max=2833 avg=771 chars\n", "\n", "File : ptc01302411420_chunks.json\n", " doc_id : ptc01302411420\n", " total chunks : 64 (52 text + 12 table)\n", " chunk size : min=12 max=2003 avg=708 chars\n", "\n", "============================================================\n", "TEXT chunk example:\n", " chunk_id : a-wide-moat-focus-provides-differentiation_text_0000\n", " is_atomic : False\n", " heading_path : \n", " page_num : 1\n", " text : Director of Equity Research, Index Strategies...\n", "\n", "TABLE chunk example:\n", " chunk_id : a-wide-moat-focus-provides-differentiation_table_000\n", " is_atomic : True\n", " rows x cols : 4 x 2\n", " col_headers : ['0', '1']\n", " text preview :\n", "| Annualized Return% | 10.39 |\n", "|-------------------------------|---------|\n", "| Annualized Standard Deviation | 19.95 |\n", "| Sharpe Ratio | 0.45 |\n", "| Maximum Drawdown% | -50.95 |\n" ] } ], "source": [ "all_chunks_by_file = {}\n", "\n", "for cf in chunk_files:\n", " with open(cf) as f:\n", " data = json.load(f)\n", " all_chunks_by_file[cf.name] = data\n", "\n", " chunks = data[\"chunks\"]\n", " text_chunks = [c for c in chunks if c[\"chunk_type\"] == \"text\"]\n", " tbl_chunks = [c for c in chunks if c[\"chunk_type\"] == \"table\"]\n", " sizes = [len(c[\"text\"]) for c in chunks]\n", "\n", " print(f\"File : {cf.name}\")\n", " print(f\" doc_id : {data['doc_id']}\")\n", " print(f\" total chunks : {data['total_chunks']} ({data['text_chunks']} text + {data['table_chunks']} table)\")\n", " print(f\" chunk size : min={min(sizes)} max={max(sizes)} avg={sum(sizes)//len(sizes)} chars\")\n", " print()\n", "\n", "# Show structure of one text chunk and one table chunk\n", "all_chunks = all_chunks_by_file[chunk_files[0].name][\"chunks\"]\n", "text_ex = next(c for c in all_chunks if c[\"chunk_type\"] == \"text\")\n", "table_ex = next(c for c in all_chunks if c[\"chunk_type\"] == \"table\")\n", "\n", "print(\"=\" * 60)\n", "print(\"TEXT chunk example:\")\n", "print(f\" chunk_id : {text_ex['chunk_id']}\")\n", "print(f\" is_atomic : {text_ex['is_atomic']}\")\n", "print(f\" heading_path : {text_ex['metadata'].get('heading_path', '')}\")\n", "print(f\" page_num : {text_ex['metadata'].get('page_num')}\")\n", "print(f\" text : {text_ex['text'][:200]}...\")\n", "print()\n", "print(\"TABLE chunk example:\")\n", "print(f\" chunk_id : {table_ex['chunk_id']}\")\n", "print(f\" is_atomic : {table_ex['is_atomic']}\")\n", "print(f\" rows x cols : {table_ex['metadata'].get('rows')} x {table_ex['metadata'].get('cols')}\")\n", "print(f\" col_headers : {table_ex['metadata'].get('col_headers')}\")\n", "print(f\" text preview :\")\n", "print(table_ex[\"text\"][:300])" ] }, { "cell_type": "markdown", "id": "step4-header", "metadata": {}, "source": [ "## STEP 4 — Set Up ChromaDB Collection\n", "\n", "ChromaDB stores vectors on disk in a persistent directory. We use a **single collection** for all document types.\n", "\n", "### Why one collection for everything?\n", "```\n", "Separate collections: One collection:\n", " morningstar_collection financial_docs\n", " sec_filings_collection ├── source=morningstar\n", " ├── source=sec_edgar\n", " └── doc_type=10-K / research_report\n", "```\n", "- **Cross-document retrieval**: a query about Apple revenue can match both a Morningstar report AND an Apple 10-K — this cross-document signal is valuable\n", "- **Metadata filters**: when you want source-specific results, pass `where={\"source\": \"morningstar\"}` at query time\n", "- **Single embedding space**: cosine similarity across all documents is meaningful because all vectors are from the same model\n", "\n", "### Distance metric: cosine\n", "Sentence transformers are trained to produce unit-norm vectors. Cosine similarity is the natural distance metric — the dot product of two unit vectors equals their cosine similarity." ] }, { "cell_type": "code", "execution_count": 3, "id": "step4-code", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n", "I0000 00:00:1782113159.449906 3097344 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n", "I0000 00:00:1782113159.482262 3097344 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n", "To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", "WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n", "I0000 00:00:1782113160.096890 3097344 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Collection : 'financial_docs'\n", "Embedding model: sentence-transformers/all-MiniLM-L6-v2\n", "Distance space : cosine\n", "Persist path : /home/pushkardeshpand/Documents/Morningstar RAG Pipeline/data/vectorstore\n", "Vectors stored : 0\n" ] } ], "source": [ "import chromadb\n", "from chromadb.utils.embedding_functions import SentenceTransformerEmbeddingFunction\n", "\n", "EMBEDDING_MODEL = \"sentence-transformers/all-MiniLM-L6-v2\"\n", "COLLECTION_NAME = \"financial_docs\"\n", "\n", "# Persistent client — collection data survives across sessions\n", "VECTORSTORE_DIR.mkdir(parents=True, exist_ok=True)\n", "client = chromadb.PersistentClient(path=str(VECTORSTORE_DIR))\n", "\n", "# Embedding function — same model used for indexing AND retrieval\n", "ef = SentenceTransformerEmbeddingFunction(model_name=EMBEDDING_MODEL)\n", "\n", "# Get or create the collection\n", "collection = client.get_or_create_collection(\n", " name = COLLECTION_NAME,\n", " embedding_function = ef,\n", " metadata = {\"hnsw:space\": \"cosine\"},\n", ")\n", "\n", "print(f\"Collection : '{COLLECTION_NAME}'\")\n", "print(f\"Embedding model: {EMBEDDING_MODEL}\")\n", "print(f\"Distance space : cosine\")\n", "print(f\"Persist path : {VECTORSTORE_DIR}\")\n", "print(f\"Vectors stored : {collection.count()}\")" ] }, { "cell_type": "markdown", "id": "step5-header", "metadata": {}, "source": [ "## STEP 5 — Embed One Document (Demo)\n", "\n", "We embed the PTC research report chunk by chunk and upsert into ChromaDB.\n", "\n", "### What happens to each chunk\n", "```\n", "chunk.text → all-MiniLM-L6-v2 → [0.032, -0.118, 0.204, ...] (384 floats)\n", " ↓\n", " ChromaDB stores:\n", " id = \"ptc01302411420_text_0001\"\n", " document = chunk.text\n", " vector = [0.032, -0.118, ...]\n", " metadata = {source, doc_type, page_num, ...}\n", "```\n", "\n", "### Metadata constraints\n", "ChromaDB only accepts `str | int | float | bool` — not lists or None.\n", "We sanitise before inserting: `None → \"\"`, `list → JSON string`." ] }, { "cell_type": "code", "execution_count": 4, "id": "step5-code", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Embedding 64 chunks from ptc01302411420_chunks.json\n", " Text chunks : 52\n", " Table chunks : 12\n", "\n", "Upserted 64 chunks in 1.8s\n", "Collection total: 64 vectors\n" ] } ], "source": [ "import time\n", "\n", "def sanitize_metadata(meta: dict) -> dict:\n", " \"\"\"Convert metadata to ChromaDB-compatible scalar values.\"\"\"\n", " clean = {}\n", " for k, v in meta.items():\n", " if isinstance(v, (str, int, float, bool)):\n", " clean[k] = v\n", " elif v is None:\n", " clean[k] = \"\"\n", " elif isinstance(v, list):\n", " clean[k] = json.dumps(v, ensure_ascii=False)\n", " else:\n", " clean[k] = str(v)\n", " return clean\n", "\n", "\n", "# Load PTC chunks\n", "ptc_file = CHUNKS_DIR / \"morningstar\" / \"ptc01302411420_chunks.json\"\n", "with open(ptc_file) as f:\n", " ptc_data = json.load(f)\n", "\n", "chunks = ptc_data[\"chunks\"]\n", "BATCH_SIZE = 100\n", "\n", "ids = [c[\"chunk_id\"] for c in chunks]\n", "documents = [c[\"text\"] for c in chunks]\n", "metadatas = []\n", "for c in chunks:\n", " meta = sanitize_metadata(c[\"metadata\"])\n", " meta[\"chunk_type\"] = c[\"chunk_type\"]\n", " meta[\"is_atomic\"] = c[\"is_atomic\"]\n", " meta[\"doc_id\"] = c[\"doc_id\"]\n", " metadatas.append(meta)\n", "\n", "print(f\"Embedding {len(chunks)} chunks from {ptc_file.name}\")\n", "print(f\" Text chunks : {ptc_data['text_chunks']}\")\n", "print(f\" Table chunks : {ptc_data['table_chunks']}\")\n", "print()\n", "\n", "start = time.time()\n", "for i in range(0, len(ids), BATCH_SIZE):\n", " collection.upsert(\n", " ids = ids[i : i + BATCH_SIZE],\n", " documents = documents[i : i + BATCH_SIZE],\n", " metadatas = metadatas[i : i + BATCH_SIZE],\n", " )\n", "\n", "elapsed = time.time() - start\n", "print(f\"Upserted {len(chunks)} chunks in {elapsed:.1f}s\")\n", "print(f\"Collection total: {collection.count()} vectors\")" ] }, { "cell_type": "markdown", "id": "step6-header", "metadata": {}, "source": [ "## STEP 6 — Inspect the Collection\n", "\n", "Verify what is stored: vector count, metadata fields, and peek at a few records." ] }, { "cell_type": "code", "execution_count": 5, "id": "step6-code", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total vectors in collection: 64\n", "\n", "Peek at first 3 stored vectors:\n", "------------------------------------------------------------\n", "\n", " [1] id : ptc01302411420_text_0000\n", " source : morningstar\n", " doc_type : research_report\n", " type : text\n", " page : 1\n", " heading : Morningstar Indexes Calculation Methodology\n", " text : January 2024...\n", "\n", " [2] id : ptc01302411420_text_0001\n", " source : morningstar\n", " doc_type : research_report\n", " type : text\n", " page : 3\n", " heading : Overview\n", " text : This document outlines the calculation methodology for the Morningstar Global Equity Indexes. An ind...\n", "\n", " [3] id : ptc01302411420_text_0002\n", " source : morningstar\n", " doc_type : research_report\n", " type : text\n", " page : 3\n", " heading : Index Divisor\n", " text : Morningstar Indexes use the concept of an \"index divisor\" to calculate daily index levels. The perfo...\n", "\n", "Breakdown by type:\n", " text : 52\n", " table : 12\n" ] } ], "source": [ "# Total count\n", "print(f\"Total vectors in collection: {collection.count()}\")\n", "print()\n", "\n", "# Peek at first 3 stored items\n", "peek = collection.peek(limit=3)\n", "\n", "print(\"Peek at first 3 stored vectors:\")\n", "print(\"-\" * 60)\n", "for i in range(len(peek[\"ids\"])):\n", " print(f\"\\n [{i+1}] id : {peek['ids'][i]}\")\n", " m = peek[\"metadatas\"][i]\n", " print(f\" source : {m.get('source')}\")\n", " print(f\" doc_type : {m.get('doc_type')}\")\n", " print(f\" type : {m.get('chunk_type')}\")\n", " print(f\" page : {m.get('page_num')}\")\n", " print(f\" heading : {m.get('heading_path', '')[:60]}\")\n", " print(f\" text : {peek['documents'][i][:100]}...\")\n", "\n", "print()\n", "\n", "# Breakdown by chunk_type\n", "all_meta = collection.get(include=[\"metadatas\"])[\"metadatas\"]\n", "text_count = sum(1 for m in all_meta if m.get(\"chunk_type\") == \"text\")\n", "table_count = sum(1 for m in all_meta if m.get(\"chunk_type\") == \"table\")\n", "print(f\"Breakdown by type:\")\n", "print(f\" text : {text_count}\")\n", "print(f\" table : {table_count}\")" ] }, { "cell_type": "markdown", "id": "step7-header", "metadata": {}, "source": [ "## STEP 7 — Test Similarity Queries\n", "\n", "Now we run similarity searches to verify the embeddings are working correctly.\n", "\n", "ChromaDB returns results sorted by **cosine distance** (lower = more similar, 0 = identical).\n", "\n", "We test three query types:\n", "1. **Plain text query** — retrieves the best-matching text chunks\n", "2. **Table query** — retrieves the best-matching table chunks\n", "3. **Metadata-filtered query** — restricts results to a specific document" ] }, { "cell_type": "code", "execution_count": 6, "id": "step7a-code", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "============================================================\n", "Query: index divisor calculation methodology\n", "============================================================\n", "\n", " [1] id : ptc01302411420_text_0003\n", " distance : 0.3797 (lower = more similar)\n", " type : text\n", " heading : Index Divisor\n", " page : 3\n", " text : The index divisor remains unchanged unless there is a change in index composition, which can be due to corporate actions, changes in shares outstanding and the float factor, or the...\n", "\n", " [2] id : ptc01302411420_text_0002\n", " distance : 0.3863 (lower = more similar)\n", " type : text\n", " heading : Index Divisor\n", " page : 3\n", " text : Morningstar Indexes use the concept of an \"index divisor\" to calculate daily index levels. The performance of the index is linked to change in the market value of its constituents....\n", "\n", " [3] id : ptc01302411420_text_0006\n", " distance : 0.3942 (lower = more similar)\n", " type : text\n", " heading : Where:\n", " page : 4\n", " text : \n", "Where P d and Q d represent security price and shares of deleted securities, while P a and Q a represent security price and shares of added securities....\n" ] } ], "source": [ "def show_results(results, title):\n", " \"\"\"Pretty-print similarity search results.\"\"\"\n", " print(f\"\\n{'='*60}\")\n", " print(f\"Query: {title}\")\n", " print(\"=\" * 60)\n", " for i, r in enumerate(results[\"ids\"][0]):\n", " m = results[\"metadatas\"][0][i]\n", " doc = results[\"documents\"][0][i]\n", " dist = results[\"distances\"][0][i]\n", " print(f\"\\n [{i+1}] id : {r}\")\n", " print(f\" distance : {dist:.4f} (lower = more similar)\")\n", " print(f\" type : {m.get('chunk_type')}\")\n", " print(f\" heading : {m.get('heading_path', m.get('section_title', ''))[:60]}\")\n", " print(f\" page : {m.get('page_num')}\")\n", " print(f\" text : {doc[:180]}...\" if len(doc) > 180 else f\" text : {doc}\")\n", "\n", "\n", "# ── Query 1: Plain text search ────────────────────────────────────────────────\n", "results = collection.query(\n", " query_texts = [\"index divisor calculation methodology\"],\n", " n_results = 3,\n", " include = [\"documents\", \"metadatas\", \"distances\"],\n", ")\n", "show_results(results, \"index divisor calculation methodology\")" ] }, { "cell_type": "code", "execution_count": 7, "id": "step7b-code", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "============================================================\n", "Query: currency hedge return calculation [tables only]\n", "============================================================\n", "\n", " [1] id : ptc01302411420_table_007\n", " distance : 0.2946 (lower = more similar)\n", " type : table\n", " heading : \n", " page : 17\n", " text : | Where: | | Hedge Return ...\n", "\n", " [2] id : ptc01302411420_table_009\n", " distance : 0.3713 (lower = more similar)\n", " type : table\n", " heading : \n", " page : 18\n", " text : | HR t | = | Hedge Return on day t |\n", "|------------------------|-----|---------------------------------...\n", "\n", " [3] id : ptc01302411420_table_008\n", " distance : 0.5734 (lower = more similar)\n", " type : table\n", " heading : \n", " page : 18\n", " text : | FXRate i1-1d | = | Spot rate of currency i as of one business day before the previous rebalance date |\n", "|------------...\n" ] } ], "source": [ "# ── Query 2: Table-specific search ───────────────────────────────────────────\n", "# Filter to only table chunks — useful when you want structured data back\n", "results = collection.query(\n", " query_texts = [\"currency hedge return calculation\"],\n", " n_results = 3,\n", " where = {\"chunk_type\": \"table\"},\n", " include = [\"documents\", \"metadatas\", \"distances\"],\n", ")\n", "show_results(results, \"currency hedge return calculation [tables only]\")" ] }, { "cell_type": "code", "execution_count": 8, "id": "step7c-code", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "============================================================\n", "Query: equal weighted index rebalancing [no filter]\n", "============================================================\n", "\n", " [1] id : ptc01302411420_text_0008\n", " distance : 0.2988 (lower = more similar)\n", " type : text\n", " heading : Equal Weighted Indexes\n", " page : 5\n", " text : Equal weighted indexes assign equal weightings to each constituent at rebalancing. The weights drift from their original assigned weights as the price of underlying stocks changes ...\n", "\n", " [2] id : ptc01302411420_text_0012\n", " distance : 0.3960 (lower = more similar)\n", " type : text\n", " heading : Capped-Weighted Indexes\n", " page : 7\n", " text : Capped-weighted indexes constrain the maximum weight of a single constituent and/or the sum of the weights of all securities representing a defined group. These groups can be defin...\n", "\n", " [3] id : ptc01302411420_text_0011\n", " distance : 0.4353 (lower = more similar)\n", " type : text\n", " heading : Dividend Dollar-Weighted Indexes\n", " page : 6\n", " text : Wi(t) = Uncapped weight of security i in index at rebalancing time (t) based on float market cap\n", "Or:\n", "Or:\n", "\n", " [4] id : ptc01302411420_text_0010\n", " distance : 0.4690 (lower = more similar)\n", " type : text\n", " heading : Dividend Dollar-Weighted Indexes\n", " page : 6\n", " text : Dividend dollar-weighted indexes are those where the constituents are weighted according to the total dividends paid by the company to investors. Consequently, the available divide...\n", "\n", " [5] id : ptc01302411420_text_0004\n", " distance : 0.4944 (lower = more similar)\n", " type : text\n", " heading : Market Capitalization and Float Market Capitalization Weight\n", " page : 3\n", " text : In market-cap-weighted indexes the weight of each constituent is determined by dividing its market capitalization by the total market capitalization of the index, without accountin...\n" ] } ], "source": [ "# ── Query 3: Mixed search — any type, top result per type ─────────────────────\n", "# No filter: retrieves the single best match regardless of chunk type\n", "results = collection.query(\n", " query_texts = [\"equal weighted index rebalancing\"],\n", " n_results = 5,\n", " include = [\"documents\", \"metadatas\", \"distances\"],\n", ")\n", "show_results(results, \"equal weighted index rebalancing [no filter]\")" ] }, { "cell_type": "markdown", "id": "step8-header", "metadata": {}, "source": [ "## STEP 8 — Batch Embed All Documents\n", "\n", "Now use the `DocumentEmbedder` class from `src/embedder.py` to embed all available chunk files in one call.\n", "\n", "`DocumentEmbedder` handles:\n", "- Skipping chunks already in the store (idempotent)\n", "- Batched upserts\n", "- Metadata sanitisation automatically" ] }, { "cell_type": "code", "execution_count": 9, "id": "step8-code", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2026-06-22 13:44:01,979 INFO ChromaDB collection 'financial_docs' ready (64 vectors already stored)\n", "2026-06-22 13:44:01,980 INFO Found 2 chunk files under /home/pushkardeshpand/Documents/Morningstar RAG Pipeline/data/chunks\n", "2026-06-22 13:44:01,981 INFO Embedding: a-wide-moat-focus-provides-differentiation_chunks.json (32 new, 0 already stored)\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "a1be898fa08643418e24c653ff3caa00", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Batches: 0%| | 0/1 [00:00 0 else (\"skipped (already stored)\" if n == 0 else \"FAILED\")\n", " print(f\" {fname:55s} {status}\")\n", "\n", "# Final collection stats\n", "stats = embedder.get_stats()\n", "print(f\"\\nCollection summary:\")\n", "print(f\" Total vectors : {stats['total_vectors']}\")\n", "print(f\" Sources : {stats['sources']}\")\n", "print(f\" Documents : {stats['doc_ids']}\")\n", "print(f\" Chunk types : {stats['chunk_types']}\")" ] }, { "cell_type": "markdown", "id": "uxw2nn4hst", "metadata": {}, "source": [ "---\n", "\n", "## STEP 9 — Chunk SEC Filings\n", "\n", "> **Prerequisite:** Run `sec_processor.ipynb` first to generate `_docling.json` files\n", "> for all 14 Apple SEC filings. Without those files, HybridChunker cannot run and\n", "> `DocumentChunker` will skip every SEC filing silently.\n", "\n", "Once `_docling.json` files exist, `DocumentChunker.chunk_all()` processes SEC filings\n", "with the **exact same HybridChunker pipeline** used for Morningstar PDFs — same chunk\n", "format, same metadata structure, same deterministic IDs.\n", "\n", "### What SEC chunks look like vs Morningstar chunks\n", "\n", "| Field | Morningstar | SEC Filing |\n", "|---|---|---|\n", "| `source` | `morningstar` | `sec_edgar` |\n", "| `doc_type` | `research_report` | `10-K` / `10-Q` / `8-K` |\n", "| `ticker` | `PTC` | `AAPL` |\n", "| `page_num` | integer (PDF page) | `null` (HTML has no pages) |\n", "| `fiscal_year` | — | `2024` |\n", "| `heading_path` | `\"Index Divisor\"` | `\"PART I > Item 1. Business\"` |" ] }, { "cell_type": "code", "execution_count": 10, "id": "5qsb9d225af", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2026-06-22 14:34:25,617 INFO Found 14 processed documents\n", "2026-06-22 14:34:25,618 INFO Chunking: 10-K_2023.json\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "SEC processed JSONs : 14\n", "SEC _docling.json : 14\n", "\n", "All 14 SEC filings have _docling.json — ready to chunk.\n", "\n", "Chunking SEC filings...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "'(ReadTimeoutError(\"HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)\"), '(Request ID: 88843094-ea05-4ce0-9784-c58104ae4ae5)')' thrown while requesting HEAD https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/tokenizer_config.json\n", "2026-06-22 14:34:36,114 WARNING '(ReadTimeoutError(\"HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)\"), '(Request ID: 88843094-ea05-4ce0-9784-c58104ae4ae5)')' thrown while requesting HEAD https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/tokenizer_config.json\n", "Retrying in 1s [Retry 1/5].\n", "2026-06-22 14:34:36,115 WARNING Retrying in 1s [Retry 1/5].\n", "Token indices sequence length is longer than the specified maximum sequence length for this model (532 > 512). Running this sequence through the model will result in indexing errors\n", "2026-06-22 14:34:56,271 INFO Text chunks : 139\n", "2026-06-22 14:34:56,272 INFO Table chunks : 66\n", "2026-06-22 14:34:56,279 INFO Saved → 10-K_2023_chunks.json (663.2 KB) [139 text + 66 table = 205 total]\n", "2026-06-22 14:34:56,280 INFO Chunking: 10-K_2024.json\n", "Token indices sequence length is longer than the specified maximum sequence length for this model (532 > 512). Running this sequence through the model will result in indexing errors\n", "2026-06-22 14:35:14,849 INFO Text chunks : 154\n", "2026-06-22 14:35:14,849 INFO Table chunks : 63\n", "2026-06-22 14:35:14,855 INFO Saved → 10-K_2024_chunks.json (669.1 KB) [154 text + 63 table = 217 total]\n", "2026-06-22 14:35:14,856 INFO Chunking: 10-K_2025.json\n", "Token indices sequence length is longer than the specified maximum sequence length for this model (532 > 512). Running this sequence through the model will result in indexing errors\n", "2026-06-22 14:35:34,946 INFO Text chunks : 160\n", "2026-06-22 14:35:34,947 INFO Table chunks : 62\n", "2026-06-22 14:35:34,953 INFO Saved → 10-K_2025_chunks.json (677.4 KB) [160 text + 62 table = 222 total]\n", "2026-06-22 14:35:34,954 INFO Chunking: 10-Q_2024_Q3.json\n", "Token indices sequence length is longer than the specified maximum sequence length for this model (532 > 512). Running this sequence through the model will result in indexing errors\n", "2026-06-22 14:35:43,372 INFO Text chunks : 40\n", "2026-06-22 14:35:43,372 INFO Table chunks : 39\n", "2026-06-22 14:35:43,375 INFO Saved → 10-Q_2024_Q3_chunks.json (341.2 KB) [40 text + 39 table = 79 total]\n", "2026-06-22 14:35:43,376 INFO Chunking: 10-Q_2025_Q1.json\n", "Token indices sequence length is longer than the specified maximum sequence length for this model (532 > 512). Running this sequence through the model will result in indexing errors\n", "2026-06-22 14:35:50,454 INFO Text chunks : 49\n", "2026-06-22 14:35:50,455 INFO Table chunks : 37\n", "2026-06-22 14:35:50,459 INFO Saved → 10-Q_2025_Q1_chunks.json (327.9 KB) [49 text + 37 table = 86 total]\n", "2026-06-22 14:35:50,460 INFO Chunking: 10-Q_2025_Q2.json\n", "Token indices sequence length is longer than the specified maximum sequence length for this model (532 > 512). Running this sequence through the model will result in indexing errors\n", "2026-06-22 14:35:59,010 INFO Text chunks : 53\n", "2026-06-22 14:35:59,010 INFO Table chunks : 37\n", "2026-06-22 14:35:59,014 INFO Saved → 10-Q_2025_Q2_chunks.json (361.3 KB) [53 text + 37 table = 90 total]\n", "2026-06-22 14:35:59,015 INFO Chunking: 10-Q_2025_Q3.json\n", "Token indices sequence length is longer than the specified maximum sequence length for this model (532 > 512). Running this sequence through the model will result in indexing errors\n", "2026-06-22 14:36:07,669 INFO Text chunks : 46\n", "2026-06-22 14:36:07,669 INFO Table chunks : 38\n", "2026-06-22 14:36:07,674 INFO Saved → 10-Q_2025_Q3_chunks.json (352.4 KB) [46 text + 38 table = 84 total]\n", "2026-06-22 14:36:07,674 INFO Chunking: 10-Q_2026_Q1.json\n", "Token indices sequence length is longer than the specified maximum sequence length for this model (532 > 512). Running this sequence through the model will result in indexing errors\n", "2026-06-22 14:36:16,124 INFO Text chunks : 43\n", "2026-06-22 14:36:16,125 INFO Table chunks : 38\n", "2026-06-22 14:36:16,129 INFO Saved → 10-Q_2026_Q1_chunks.json (347.6 KB) [43 text + 38 table = 81 total]\n", "2026-06-22 14:36:16,130 INFO Chunking: 10-Q_2026_Q2.json\n", "Token indices sequence length is longer than the specified maximum sequence length for this model (532 > 512). Running this sequence through the model will result in indexing errors\n", "2026-06-22 14:36:28,636 INFO Text chunks : 64\n", "2026-06-22 14:36:28,637 INFO Table chunks : 39\n", "2026-06-22 14:36:28,642 INFO Saved → 10-Q_2026_Q2_chunks.json (435.3 KB) [64 text + 39 table = 103 total]\n", "2026-06-22 14:36:28,643 INFO Chunking: 8-K_2026-01-02.json\n", "2026-06-22 14:36:29,603 INFO Text chunks : 3\n", "2026-06-22 14:36:29,604 INFO Table chunks : 2\n", "2026-06-22 14:36:29,605 INFO Saved → 8-K_2026-01-02_chunks.json (9.0 KB) [3 text + 2 table = 5 total]\n", "2026-06-22 14:36:29,606 INFO Chunking: 8-K_2026-01-29.json\n", "Token indices sequence length is longer than the specified maximum sequence length for this model (1134 > 512). Running this sequence through the model will result in indexing errors\n", "2026-06-22 14:36:30,632 INFO Text chunks : 5\n", "2026-06-22 14:36:30,633 INFO Table chunks : 14\n", "2026-06-22 14:36:30,635 INFO Saved → 8-K_2026-01-29_chunks.json (33.2 KB) [5 text + 14 table = 19 total]\n", "2026-06-22 14:36:30,637 INFO Chunking: 8-K_2026-02-24.json\n", "Token indices sequence length is longer than the specified maximum sequence length for this model (704 > 512). Running this sequence through the model will result in indexing errors\n", "2026-06-22 14:36:31,588 INFO Text chunks : 6\n", "2026-06-22 14:36:31,589 INFO Table chunks : 8\n", "2026-06-22 14:36:31,591 INFO Saved → 8-K_2026-02-24_chunks.json (21.9 KB) [6 text + 8 table = 14 total]\n", "2026-06-22 14:36:31,593 INFO Chunking: 8-K_2026-04-20.json\n", "2026-06-22 14:36:32,501 INFO Text chunks : 5\n", "2026-06-22 14:36:32,502 INFO Table chunks : 2\n", "2026-06-22 14:36:32,503 INFO Saved → 8-K_2026-04-20_chunks.json (11.8 KB) [5 text + 2 table = 7 total]\n", "2026-06-22 14:36:32,504 INFO Chunking: 8-K_2026-04-30.json\n", "Token indices sequence length is longer than the specified maximum sequence length for this model (1134 > 512). Running this sequence through the model will result in indexing errors\n", "2026-06-22 14:36:33,499 INFO Text chunks : 5\n", "2026-06-22 14:36:33,499 INFO Table chunks : 14\n", "2026-06-22 14:36:33,500 INFO Saved → 8-K_2026-04-30_chunks.json (33.2 KB) [5 text + 14 table = 19 total]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Filing Chunks\n", "-------------------------------------------------------\n", " 10-K_2023.json 205\n", " 10-K_2024.json 217\n", " 10-K_2025.json 222\n", " 10-Q_2024_Q3.json 79\n", " 10-Q_2025_Q1.json 86\n", " 10-Q_2025_Q2.json 90\n", " 10-Q_2025_Q3.json 84\n", " 10-Q_2026_Q1.json 81\n", " 10-Q_2026_Q2.json 103\n", " 8-K_2026-01-02.json 5\n", " 8-K_2026-01-29.json 19\n", " 8-K_2026-02-24.json 14\n", " 8-K_2026-04-20.json 7\n", " 8-K_2026-04-30.json 19\n", "\n", " Total SEC chunks : 1231\n", "\n", "All chunk files (16):\n", " morningstar/a-wide-moat-focus-provides-differentiation_chunks.json (22 text + 10 table = 32 chunks)\n", " morningstar/ptc01302411420_chunks.json (52 text + 12 table = 64 chunks)\n", " sec_filings/AAPL/10-K_2023_chunks.json (139 text + 66 table = 205 chunks)\n", " sec_filings/AAPL/10-K_2024_chunks.json (154 text + 63 table = 217 chunks)\n", " sec_filings/AAPL/10-K_2025_chunks.json (160 text + 62 table = 222 chunks)\n", " sec_filings/AAPL/10-Q_2024_Q3_chunks.json (40 text + 39 table = 79 chunks)\n", " sec_filings/AAPL/10-Q_2025_Q1_chunks.json (49 text + 37 table = 86 chunks)\n", " sec_filings/AAPL/10-Q_2025_Q2_chunks.json (53 text + 37 table = 90 chunks)\n", " sec_filings/AAPL/10-Q_2025_Q3_chunks.json (46 text + 38 table = 84 chunks)\n", " sec_filings/AAPL/10-Q_2026_Q1_chunks.json (43 text + 38 table = 81 chunks)\n", " sec_filings/AAPL/10-Q_2026_Q2_chunks.json (64 text + 39 table = 103 chunks)\n", " sec_filings/AAPL/8-K_2026-01-02_chunks.json (3 text + 2 table = 5 chunks)\n", " sec_filings/AAPL/8-K_2026-01-29_chunks.json (5 text + 14 table = 19 chunks)\n", " sec_filings/AAPL/8-K_2026-02-24_chunks.json (6 text + 8 table = 14 chunks)\n", " sec_filings/AAPL/8-K_2026-04-20_chunks.json (5 text + 2 table = 7 chunks)\n", " sec_filings/AAPL/8-K_2026-04-30_chunks.json (5 text + 14 table = 19 chunks)\n" ] } ], "source": [ "from chunker import DocumentChunker\n", "\n", "SEC_PROCESSED_DIR = PROJECT_ROOT / \"data\" / \"processed\" / \"sec_filings\"\n", "SEC_CHUNKS_DIR = CHUNKS_DIR / \"sec_filings\"\n", "\n", "# Verify _docling.json files exist before chunking\n", "sec_jsons = sorted(f for f in SEC_PROCESSED_DIR.rglob(\"*.json\")\n", " if not f.name.endswith(\"_docling.json\"))\n", "sec_docling = sorted(SEC_PROCESSED_DIR.rglob(\"*_docling.json\"))\n", "\n", "print(f\"SEC processed JSONs : {len(sec_jsons)}\")\n", "print(f\"SEC _docling.json : {len(sec_docling)}\")\n", "\n", "if len(sec_docling) == 0:\n", " print(\"\\nNo _docling.json files found.\")\n", " print(\"Run sec_processor.ipynb first, then re-run this cell.\")\n", "else:\n", " missing = [j for j in sec_jsons\n", " if not j.with_name(j.stem + \"_docling.json\").exists()]\n", " if missing:\n", " print(f\"\\nWARNING: {len(missing)} file(s) missing _docling.json:\")\n", " for m in missing:\n", " print(f\" {m.name}\")\n", " else:\n", " print(f\"\\nAll {len(sec_jsons)} SEC filings have _docling.json — ready to chunk.\")\n", "\n", " print(\"\\nChunking SEC filings...\")\n", " chunker = DocumentChunker(chunks_dir=CHUNKS_DIR)\n", " summary = chunker.chunk_all(processed_dir=SEC_PROCESSED_DIR, force=False)\n", "\n", " print(f\"\\n{'Filing':45s} {'Chunks':>6}\")\n", " print(\"-\" * 55)\n", " total_sec_chunks = 0\n", " for fname, n in summary.items():\n", " print(f\" {fname:43s} {n:>6}\")\n", " total_sec_chunks += n\n", " print(f\"\\n Total SEC chunks : {total_sec_chunks}\")\n", "\n", " # Show all chunk files now available\n", " all_chunk_files = sorted(CHUNKS_DIR.rglob(\"*_chunks.json\"))\n", " print(f\"\\nAll chunk files ({len(all_chunk_files)}):\")\n", " for cf in all_chunk_files:\n", " with open(cf) as f:\n", " d = json.load(f)\n", " print(f\" {cf.relative_to(CHUNKS_DIR)} \"\n", " f\"({d['text_chunks']} text + {d['table_chunks']} table = {d['total_chunks']} chunks)\")" ] }, { "cell_type": "markdown", "id": "7p9y6on7weu", "metadata": {}, "source": [ "## STEP 10 — Embed SEC Chunks into ChromaDB\n", "\n", "`DocumentEmbedder.embed_all()` scans the entire `data/chunks/` tree, so it picks up\n", "SEC chunk files automatically alongside the Morningstar ones.\n", "\n", "Upsert semantics mean this is **always safe to re-run** — chunks already in the store\n", "are updated in-place, never duplicated." ] }, { "cell_type": "code", "execution_count": 12, "id": "rs1it18cko", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2026-06-22 14:40:22,885 INFO Found 16 chunk files under /home/pushkardeshpand/Documents/Morningstar RAG Pipeline/data/chunks\n", "2026-06-22 14:40:22,887 INFO SKIP a-wide-moat-focus-provides-differentiation_chunks.json (all 32 chunks already in store)\n", "2026-06-22 14:40:22,889 INFO SKIP ptc01302411420_chunks.json (all 64 chunks already in store)\n", "2026-06-22 14:40:22,894 INFO SKIP 10-K_2023_chunks.json (all 205 chunks already in store)\n", "2026-06-22 14:40:22,898 INFO SKIP 10-K_2024_chunks.json (all 217 chunks already in store)\n", "2026-06-22 14:40:22,902 INFO SKIP 10-K_2025_chunks.json (all 222 chunks already in store)\n", "2026-06-22 14:40:22,905 INFO SKIP 10-Q_2024_Q3_chunks.json (all 79 chunks already in store)\n", "2026-06-22 14:40:22,908 INFO SKIP 10-Q_2025_Q1_chunks.json (all 86 chunks already in store)\n", "2026-06-22 14:40:22,912 INFO SKIP 10-Q_2025_Q2_chunks.json (all 90 chunks already in store)\n", "2026-06-22 14:40:22,916 INFO SKIP 10-Q_2025_Q3_chunks.json (all 84 chunks already in store)\n", "2026-06-22 14:40:22,919 INFO SKIP 10-Q_2026_Q1_chunks.json (all 81 chunks already in store)\n", "2026-06-22 14:40:22,922 INFO SKIP 10-Q_2026_Q2_chunks.json (all 103 chunks already in store)\n", "2026-06-22 14:40:22,923 INFO SKIP 8-K_2026-01-02_chunks.json (all 5 chunks already in store)\n", "2026-06-22 14:40:22,924 INFO SKIP 8-K_2026-01-29_chunks.json (all 19 chunks already in store)\n", "2026-06-22 14:40:22,925 INFO SKIP 8-K_2026-02-24_chunks.json (all 14 chunks already in store)\n", "2026-06-22 14:40:22,926 INFO SKIP 8-K_2026-04-20_chunks.json (all 7 chunks already in store)\n", "2026-06-22 14:40:22,928 INFO SKIP 8-K_2026-04-30_chunks.json (all 19 chunks already in store)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Embedding all chunk files (skipping already-stored chunks)...\n", "\n", "============================================================\n", "Embedding complete (0.0s)\n", "============================================================\n", "\n", " File Result\n", "---------------------------------------------------------------------------\n", " a-wide-moat-focus-provides-differentiation_chunks.json skipped (already stored)\n", " ptc01302411420_chunks.json skipped (already stored)\n", " 10-K_2023_chunks.json skipped (already stored)\n", " 10-K_2024_chunks.json skipped (already stored)\n", " 10-K_2025_chunks.json skipped (already stored)\n", " 10-Q_2024_Q3_chunks.json skipped (already stored)\n", " 10-Q_2025_Q1_chunks.json skipped (already stored)\n", " 10-Q_2025_Q2_chunks.json skipped (already stored)\n", " 10-Q_2025_Q3_chunks.json skipped (already stored)\n", " 10-Q_2026_Q1_chunks.json skipped (already stored)\n", " 10-Q_2026_Q2_chunks.json skipped (already stored)\n", " 8-K_2026-01-02_chunks.json skipped (already stored)\n", " 8-K_2026-01-29_chunks.json skipped (already stored)\n", " 8-K_2026-02-24_chunks.json skipped (already stored)\n", " 8-K_2026-04-20_chunks.json skipped (already stored)\n", " 8-K_2026-04-30_chunks.json skipped (already stored)\n", "\n", "============================================================\n", "Collection: 'financial_docs'\n", "============================================================\n", " Total vectors : 1327\n", " Sources : ['morningstar', 'sec_edgar']\n", " Documents : 16 ['10-K_2023', '10-K_2024', '10-K_2025', '10-Q_2024_Q3', '10-Q_2025_Q1', '10-Q_2025_Q2', '10-Q_2025_Q3', '10-Q_2026_Q1', '10-Q_2026_Q2', '8-K_2026-01-02', '8-K_2026-01-29', '8-K_2026-02-24', '8-K_2026-04-20', '8-K_2026-04-30', 'a-wide-moat-focus-provides-differentiation', 'ptc01302411420']\n", " Chunk types : ['table', 'text']\n", "\n", "Breakdown by source:\n", " morningstar 96 vectors\n", " sec_edgar 1231 vectors\n", "\n", "Breakdown by doc_type:\n", " 10-K 644 vectors\n", " 10-Q 523 vectors\n", " 8-K 64 vectors\n", " research_report 96 vectors\n", "\n", "Breakdown by chunk_type:\n", " table 481 vectors\n", " text 846 vectors\n" ] } ], "source": [ "import time\n", "\n", "# Re-use the embedder already initialised above\n", "# embed_all() scans CHUNKS_DIR recursively — picks up both morningstar/ and sec_filings/\n", "print(\"Embedding all chunk files (skipping already-stored chunks)...\")\n", "start = time.time()\n", "summary = embedder.embed_all(chunks_dir=CHUNKS_DIR, force=False)\n", "elapsed = time.time() - start\n", "\n", "print(f\"\\n{'='*60}\")\n", "print(f\"Embedding complete ({elapsed:.1f}s)\")\n", "print(f\"{'='*60}\")\n", "print(f\"\\n {'File':55s} {'Result'}\")\n", "print(\"-\" * 75)\n", "for fname, n in summary.items():\n", " if n > 0:\n", " status = f\"{n} new vectors upserted\"\n", " elif n == 0:\n", " status = \"skipped (already stored)\"\n", " else:\n", " status = \"FAILED\"\n", " print(f\" {fname:55s} {status}\")\n", "\n", "# Collection-wide stats\n", "stats = embedder.get_stats()\n", "print(f\"\\n{'='*60}\")\n", "print(f\"Collection: '{COLLECTION_NAME}'\")\n", "print(f\"{'='*60}\")\n", "print(f\" Total vectors : {stats['total_vectors']}\")\n", "print(f\" Sources : {stats['sources']}\")\n", "print(f\" Documents : {len(stats['doc_ids'])} {stats['doc_ids']}\")\n", "print(f\" Chunk types : {stats['chunk_types']}\")\n", "\n", "# Breakdown by source\n", "all_meta = collection.get(include=[\"metadatas\"])[\"metadatas\"]\n", "from collections import Counter\n", "source_counts = Counter(m.get(\"source\", \"?\") for m in all_meta)\n", "doctype_counts = Counter(m.get(\"doc_type\", \"?\") for m in all_meta)\n", "type_counts = Counter(m.get(\"chunk_type\", \"?\") for m in all_meta)\n", "\n", "print(f\"\\nBreakdown by source:\")\n", "for src, n in sorted(source_counts.items()):\n", " print(f\" {src:20s} {n:>5} vectors\")\n", "\n", "print(f\"\\nBreakdown by doc_type:\")\n", "for dt, n in sorted(doctype_counts.items()):\n", " print(f\" {dt:20s} {n:>5} vectors\")\n", "\n", "print(f\"\\nBreakdown by chunk_type:\")\n", "for ct, n in sorted(type_counts.items()):\n", " print(f\" {ct:20s} {n:>5} vectors\")" ] }, { "cell_type": "markdown", "id": "4rl2p01fvil", "metadata": {}, "source": [ "## STEP 11 — Verify SEC-Specific Retrieval\n", "\n", "With SEC filings in the store we can test four retrieval patterns:\n", "\n", "1. **Unfiltered** — returns best match across all documents (Morningstar + SEC)\n", "2. **Source filter** — restrict to `sec_edgar` only\n", "3. **Doc type + fiscal year** — retrieve from a specific Apple 10-K\n", "4. **Table filter** — retrieve financial statement tables from SEC filings\n", "\n", "These patterns show the full power of storing metadata alongside vectors —\n", "the same collection serves completely different retrieval strategies." ] }, { "cell_type": "code", "execution_count": 13, "id": "0rpjqr69hphl", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "cb045afed4b046559b3186ec45430163", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Batches: 0%| | 0/1 [00:00 max_text else ''}\")\n", "\n", "QUERY_OPTS = dict(n_results=3, include=[\"documents\", \"metadatas\", \"distances\"])\n", "\n", "# ── Query 1: No filter — cross-document ───────────────────────────────────────\n", "r1 = collection.query(query_texts=[\"Apple revenue growth fiscal year 2024\"], **QUERY_OPTS)\n", "show_hits(r1, \"Apple revenue growth [no filter — all sources]\")" ] }, { "cell_type": "code", "execution_count": 14, "id": "qot6taqwhvo", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "6fe1d372b4c947669c5fb5d88f9ee0a0", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Batches: 0%| | 0/1 [00:005}\")\n", "print(f\"\\n By document type:\")\n", "for dt, n in sorted(doctype_counts.items()):\n", " print(f\" {dt:25s} {n:>5}\")\n", "print(f\"\\n By chunk type:\")\n", "for ct, n in sorted(type_counts.items()):\n", " print(f\" {ct:25s} {n:>5}\")\n", "print(f\"\\n Persist path : {VECTORSTORE_DIR}\")\n", "print(f\"\\nReady for Phase 5 — Retrieval chain.\")" ] }, { "cell_type": "code", "execution_count": null, "id": "a312297d-af86-4234-a829-bc9963af9053", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.7" } }, "nbformat": 4, "nbformat_minor": 5 }