VECTOR_DB_EXPLAINED.md

Vector Database — Implementation Guide

A walkthrough of what we built, how it works, and the core concepts you need to understand.

---

Table of Contents

1. Project Overview
2. File Structure
3. Core Concept: What is an Embedding?
4. Core Concept: What is a Vector Database?
5. Core Concept: Cosine Similarity
6. Code Walkthrough: vector_db.py
7. Code Walkthrough: demo.py
8. The Data File: data.json
9. Reading the Search Results
10. How This Scales to Production
11. Glossary

---

1. Project Overview

We built a minimal vector database from scratch using only:

Component	What it does
sentence-transformers	Converts text into a list of numbers (an embedding)
numpy	Efficient math on those number lists
Python json	Persists the data to disk
Our own code	The search logic (cosine similarity ranking)

The full pipeline in one diagram:

Your Text
    |
    v
[ Embedding Model ]   <-- all-MiniLM-L6-v2 (runs locally on your machine)
    |
    v
[0.12, -0.44, 0.87, ..., 0.03]   <-- 384 numbers representing the meaning
    |
    v
[ data.json ]   <-- stored alongside your original text and metadata
    |
    v
[ Cosine Similarity ]   <-- at search time, compare query vector vs all stored vectors
    |
    v
Ranked Results (most similar first)

---

2. File Structure

ai/
├── vector_db.py            # The VectorDB class (embed, store, search)
├── demo.py                 # Demo: populates the DB and runs queries
├── data.json               # Generated file: stores all records + embeddings
└── VECTOR_DB_EXPLAINED.md  # This file

---

3. Core Concept: What is an Embedding?

This is the most important concept in the entire project.

An embedding is a list of numbers that represents the meaning of a piece of text.

The key insight: texts with similar meanings get similar numbers.

Example

"Machine learning models learn patterns from data"
    --> [0.12, -0.44, 0.87, 0.03, ...]   (384 numbers)

"AI systems improve by training on examples"
    --> [0.11, -0.41, 0.89, 0.04, ...]   (384 numbers, very close!)

"Pizza originated in Naples"
    --> [-0.55, 0.72, -0.10, 0.91, ...]  (384 numbers, very different)

The two ML sentences land close together in "meaning space" even though they share no words in common. This is what makes semantic search powerful.

How the model creates embeddings

The all-MiniLM-L6-v2 model is a neural network trained on hundreds of millions of sentence pairs. During training, it learned to push similar sentences close together and push unrelated sentences apart.

When you call model.encode("some text"), the model passes your text through its layers and returns that final 384-number vector. You do not need to understand the neural network internals — just know the output is a compact numerical "fingerprint" of the meaning.

Why 384 dimensions?

all-MiniLM-L6-v2 produces 384-dimensional vectors. This is a design choice of the model — larger models like all-mpnet-base-v2 produce 768 dimensions and are more accurate but slower. 384 is the sweet spot for speed vs. quality.

---

4. Core Concept: What is a Vector Database?

A vector database stores embeddings alongside your original data and lets you search by meaning rather than by exact keywords.

Traditional database search (keyword)

SELECT * FROM docs WHERE text LIKE '%machine learning%'

This only finds documents that contain those exact words. It would miss:

• "AI training on data"
• "neural network optimization"
• "computers that learn"

Vector database search (semantic)

Query: "how do computers learn from data?"
  --> embed the query --> [0.12, -0.39, ...]
  --> compare against every stored vector
  --> return the closest ones

This finds documents about machine learning, AI, and training — even if they use completely different words.

What makes it a "database"

A vector database needs to do three things:

1. Store — keep the embeddings (and the original data) somewhere persistent
2. Index — organize embeddings so searching is fast (our JSON version skips this; production DBs use HNSW trees)
3. Query — given a new vector, find the most similar stored vectors quickly

---

5. Core Concept: Cosine Similarity

This is how we measure "how similar" two vectors are.

The intuition

Think of each embedding vector as an arrow pointing in some direction in 384-dimensional space. Two sentences with similar meanings point in nearly the same direction. Cosine similarity measures the angle between those arrows.

Similar meaning  --> small angle  --> cosine close to 1.0
Unrelated        --> large angle  --> cosine close to 0.0
Opposite meaning --> 180 degrees  --> cosine close to -1.0

The formula

cosine_similarity(A, B) = (A · B) / (|A| × |B|)

Where:

• A · B is the dot product: multiply each pair of numbers and sum them up
• |A| is the magnitude of vector A: square root of the sum of squares

Our implementation

@staticmethod
def _cosine_similarity(a: list[float], b: list[float]) -> float:
    dot   = sum(x * y for x, y in zip(a, b))   # dot product
    mag_a = math.sqrt(sum(x * x for x in a))   # |A|
    mag_b = math.sqrt(sum(x * x for x in b))   # |B|
    return dot / (mag_a * mag_b)

Why cosine and not Euclidean distance?

Euclidean distance measures how far apart two points are. Cosine similarity measures the angle between them. For text embeddings, the angle matters more than the raw distance — a short sentence and a long sentence about the same topic will have different magnitudes but the same direction.

Score interpretation from our demo output

#1  score=0.6257  ############    <- strongly related
#2  score=0.2572  #####           <- loosely related
#3  score=0.2546  #####           <- loosely related

Scores above 0.5 are typically strong matches. Below 0.2 is often noise. The # bar gives you a quick visual feel.

---

6. Code Walkthrough: vector_db.py

class VectorDB:
    def __init__(self, db_path: str = "data.json", model_name: str = "all-MiniLM-L6-v2"):
        self.db_path = db_path
        self._model_name = model_name
        self._model = None          # loaded lazily so imports stay fast
        self._records: list[dict] = []
        if os.path.exists(db_path):
            self.load()

Why self._model = None (lazy loading)? Loading the model takes ~1 second and ~80MB of RAM. By setting it to None initially and only loading it the first time _get_model() is called, the class is cheap to create even if you only want to read the JSON file.

---

def _get_model(self):
    if self._model is None:
        from sentence_transformers import SentenceTransformer
        self._model = SentenceTransformer(self._model_name)
    return self._model

Why import inside the function? The from sentence_transformers import ... line is placed inside the method (not at the top of the file). This means the slow import only happens when you actually need to embed something — not when the module is first loaded.

---

def add(self, text: str, metadata: dict[str, Any] | None = None) -> None:
    model = self._get_model()
    embedding: list[float] = model.encode(text).tolist()
    record = {
        "id": len(self._records),
        "text": text,
        "metadata": metadata or {},
        "embedding": embedding,       # <-- the 384 numbers
    }
    self._records.append(record)

What model.encode(text) returns: A NumPy array of shape (384,). We call .tolist() to convert it to a plain Python list so it can be serialized to JSON. NumPy arrays cannot be written to JSON directly.

What metadata is for: Any extra information you want to store alongside the text — category, source URL, date, author, etc. It is stored in JSON and returned with search results so you can use it in your application.

---

def search(self, query: str, top_k: int = 5) -> list[dict]:
    query_vec: list[float] = model.encode(query).tolist()

    scored = [
        {
            "score": self._cosine_similarity(query_vec, rec["embedding"]),
            ...
        }
        for rec in self._records
    ]

    scored.sort(key=lambda r: r["score"], reverse=True)
    return scored[:top_k]

The brute-force search: We compute cosine similarity between the query and every single record — this is called a linear scan or flat search. It is simple and exact, but slow at scale (O(n) comparisons). Production vector DBs use approximate nearest-neighbor indexes (like HNSW) to do this in O(log n).

reverse=True: We sort highest score first because cosine similarity of 1.0 means identical. We want the best match at the top.

---

7. Code Walkthrough: demo.py

DOCUMENTS = [
    ("Python is a high-level programming language...", {"category": "tech", "topic": "programming"}),
    ("Machine learning models learn patterns...",      {"category": "tech", "topic": "AI/ML"}),
    ...
]

Each entry is a tuple of (text, metadata). The categories are deliberately diverse so that the semantic search has to cross category boundaries — proving it works by meaning, not by keyword matching.

---

def build_db():
    db = VectorDB(db_path="data.json")
    for text, meta in DOCUMENTS:
        db.add(text, meta)
    db.save()
    return db

Why db.save() at the end and not inside db.add()? Writing to disk on every single add() call would be slow — we batch all the embeddings in memory first, then flush to disk once. This is the same reason databases have write buffers.

---

queries = [
    "how do computers learn from data?",      # should find AI/ML
    "space and the universe",                  # should find astronomy
    "food from Italy",                         # should find Italian cuisine
    "athletic competition between nations",    # should find Olympics
    "storing and querying high-dimensional data",  # should find vector DB
]

Notice that none of these queries share words with their target documents. The match is purely semantic. This is the whole point of vector search.

---

8. The Data File: data.json

Open data.json after running the demo. Each record looks like this:

{
  "id": 1,
  "text": "Machine learning models learn patterns from data without being explicitly programmed.",
  "metadata": {
    "category": "tech",
    "topic": "AI/ML"
  },
  "embedding": [
    0.04859826713800430,
    -0.04706305265426636,
    0.07049278914928436,
    ...                    // 381 more numbers
    0.02341187000274658
  ]
}

Key observations:

• The embedding has exactly 384 numbers for every record, regardless of how long or short the text is.
• The numbers are between roughly -1 and 1 (the model normalizes them).
• The raw text and metadata are stored alongside the vector — this is how you get the original content back after finding a match.
• The file for 16 documents is about 430 KB. Most of that is the embedding numbers. At 10,000 documents it would be ~270 MB — this is why production systems use binary formats (like .npy or Parquet) instead of JSON.

---

9. Reading the Search Results

Here is the actual output from our demo, annotated:

Query: "how do computers learn from data?"
----------------------------------------------------------------
#1  score=0.6257  ############
     [tech / AI/ML]
     Machine learning models learn patterns from data without being explicitly programmed.

#2  score=0.2572  #####
     [tech / programming]
     Python is a high-level programming language...

#3  score=0.2546  #####
     [tech / database]
     Vector databases store high-dimensional embeddings...

• #1 (0.63) — Near-perfect match. The model understood that "computers learning from data" = "machine learning models learning patterns from data."
• #2 and #3 (0.25) — Loosely related (both are tech topics) but not real matches.
• The gap between #1 and #2 (0.63 vs 0.25) is the model confidently separating the right answer from everything else.

Query: "storing and querying high-dimensional data"
----------------------------------------------------------------
#1  score=0.6023  ############
     [tech / database]
     Vector databases store high-dimensional embeddings and support similarity search.

This one is almost cheating — the document literally contains the phrase "high-dimensional embeddings." But notice the score is still only 0.60, not 1.0, because the phrasing is different.

---

10. How This Scales to Production

Our implementation works for hundreds of documents. Here is what production vector databases add:

Problem	Our solution	Production solution
Search speed	Linear scan — compare every record	HNSW or IVF index — approximate nearest neighbor in O(log n)
Storage	JSON text — large and slow to parse	Binary formats (.npy, Parquet, mmap)
Persistence	Single file	Distributed storage with replication
Updates	Reload entire file	Incremental inserts and deletes
Scale	~1,000 docs before it feels slow	Pinecone, Weaviate, Qdrant handle billions
Filtering	None	Filter by metadata before or after vector search

Popular production vector databases:

• Pinecone — fully managed cloud service, easiest to start with
• Weaviate — open source, supports hybrid (keyword + vector) search
• Qdrant — open source, written in Rust, very fast
• pgvector — adds vector search to regular PostgreSQL

The concepts are identical to what we built. They just replace the JSON file and the linear scan with faster infrastructure.

---

11. Glossary

Term	Definition
Embedding	A list of numbers that represents the meaning of a piece of text
Embedding model	A neural network trained to produce embeddings (we use all-MiniLM-L6-v2)
Vector	The list of numbers itself — 384 floats in our case
Dimension	The length of that list. Our model uses 384 dimensions
Cosine similarity	A score from -1 to 1 measuring how similar two vectors are by their angle
Dot product	A·B — multiply each pair of numbers and sum; part of the cosine formula
Magnitude	The "length" of a vector: sqrt(sum of squares)
Semantic search	Searching by meaning rather than by exact keyword matches
Corpus	The collection of documents stored in your database
top_k	How many results to return (we used 3 in the demo)
Linear scan / flat search	Comparing the query against every single stored vector — exact but slow
HNSW	Hierarchical Navigable Small World — a graph-based index for fast approximate nearest-neighbor search
Lazy loading	Delaying an expensive operation (like loading the model) until it is actually needed
Metadata	Extra data stored alongside the embedding — category, date, URL, etc.
Normalize	Scale a vector so its magnitude equals 1; makes cosine similarity equivalent to dot product