Spaces:

ruidiao
/

QuoteSearch

Running

App Files Files Community

ruidiao commited on Oct 18, 2025

Commit

1e32f6a

1 Parent(s): 785042b

code

Browse files

Files changed (6) hide show

README.md +39 -12
index.html +32 -19
main.js +123 -0
offline_processing.py +122 -0
transformers.min.js +0 -0
worker.js +322 -0

README.md CHANGED Viewed

@@ -1,12 +1,39 @@
----
-title: ClientSideQuoteSearch
-emoji: 🔥
-colorFrom: purple
-colorTo: purple
-sdk: static
-pinned: false
-license: mit
-short_description: 'Client-side AI quote search: fast, private, no servers.'
----
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

+# Client-Side Semantic Quote Retrieval Engine
+This project implements a "Zero-Infrastructure" client-side semantic quote retrieval engine. All computationally intensive tasks, including vectorization, indexing, and model quantization, are handled offline. The search and retrieval operations then occur entirely within the user's browser, ensuring data privacy and minimal operational overhead.
+## Project Overview
+The core idea is to pre-process a large dataset of quotes into a highly optimized, quantized vector index. This index, along with a small machine learning model, is then loaded by the client-side application. When a user enters a query, the application generates an embedding for the query and performs a fast Approximate Nearest Neighbor Search (ANNS) directly in the browser to find semantically similar quotes.
+## Data Source
+The quotes data used in this project was sourced from:
+[https://archive.org/details/quotes_20230625](https://archive.org/details/quotes_20230625)
+## Setup and Usage
+1.  **Offline Data Processing**:
+    *   Ensure you have Python and the necessary libraries (e.g., `pandas`, `numpy`, `torch`, `sentence-transformers`, `tqdm`) installed.
+    *   Run the `offline_processing.py` script to generate the `quotes_index.bin` file. This file contains the pre-computed embeddings and metadata.
+    ```bash
+    python offline_processing.py
+    ```
+    *   *Note: This step can be time-consuming for large datasets, especially the first time as the embedding model needs to be downloaded.*
+    *   **Important:** The script now includes validation to ensure that categories in the CSV do not contain uppercase letters. Rows with invalid categories will be ignored.
+2.  **Client-Side Application**:
+    *   Open `index.html` in your web browser.
+    *   The search input and button are immediately available.
+    *   The application will first check for a cached index. If not found, it will display a message indicating a significant one-time download (for the model and index) which will occur on your first search.
+    *   The `quotes_index.bin` is loaded (from cache or downloaded) and the necessary machine learning model (via `transformers.js`) is downloaded on demand during your first search.
+    *   A progress bar with detailed status will be shown during downloads and processing, disappearing once complete.
+    *   A "Delete Cached Data" button will appear when data is cached, allowing you to clear local storage.
+    *   Enter your search queries to retrieve semantically similar quotes.
+## Technologies Used
+*   **Frontend**: HTML, CSS (Tailwind CSS), JavaScript
+*   **Offline Processing**: Python (pandas, numpy, torch, sentence-transformers)
+*   **Embedding Model**: nomic-ai/nomic-embed-text-v1.5
+*   **Client-Side ML**: transformers.js, Web Workers

index.html CHANGED Viewed

@@ -1,19 +1,32 @@
-<!doctype html>
-<html>
-	<head>
-		<meta charset="utf-8" />
-		<meta name="viewport" content="width=device-width" />
-		<title>My static Space</title>
-		<link rel="stylesheet" href="style.css" />
-	</head>
-	<body>
-		<div class="card">
-			<h1>Welcome to your static Space!</h1>
-			<p>You can modify this app directly by editing <i>index.html</i> in the Files and versions tab.</p>
-			<p>
-				Also don't forget to check the
-				<a href="https://huggingface.co/docs/hub/spaces" target="_blank">Spaces documentation</a>.
-			</p>
-		</div>
-	</body>
-</html>

+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Client-Side Quote Search</title>
+    <script src="https://cdn.tailwindcss.com"></script>
+</head>
+<body class="bg-gray-100 flex items-center justify-center min-h-screen">
+            <div class="container mx-auto p-8 bg-white rounded-lg shadow-lg w-full max-w-2xl relative">
+            <div id="delete-button-container" class="flex justify-end mb-4 hidden">
+                <button id="delete-data-button" class="bg-red-500 text-white px-3 py-1 rounded-lg text-sm hover:bg-red-600 focus:outline-none focus:ring-2 focus:ring-red-500">Delete Cached Data</button>
+            </div>
+            <h1 class="text-3xl font-bold text-center text-gray-800 mb-4">Client-Side Quote Search</h1>
+            <div id="data-info" class="text-center text-gray-600 mb-2"></div>
+            <div id="status" class="text-center text-gray-600 mb-4"></div>
+            <div id="progress-container" class="w-full bg-gray-200 rounded-full h-4 mb-4 hidden">
+                <div id="progress-bar" class="bg-blue-500 h-4 rounded-full" style="width: 0%"></div>
+            </div>
+            <div class="flex gap-2 mb-4">
+                <textarea id="search-input" class="w-full p-2 border border-gray-300 rounded-lg focus:ring-2 focus:ring-blue-500 focus:outline-none" placeholder="Enter your query..."></textarea>
+                <button id="search-button" class="bg-blue-500 text-white px-4 py-2 rounded-lg hover:bg-blue-600 focus:outline-none focus:ring-2 focus:ring-blue-500">Search</button>
+            </div>
+        <div id="results"></div>
+    </div>
+    <script src="main.js"></script>
+</body>
+</html>

main.js ADDED Viewed

	@@ -0,0 +1,123 @@

+const statusDiv = document.getElementById('status');
+const searchInput = document.getElementById('search-input');
+const resultsDiv = document.getElementById('results');
+const searchButton = document.getElementById('search-button');
+const progressContainer = document.getElementById('progress-container');
+const progressBar = document.getElementById('progress-bar');
+const dataInfoDiv = document.getElementById('data-info');
+const deleteDataButton = document.getElementById('delete-data-button');
+const deleteButtonContainer = document.getElementById('delete-button-container');
+const worker = new Worker('worker.js', { type: 'module' });
+// Initial state
+// searchInput.disabled = true; // Removed to enable during initialization
+// searchButton.disabled = true; // Removed to enable during initialization
+deleteButtonContainer.classList.add('hidden');
+// Helper to format bytes to MB
+function formatBytesToMB(bytes) {
+    return (bytes / (1024 * 1024)).toFixed(2);
+}
+worker.onmessage = (event) => {
+    const { type, payload } = event.data;
+    if (type === 'loading') {
+        statusDiv.textContent = payload;
+        progressContainer.classList.remove('hidden');
+    } else if (type === 'progress') {
+        const progressPercentage = Math.round(payload.progress);
+        progressBar.style.width = `${progressPercentage || 0}%`;
+        statusDiv.textContent = payload.detail;
+        progressContainer.classList.remove('hidden');
+    } else if (type === 'ready') {
+        statusDiv.textContent = ''; // Clear status when index is ready
+        progressContainer.classList.add('hidden');
+        searchInput.disabled = false;
+        searchButton.disabled = false;
+        deleteButtonContainer.classList.remove('hidden'); // Ensure button is shown when index is ready
+    } else if (type === 'results') {
+        // statusDiv.textContent = 'Search complete.'; // Removed as results themselves convey completion
+        progressContainer.classList.add('hidden'); // Hide progress bar
+        renderResults(payload);
+        statusDiv.textContent = ''; // Clear status after search is complete
+        deleteButtonContainer.classList.remove('hidden'); // Ensure delete button is visible
+    } else if (type === 'indexSize') {
+        if (payload.indexCached || payload.modelCached) {
+            deleteButtonContainer.classList.remove('hidden'); // Ensure button is shown if index or model is cached
+        }
+        if (payload.indexCached) {
+            dataInfoDiv.textContent = ''; // Clear data info if cached
+            searchInput.disabled = false;
+            searchButton.disabled = false;
+            if (payload.modelCached) {
+                statusDiv.textContent = ''; // Clear status if both index and model are loaded
+            } else {
+                statusDiv.textContent = ''; // Clear status if index is loaded, model will load on first search
+            }
+        } else {
+            dataInfoDiv.textContent = ''; // Clear data info if not cached
+            deleteButtonContainer.classList.add('hidden'); // Hide delete button if not cached
+            searchInput.disabled = false; // Ensure input is enabled
+            searchButton.disabled = false; // Ensure button is enabled
+            statusDiv.textContent = `To enable search, a one-time download of approximately ${formatBytesToMB(payload.size)} MB is required. This will happen when you perform your first search.`;
+        }
+    } else if (type === 'dataDeleted') {
+        statusDiv.textContent = payload;
+        dataInfoDiv.textContent = ''; // Clear data info
+        deleteButtonContainer.classList.add('hidden');
+        // searchInput.disabled = true; // Removed: state will be set by getIndexSize
+        // searchButton.disabled = true; // Removed: state will be set by getIndexSize
+        progressContainer.classList.add('hidden'); // Hide progress bar
+        // Re-request size info after deletion
+        worker.postMessage({ type: 'getIndexSize' });
+    } else if (type === 'error') {
+        statusDiv.textContent = `Error: ${payload}`;
+        progressContainer.classList.add('hidden');
+    }
+};
+// Send initial request for index size when the page loads
+worker.postMessage({ type: 'getIndexSize' });
+searchButton.addEventListener('click', () => {
+    const query = searchInput.value;
+    if (query) {
+        worker.postMessage({ type: 'search', payload: query });
+        statusDiv.textContent = 'Searching...';
+        resultsDiv.innerHTML = ''; // Clear previous results
+    } else {
+        resultsDiv.innerHTML = '<p class="text-gray-500">Please enter a query.</p>';
+    }
+});
+deleteDataButton.addEventListener('click', () => {
+    if (confirm('Are you sure you want to delete the cached data?')) {
+        worker.postMessage({ type: 'deleteData' });
+    }
+});
+function renderResults(results) {
+    resultsDiv.innerHTML = '';
+    if (results.length === 0) {
+        resultsDiv.innerHTML = '<p class="text-gray-500">No results found.</p>';
+        return;
+    }
+    results.forEach(result => {
+        const card = document.createElement('div');
+        card.className = 'bg-gray-50 p-4 rounded-lg shadow mb-4';
+        // Handle null author and category gracefully
+        const authorText = result.author ? `- ${result.author}` : '- Unknown Author';
+        const categoryText = result.category ? `Categories: ${result.category}` : 'Categories: N/A';
+        card.innerHTML = `
+            <p class="text-lg font-medium text-gray-800">"${result.quote}"</p>
+            <p class="text-right text-gray-600 italic mt-2">${authorText}</p>
+            <p class="text-sm text-gray-500 mt-2">${categoryText}</p>
+        `;
+        resultsDiv.appendChild(card);
+    });
+}

offline_processing.py ADDED Viewed

	@@ -0,0 +1,122 @@

+import pandas as pd
+import numpy as np
+import torch
+from sentence_transformers import SentenceTransformer
+import json
+import struct
+# 1. Constants
+MODEL_NAME = 'nomic-ai/nomic-embed-text-v1.5'
+EMBEDDING_DIM = 256
+INPUT_CSV_PATH = 'data/quotes.csv'
+OUTPUT_BINARY_PATH = 'data/quotes_index.bin'
+BATCH_SIZE = 64 # Optimized batch size for GPU processing
+# 2. Load Data
+def load_quotes(file_path):
+    """Loads quotes from a CSV file using pandas, skipping malformed lines."""
+    # Define column names explicitly
+    column_names = ['quote', 'author', 'category']
+    # Read CSV with explicit separator, quote character, and skip bad lines
+    # header=None because we are providing names explicitly
+    df = pd.read_csv(
+        file_path,
+        sep=',',
+        quotechar='"',
+        header=None,
+        names=column_names,
+        on_bad_lines='skip' # Skip lines that pandas cannot parse into 3 columns
+    )
+    # Filter out rows where the category contains uppercase letters
+    initial_rows = len(df)
+    df = df[df['category'].apply(lambda x: isinstance(x, str) and not any(c.isupper() for c in x))]
+    filtered_rows = len(df)
+    if initial_rows - filtered_rows > 0:
+        print(f"Ignored {initial_rows - filtered_rows} rows due to uppercase letters in category.")
+    # Prepend the required prefix for retrieval tasks
+    df['quote_for_embedding'] = "search_query: " + df['quote']
+    return df.to_dict('records')
+# 3. Generate Embeddings
+def generate_embeddings(quotes, model_name, embedding_dim):
+    """Generates and truncates embeddings for a list of quotes."""
+    model = SentenceTransformer(model_name, trust_remote_code=True)
+    # The model automatically uses the GPU if available
+    embeddings = model.encode(
+        [q['quote_for_embedding'] for q in quotes],
+        convert_to_tensor=True,
+        batch_size=BATCH_SIZE,
+        show_progress_bar=True # Display progress bar for embedding generation
+    )
+    # Truncate embeddings to the desired dimension
+    truncated_embeddings = embeddings[:, :embedding_dim]
+    return truncated_embeddings.cpu().numpy()
+# 4. Quantize Embeddings
+def quantize_embeddings(embeddings):
+    """Quantizes float32 embeddings to int8."""
+    # Calculate the scale factor
+    abs_max = np.abs(embeddings).max()
+    scale = 127.0 / abs_max if abs_max != 0 else 0
+    # Quantize and clip
+    quantized_embeddings = np.clip(embeddings * scale, -127, 127).astype(np.int8)
+    return quantized_embeddings, scale
+def main():
+    """Main function to run the offline processing pipeline."""
+    print("Starting offline processing...")
+    # Load quotes
+    quotes = load_quotes(INPUT_CSV_PATH)
+    print(f"Loaded {len(quotes)} quotes.")
+    # Generate embeddings
+    print("Generating embeddings...")
+    float_embeddings = generate_embeddings(quotes, MODEL_NAME, EMBEDDING_DIM)
+    print(f"Generated float embeddings with shape: {float_embeddings.shape}")
+    # Quantize embeddings
+    print("Quantizing embeddings...")
+    quantized_embeddings, scale = quantize_embeddings(float_embeddings)
+    print(f"Quantized embeddings with shape: {quantized_embeddings.shape}")
+    print(f"Quantization scale factor: {scale}")
+    # Prepare metadata
+    metadata = [
+        {"quote": q["quote"], "author": q["author"], "category": q["category"]}
+        for q in quotes
+    ]
+    # Replace NaN values with None for JSON compatibility
+    for item in metadata:
+        for key, value in item.items():
+            if isinstance(value, float) and np.isnan(value):
+                item[key] = None
+    metadata_json = json.dumps(metadata, separators=(",", ":"))
+    metadata_bytes = metadata_json.encode('utf-8')
+    # Pack data into a binary file
+    print("Packaging data into binary file...")
+    with open(OUTPUT_BINARY_PATH, 'wb') as f:
+        # Header
+        f.write(struct.pack('<I', len(quotes)))  # 4 bytes: number of quotes
+        f.write(struct.pack('<H', EMBEDDING_DIM)) # 2 bytes: embedding dimension
+        f.write(struct.pack('<f', scale)) # 4 bytes: quantization scale factor
+        f.write(struct.pack('<I', len(metadata_bytes))) # 4 bytes: metadata size
+        # Metadata
+        f.write(metadata_bytes)
+        # Embeddings
+        f.write(quantized_embeddings.tobytes())
+    print(f"Offline processing complete. Index file saved to {OUTPUT_BINARY_PATH}")
+if __name__ == "__main__":
+    main()

transformers.min.js ADDED Viewed

The diff for this file is too large to render. See raw diff

worker.js ADDED Viewed

	@@ -0,0 +1,322 @@

+// 1. Import transformers.js
+import { pipeline, env } from './transformers.min.js';
+// 2. Constants
+env.allowLocalModels = true;
+const MODEL_NAME = 'nomic-ai/nomic-embed-text-v1.5';
+const EMBEDDING_DIM = 256;
+const INDEX_PATH = 'data/quotes_index.bin';
+const APPROX_MODEL_SIZE_MB = 180; // Actual observed size of nomic-embed-text-v1.5 model
+const DB_NAME = 'QuoteSearchDB';
+const DB_VERSION = 1;
+const STORE_NAME = 'quoteIndex';
+let model;
+let indexData;
+let isReady = false;
+// IndexedDB Helper Functions
+function openDB() {
+    return new Promise((resolve, reject) => {
+        const request = indexedDB.open(DB_NAME, DB_VERSION);
+        request.onupgradeneeded = (event) => {
+            const db = event.target.result;
+            db.createObjectStore(STORE_NAME, { keyPath: 'id' });
+        };
+        request.onsuccess = (event) => {
+            resolve(event.target.result);
+        };
+        request.onerror = (event) => {
+            reject('IndexedDB error: ' + event.target.errorCode);
+        };
+    });
+}
+async function getFromDB(key) {
+    const db = await openDB();
+    return new Promise((resolve, reject) => {
+        const transaction = db.transaction([STORE_NAME], 'readonly');
+        const store = transaction.objectStore(STORE_NAME);
+        const request = store.get(key);
+        request.onsuccess = () => {
+            resolve(request.result ? request.result.value : null);
+        };
+        request.onerror = () => {
+            reject('Error getting data from DB');
+        };
+    });
+}
+async function putInDB(key, value) {
+    const db = await openDB();
+    return new Promise((resolve, reject) => {
+        const transaction = db.transaction([STORE_NAME], 'readwrite');
+        const store = transaction.objectStore(STORE_NAME);
+        const request = store.put({ id: key, value: value });
+        request.onsuccess = () => {
+            resolve();
+        };
+        request.onerror = () => {
+            reject('Error putting data in DB');
+        };
+    });
+}
+async function deleteFromDB(key) {
+    const db = await openDB();
+    return new Promise((resolve, reject) => {
+        const transaction = db.transaction([STORE_NAME], 'readwrite');
+        const store = transaction.objectStore(STORE_NAME);
+        const request = store.delete(key);
+        request.onsuccess = () => {
+            resolve();
+        };
+        request.onerror = () => {
+            reject('Error deleting data from DB');
+        };
+    });
+}
+// 3. Load Model and Index
+async function loadModel() {
+    try {
+        self.postMessage({ type: 'loading', payload: 'Loading model...' });
+        // Load the model with a progress callback
+        model = await pipeline('feature-extraction', MODEL_NAME, {
+            progress_callback: (progress) => {
+                const detailMessage = `Model ${progress.status}: ${progress.file || ''} ${Math.floor(progress.progress || 0)}%`;
+                self.postMessage({ type: 'progress', payload: { ...progress, detail: detailMessage } });
+            }
+        });
+    } catch (error) {
+        console.error('Error loading model:', error);
+        self.postMessage({ type: 'error', payload: error.message });
+        throw error; // Re-throw to prevent further execution if model fails to load
+    }
+}
+async function loadIndex() {
+    try {
+        self.postMessage({ type: 'loading', payload: 'Checking cached index...' });
+        const cachedIndex = await getFromDB('quoteIndexData');
+        if (cachedIndex) {
+            indexData = cachedIndex;
+            self.postMessage({ type: 'loading', payload: 'Index loaded from cache.' });
+        } else {
+            self.postMessage({ type: 'loading', payload: 'Loading index...' });
+            // Fetch and parse the index file with progress reporting
+            const response = await fetch(INDEX_PATH);
+            const contentLength = response.headers.get('Content-Length');
+            const total = parseInt(contentLength, 10);
+            let loaded = 0;
+            const reader = response.body.getReader();
+            const chunks = [];
+            while (true) {
+                const { done, value } = await reader.read();
+                if (done) {
+                    break;
+                }
+                chunks.push(value);
+                loaded += value.length;
+                const progress = {
+                    status: 'Downloading Index',
+                    progress: (loaded / total) * 100,
+                    file: INDEX_PATH,
+                    detail: `Downloading index: ${Math.floor((loaded / total) * 100)}% (${(loaded / (1024 * 1024)).toFixed(2)}MB / ${(total / (1024 * 1024)).toFixed(2)}MB)`
+                };
+                self.postMessage({ type: 'progress', payload: progress });
+            }
+            const buffer = await new Response(new Blob(chunks)).arrayBuffer();
+            // Parse the binary file
+            let offset = 0;
+            const numQuotes = new Uint32Array(buffer.slice(offset, offset + 4))[0];
+            offset += 4;
+            const embeddingDim = new Uint16Array(buffer.slice(offset, offset + 2))[0];
+            offset += 2;
+            const scale = new Float32Array(buffer.slice(offset, offset + 4))[0];
+            offset += 4;
+            const metadataSize = new Uint32Array(buffer.slice(offset, offset + 4))[0];
+            offset += 4;
+            const metadataBytes = buffer.slice(offset, offset + metadataSize);
+            offset += metadataSize;
+            const metadata = JSON.parse(new TextDecoder().decode(metadataBytes));
+            const quantizedEmbeddings = new Int8Array(buffer.slice(offset));
+            // De-quantize embeddings with progress reporting
+            const embeddings = new Float32Array(quantizedEmbeddings.length);
+            const totalEmbeddings = quantizedEmbeddings.length;
+            const updateInterval = Math.floor(totalEmbeddings / 100); // Update every 1%
+            for (let i = 0; i < totalEmbeddings; i++) {
+                embeddings[i] = quantizedEmbeddings[i] / scale;
+                if (updateInterval > 0 && i % updateInterval === 0) {
+                    const progress = {
+                        status: 'De-quantizing',
+                        progress: (i / totalEmbeddings) * 100,
+                        file: INDEX_PATH,
+                        detail: `De-quantizing embeddings: ${Math.floor((i / totalEmbeddings) * 100)}%`
+                    };
+                    self.postMessage({ type: 'progress', payload: progress });
+                }
+            }
+            indexData = {
+                metadata,
+                embeddings: reshape(embeddings, [numQuotes, embeddingDim]),
+                embeddingsByteLength: quantizedEmbeddings.byteLength // Store byteLength
+            };
+            await putInDB('quoteIndexData', indexData); // Store in IndexedDB
+        }
+    } catch (error) {
+        console.error('Error loading index:', error);
+        self.postMessage({ type: 'error', payload: error.message });
+        throw error;
+    }
+}
+async function load() {
+    try {
+        await loadIndex();
+        isReady = true;
+        self.postMessage({ type: 'ready' });
+    } catch (error) {
+        console.error('Error in worker load function:', error);
+        self.postMessage({ type: 'error', payload: error.message });
+    }
+}
+// 4. Listen for Messages
+self.onmessage = async (event) => {
+    const { type, payload } = event.data;
+    if (type === 'search') {
+        if (!indexData) {
+            self.postMessage({ type: 'loading', payload: 'Loading index before search...' });
+            await loadIndex();
+        }
+        if (!model) {
+            self.postMessage({ type: 'loading', payload: 'Loading model before search...' });
+            await loadModel();
+        }
+        self.postMessage({ type: 'loading', payload: 'Searching...' });
+        const results = await search(payload);
+        self.postMessage({ type: 'results', payload: results });
+    } else if (type === 'deleteData') {
+        self.postMessage({ type: 'loading', payload: 'Deleting cached data...' });
+        try {
+            await deleteFromDB('quoteIndexData');
+            // Attempt to clear transformers.js model cache
+            const cacheNames = await caches.keys();
+            for (const cacheName of cacheNames) {
+                if (cacheName.startsWith('transformers-cache') || cacheName.includes(MODEL_NAME.replace(/\//g, '-'))) {
+                    await caches.delete(cacheName);
+                }
+            }
+            indexData = null; // Clear in-memory data
+            isReady = false; // Mark as not ready until reloaded
+            self.postMessage({ type: 'dataDeleted', payload: 'Cached data deleted successfully.' });
+        } catch (error) {
+            self.postMessage({ type: 'error', payload: 'Failed to delete cached data.' });
+        }
+    } else if (type === 'getIndexSize') {
+        try {
+            let totalSize = 0;
+            let indexCached = false;
+            let modelCached = false;
+            // Check if index is cached
+            const cachedIndex = await getFromDB('quoteIndexData');
+            if (cachedIndex) {
+                totalSize += JSON.stringify(cachedIndex.metadata).length + cachedIndex.embeddingsByteLength;
+                indexCached = true;
+            } else {
+                const response = await fetch(INDEX_PATH, { method: 'HEAD' });
+                const contentLength = response.headers.get('Content-Length');
+                totalSize += parseInt(contentLength, 10);
+            }
+            // Add approximate model size
+            totalSize += APPROX_MODEL_SIZE_MB * 1024 * 1024; // Convert MB to bytes
+            // Heuristic: assume model is cached if `model` object is initialized
+            if (model) {
+                modelCached = true;
+            }
+            self.postMessage({ type: 'indexSize', payload: { size: totalSize, indexCached: indexCached, modelCached: modelCached } });
+        } catch (error) {
+            self.postMessage({ type: 'error', payload: 'Failed to get index size: ' + error.message });
+        }
+    }
+};
+// 5. Search Function
+async function search(query) {
+    if (!model || !indexData) {
+        return [];
+    }
+    // Generate query embedding
+    const queryEmbedding = await model("search_query: " + query, { pooling: 'mean', normalize: true });
+    const truncatedQueryEmbedding = queryEmbedding.data.slice(0, EMBEDDING_DIM);
+    // Calculate cosine similarities
+    const similarities = [];
+    for (let i = 0; i < indexData.embeddings.length; i++) {
+        const similarity = cosineSimilarity(truncatedQueryEmbedding, indexData.embeddings[i]);
+        similarities.push({ index: i, similarity });
+    }
+    // Sort by similarity
+    similarities.sort((a, b) => b.similarity - a.similarity);
+    // Get top 20 results
+    const topResults = similarities.slice(0, 20).map(item => {
+        return indexData.metadata[item.index];
+    });
+    return topResults;
+}
+// 6. Helper Functions
+function cosineSimilarity(vecA, vecB) {
+    let dotProduct = 0;
+    let normA = 0;
+    let normB = 0;
+    for (let i = 0; i < vecA.length; i++) {
+        dotProduct += vecA[i] * vecB[i];
+        normA += vecA[i] * vecA[i];
+        normB += vecB[i] * vecB[i];
+    }
+    return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB));
+}
+function reshape(array, shape) {
+    const reshaped = [];
+    let offset = 0;
+    for (let i = 0; i < shape[0]; i++) {
+        reshaped.push(array.slice(offset, offset + shape[1]));
+        offset += shape[1];
+    }
+    return reshaped;
+}