Upload folder using huggingface_hub

.gitignore

@@ -0,0 +1,14 @@
+.env
+.openai
+.dockerignore
+chat_history.pkl
+embeddings.json
+new_uploads/*
+prev_uploads/*
+
+
+# Work in Progress
+app.py
+index.html
+index.js
+index.css

2024-08-06T15:30:18.239414.json

@@ -0,0 +1 @@
+[{"timestamp": "2024-08-06T15:30:18.239414"}, {"role": "user", "content": "Tell me about a lease"}, {"role": "assistant", "content": "A lease is a legal agreement between a landlord (or lessor) and a tenant (or lessee) that outlines the terms and conditions under which a property is rented. It typically includes details such as the duration of the lease (term), the amount of rent to be paid, any security deposits required, maintenance responsibilities, and other provisions related to the use of the property. Leases can vary in length, with some being short-term (e.g., month-to-month) and others lasting several years. A lease provides both parties with rights and obligations regarding the property being rented for the specified period."}, {"full_prompt": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Context:\nTERM\n\na) The term of the Lease shall be for sixteen (16) years, commencing on the first day of May, 1998, and ending on the last day of April, 2014.\n\nPOSSESSION\n\na) Lessee shall be entitled to possession on or about May 1, 1998 or upon completion of Tenant Improvement work.\n\nb) In the event the Lessor shall permit the Lessee to occupy the Premises prior to the commencement date of the term of this Lease, such occupancy shall be subject to all the provisions of this Lease. Early possession shall not advance the termination date of this Lease, and Lessor shall not be responsible for equipment installed by the Lessee.\n\nRENT\n\na) This Lease is the type commonly referred to as a \u201cgross lease\u201d. Lessee shall pay\nQUIET ENJOYMENT\n\nLessor warrants that Lessee, upon discharge of all of Lessee\u2019s obligations hereunder, shall and may be peaceably and quietly hold and enjoy the premises for the term of the Lease.\n\nLSEKENN.DOC 7\n\n26. GROUND LEASE The Lessee acknowledges that the Lessor has entered into a Ground Lease for the property described in Exhibit A, and has constructed an office building on said property. All rights of Lessor are subject to that Ground Lease, and in the event of any dispute between the terms of this Lease and that of the Ground Lease, the Ground Lease shall control.\n\nDATED this__/ S* day of RornQ , 1998,\n\nLESSOR:\n\nHIGHLINE PROFESSIONAL BUILDING ASSOCIATES LIMITED PARTNERSHIP 10900 N.E. Fourth, Suite 1550\n\nBellevue, WA 98004\n\nBy: Lf (A vl I) hid!\n\nGeneral Pa\n\nLESSEE: Surgery Associates Michael Kennelly, M.D.\n\n16122 Eighth Street SW Suite D-1 Seattle, WA 98166\n\nLESSEE\n\nLESSEE:\n\nSurgery Associates\n\nWilliam L. White, M.D.\n\n16122 Eighth Street SW Suite D-1 Seattle, WA 98166\nWu-Hsun Yang and Geoffrey Bender\n\nFirst Amendment to Lease\n\nHighline Professional Building\n\nThe parties to this First Amendment to Lease dated the December ___, 2023 are Highline Professional Building Associates, Limited Partnership (\u201cLandlord\u201d) and Wu-Hsun Yang and Geoffrey Bender (\u201cTenant\u201d).\t\n\nWHEREAS, Landlord and Tenant have a binding Lease Agreement dated June 10, 2021, together the lease in and to the premises described in Exhibit A Legal Description of the Property more commonly known as Highline Professional Building, 16122 8th, Avenue SW, Suite D3, City of Burien, County of King, State of Washington.\n\nThe parties do hereby mutually agree to modify certain particulars of the Lease Agreement as follow:\n\n1.\tSection 4 Term\n\nSection 4 shall be modified as follows:\n\n\u201cThe term of the Lease shall be extended ten (10) years commencing August 1, 2024 and ending July 31, 2034.\u201d\n\n2.\tSection 5 Rent.\n\n\tMonthly rent for the extended term of the Lease shall be as follows:\nSouth Seattle Nephrology Associates, LLC.\n\nFirst Amendment to Lease\n\nHighline Professional Building\n\nThe parties to this First Amendment to Lease dated the December ___, 2023 are Highline Professional Building Associates, Limited Partnership (\u201cLandlord\u201d) and South Seattle Nephrology Associates, LLC. a Washington Limited Liability Company (\u201cTenant\u201d).\t\n\nWHEREAS, Landlord and Tenant have a binding Lease Agreement dated October 1, 2018, together the lease in and to the premises described in Exhibit A Legal Description of the Property more commonly known as Highline Professional Building, 16122 8th, Avenue SW, Suite E-2, City of Burien, County of King, State of Washington.\n\nThe parties do hereby mutually agree to modify certain particulars of the Lease Agreement as follow:\n\n1.\tSection 4 Term\n\nSection 4 shall be modified as follows:\n\n\u201cThe term of the Lease shall be extended three (3) years commencing January 1, 2024 and ending December 31, 2026.\u201d\n\n2.\tSection 5 Rent.\nSteven McClean MD, PC\n\nFirst Amendment to Lease\n\nHighline Professional Building\n\nThe parties to this First Amendment to Lease dated the September 15, 2022, are Highline Professional Building Associates, Limited Partnership (\u201cLandlord\u201d) and Steven P. McClean MD, PC. an individual (\u201cTenant\u201d).\t\n\nWHEREAS, Landlord and Tenant have a binding Lease Agreement dated July 5, 2012, together the lease in and to the premises described in Exhibit A Legal Description of the Property more commonly known as Highline Professional Building, 16122 8th, Avenue SW, Suite E-3, City of Burien, County of King, State of Washington.\n\nThe parties do hereby mutually agree to modify certain particulars of the Lease Agreement as follow:\n\n1.\tSection 4 Term\n\nSection 4 shall be modified as follows:\n\n\u201cThe term of the Lease shall be extended five (5) years commencing October 1, 2022 and ending September 30, 2027.\u201d\n\n2.\tSection 5 Rent.\n\n\tMonthly rent for the extended term of the Lease shall be as follows:\n\n\nQuestion: Tell me about a lease\nAnswer:"}]}]

Dockerfile

@@ -0,0 +1,63 @@
+# syntax=docker/dockerfile:1
+
+# Comments are provided throughout this file to help you get started.
+# If you need more help, visit the Dockerfile reference guide at
+# https://docs.docker.com/go/dockerfile-reference/
+
+# Want to help us make this template better? Share your feedback here: https://forms.gle/ybq9Krt8jtBL3iCk7
+
+ARG PYTHON_VERSION=3.12.4
+FROM python:${PYTHON_VERSION}-slim as base
+
+# Prevents Python from writing pyc files.
+ENV PYTHONDONTWRITEBYTECODE=1
+
+# Keeps Python from buffering stdout and stderr to avoid situations where
+# the application crashes without emitting any logs due to buffering.
+ENV PYTHONUNBUFFERED=1
+
+WORKDIR /app
+
+# Create a non-privileged user that the app will run under.
+# See https://docs.docker.com/go/dockerfile-user-best-practices/
+ARG UID=10001
+RUN adduser \
+    --disabled-password \
+    --gecos "" \
+    --home "/nonexistent" \
+    --shell "/sbin/nologin" \
+    --no-create-home \
+    --uid "${UID}" \
+    appuser
+
+# Install system dependencies.
+RUN apt-get update && apt-get install -y \
+    build-essential \
+    meson \
+    ninja-build \
+    pkg-config \
+    libpoppler-cpp-dev \
+    && rm -rf /var/lib/apt/lists/*
+
+# Download dependencies as a separate step to take advantage of Docker's caching.
+# Leverage a cache mount to /root/.cache/pip to speed up subsequent builds.
+# Leverage a bind mount to requirements.txt to avoid having to copy them into
+# into this layer.
+RUN --mount=type=cache,target=/root/.cache/pip \
+    --mount=type=bind,source=requirements.txt,target=requirements.txt \
+    python -m pip install -r requirements.txt
+
+# Switch to the non-privileged user to run the application.
+USER appuser
+
+# Copy the source code into the container.
+COPY . .
+
+# Expose the port that the application listens on.
+EXPOSE 7860
+
+# Ensure Gradio listens on all network interfaces
+ENV GRADIO_SERVER_NAME="0.0.0.0"
+
+# Run the application.
+CMD ["python", "OpenAIChat.py"]

EmbedUpsert.py

@@ -0,0 +1,121 @@
+import os
+import json
+import dotenv
+from openai import OpenAI
+from langchain.text_splitter import CharacterTextSplitter
+from langchain_community.document_loaders import DirectoryLoader
+from utils import DOC_CHUNK_SIZE, DOC_CHUNK_OVERLAP, DOC_DIRECTORY, EMBEDDING_FILE
+
+'''
+This file handles the loading and embedding of documents.
+Saves the embeddings to a json file specified by EMBEDDING_FILE.
+Uses OpenAI's text-embedding-3-small model for embeddings.
+
+Supported formats: pdf
+
+Format of saved json file:
+    List of dictionaries with keys:
+        'doc_id': {filename}_{chunk number}
+        'embeddings': List of embeddings for the chunk, one embedding for each character in chunk
+
+Specifications:
+    OpenAI embeds with a dimension of 1536 per character
+'''
+
+# Load environment variables
+dotenv.load_dotenv()
+
+def load_documents():
+    '''
+    Load documents from a specified directory into a list
+
+    Returns:
+        documents: List of documents loaded from the directory, split by chunks
+    '''
+    # Create document loaders
+    pdf_loader = DirectoryLoader(DOC_DIRECTORY, glob='*.pdf')
+    docx_loader = DirectoryLoader(DOC_DIRECTORY, glob='*.docx')
+    txt_loader = DirectoryLoader(DOC_DIRECTORY, glob='*.txt')
+    loaders = [pdf_loader]
+
+    # Load documents
+    print("Loading documents...")
+    documents = []
+    for loader in loaders:
+        try:
+            documents.extend(loader.load())
+        except Exception as e:
+            print(f"Error loading documents: {e}")
+    
+    if (len(documents) == 0):
+        print("No documents loaded.")
+        return []
+
+    # Split documents into chunks
+    text_splitter = CharacterTextSplitter(chunk_size=DOC_CHUNK_SIZE, chunk_overlap=DOC_CHUNK_OVERLAP)
+    documents = text_splitter.split_documents(documents)
+
+    # Iterate to edit metadata to include chunk number
+    # format = {filename}_{chunk number}
+    chunk_num = 1
+    prev_doc_id = documents[0].metadata['source']
+    for chunk in documents:
+        print(chunk.metadata)
+        if chunk.metadata['source'] != prev_doc_id:
+            chunk_num = 1
+            prev_doc_id = chunk.metadata['source']
+        chunk.metadata['source'] = f"{prev_doc_id}_{chunk_num}"
+        chunk_num += 1
+    
+    return documents
+
+def embed_documents(documents):
+    '''
+    Embed documents using OpenAIEmbeddings
+
+    Args:
+        documents: List of documents to embed
+
+    Returns:
+        List of JSON objects {doc_id, embeddings, metadata}
+    '''
+    # Use OpenAI to embed documents
+    client = OpenAI(
+        api_key=os.getenv("OPENAI_API_KEY")
+    )
+    embeddings = []
+    print("Embedding documents...")
+
+    # Embed each chunk
+    for chunk in documents:
+        chunk_embeddings = client.embeddings.create(
+            model="text-embedding-3-small",
+            input=chunk.page_content
+        )
+        # Extract embeddings from response
+        chunk_embedding = [record.embedding for record in chunk_embeddings.data]
+        embeddings.append({
+            'doc_id': chunk.metadata['source'],
+            'embeddings': chunk_embedding[0],
+            'metadata': {'source': chunk.metadata['source'], 'text': chunk.page_content}
+        })
+    return embeddings
+
+def save_embeddings(embeddings, filename):
+    '''
+    Save generated embedding to a json file
+
+    Args:
+        embeddings: List of embeddings to save
+        filename: Name of the file to save the embeddings
+    '''
+    print("Saving embedding...")
+    with open(filename, 'w') as file:
+        json.dump(embeddings, file)
+
+### Main code
+documents = load_documents()
+if (len(documents) == 0):
+    exit()
+embeddings = embed_documents(documents)
+save_embeddings(embeddings, EMBEDDING_FILE)

OpenAIChat.py

@@ -0,0 +1,525 @@
+import os
+import gradio
+import shutil
+import json
+import dotenv
+import pickle
+import itertools
+from collections import Counter
+from datetime import datetime
+from huggingface_hub import HfApi, HfFolder
+from openai import OpenAI as OpenAIClient
+from pinecone import ServerlessSpec
+from pinecone.grpc import PineconeGRPC as Pinecone
+from transformers import BertTokenizerFast
+from langchain.text_splitter import CharacterTextSplitter
+from langchain_community.document_loaders import DirectoryLoader
+
+'''
+This file contains the code for user prompting of the language model.
+The language model used is gpt 3.5 turbo and uses documents stored in Pinecone.
+'''
+
+# Load environment variables
+dotenv.load_dotenv()
+assert os.getenv("OPENAI_API_KEY") is not None, "Please set the OPENAI_API_KEY environment variable."
+assert os.getenv("PINECONE_API_KEY") is not None, "Please set the PINECONE_API_KEY environment variable."
+assert os.getenv("HUGGINGFACE_API_KEY") is not None, "Please set the HUGGINGFACE_API_KEY environment variable."
+
+HfFolder.save_token(os.getenv("HUGGINGFACE_API_KEY"))
+
+CHAT_HISTORY_FILE = "chat_history.pkl"
+NEW_UPLOAD_DIRECTORY = "new_uploads/"
+PREV_UPLOAD_DIRECTORY = "prev_uploads/"
+DOC_CHUNK_SIZE = 1000
+DOC_CHUNK_OVERLAP = 40
+EMBEDDING_FILE = 'embeddings.json'
+INDEX_NAME = "hybrid"
+BATCH_SIZE = 100
+INTERACTIONS_DATASET = "ryanRocks/FalconOpenAIInteractions"
+FILES_DATASET = "ryanRocks/FalconOpenAIFiles"
+
+# Ensure uploads directory exists
+os.makedirs(NEW_UPLOAD_DIRECTORY, exist_ok=True)
+os.makedirs(PREV_UPLOAD_DIRECTORY, exist_ok=True)
+
+chat_history = []
+chatbot_history = []
+
+# Initialize Pinecone database
+try:
+    pc = Pinecone(
+        api_key=os.getenv("PINECONE_API_KEY"),
+        pool_threads=30,
+        spec=ServerlessSpec(
+            cloud="aws",
+            region="us-east-1",
+        ),
+    )
+    print("Connected to Pinecone")
+except Exception as e:
+    print(f"Error initializing Pinecone: {e}")
+    exit()
+
+# Initialize index if it does not exist
+existing_indexes = [index.name for index in pc.list_indexes().indexes]
+if INDEX_NAME not in existing_indexes:
+    pc.create_index(
+            name=INDEX_NAME,
+            dimension=1536,
+            metric="dotproduct",
+            spec=ServerlessSpec(
+                cloud="aws",
+                region="us-east-1",
+            ),
+        )
+    print(f"Created index {INDEX_NAME}")
+else:
+    print(f"Index {INDEX_NAME} already exists")
+
+
+def initialize_chat_history():
+    '''
+    Initialize the chat history using the chat history pickle file.
+    '''
+    global chatbot_history
+    loaded_chat_history = []
+    if (os.path.exists(CHAT_HISTORY_FILE)):
+        with open(CHAT_HISTORY_FILE, "rb") as f:
+            loaded_chat_history = pickle.load(f)
+    else:
+        loaded_chat_history = []
+    
+    for i in range(0, len(loaded_chat_history), 2):
+        chatbot_history.append((
+            loaded_chat_history[i]['content'],
+            loaded_chat_history[i+1]['content']
+        ))
+
+async def upload_file(files):
+    '''
+    Upload files to Pinecone
+
+    Args:
+        files: List of file paths to process
+    '''
+    # Copy files to uploads directory for easier processing
+    for file in files:
+        file_path = os.path.join(NEW_UPLOAD_DIRECTORY, file.name.split('/')[-1])
+        shutil.move(file.name, file_path)
+
+    # Load documents
+    documents = read_documents()
+    dense_embeddings = dense_embed(documents)
+    sparse_embeddings = sparse_embed(documents)
+
+    #save_embeddings(dense_embeddings, EMBEDDING_FILE)
+
+    # Upsert embeddings into Pinecone
+    await upsert(dense_embeddings, sparse_embeddings)
+
+    # Move newly uploaded files to previous uploads directory
+    move_files()
+
+    return get_uploaded_files()
+
+def read_documents():
+    '''
+    Load documents from a specified directory into a list
+
+    Args:
+        file_paths: List of file paths to load documents from
+    Returns:
+        documents: List of documents loaded from the directory, split by chunks
+    '''
+    # Load documents
+    print("Loading documents...")
+    documents = []
+
+    # Declare loaders for different file types
+    pdf_loader = DirectoryLoader(NEW_UPLOAD_DIRECTORY, glob="*.pdf")
+    docx_loader = DirectoryLoader(NEW_UPLOAD_DIRECTORY, glob="*.docx")
+    txt_loader = DirectoryLoader(NEW_UPLOAD_DIRECTORY, glob="*.txt")
+
+    for loader in [pdf_loader, docx_loader, txt_loader]:
+        # Load document
+        try:
+            # Error loading documents: Expected directory, got file: '/private/var/folders/sc/_5mj781j5315nzv10s8kvs1w0000gn/T/gradio/33a9766ee3f05c368d3c7fe56f6f2356e88a4348/YuYouChen Resume.pdf'
+            documents.extend(loader.load())
+        except Exception as e:
+            print(f"Error loading documents: {e}")
+
+    if (len(documents) == 0):
+        print("No documents loaded.")
+        return []
+
+    # Split documents into chunks
+    text_splitter = CharacterTextSplitter(chunk_size=DOC_CHUNK_SIZE, chunk_overlap=DOC_CHUNK_OVERLAP)
+    documents = text_splitter.split_documents(documents)
+
+    # Iterate to edit metadata to include chunk number
+    # format = {filename}_{chunk number}
+    chunk_num = 1
+    prev_doc_id = documents[0].metadata['source']
+    for chunk in documents:
+        if chunk.metadata['source'] != prev_doc_id:
+            chunk_num = 1
+            prev_doc_id = chunk.metadata['source']
+        chunk.metadata['source'] = f"{prev_doc_id}_{chunk_num}"
+        chunk_num += 1
+    
+    print("Documents loaded")
+    return documents
+
+def dense_embed(documents):
+    '''
+    Embed documents using OpenAIEmbeddings
+
+    Args:
+        documents: List of documents to embed
+
+    Returns:
+        List of JSON objects {doc_id, embeddings, metadata}
+    '''
+    print("Generating dense embeddings...")
+    # Use OpenAI to embed documents
+    client = OpenAIClient(
+        api_key=os.getenv("OPENAI_API_KEY")
+    )
+    embeddings = []
+
+    # Embed each chunk
+    for chunk in documents:
+        chunk_embeddings = client.embeddings.create(
+            model="text-embedding-3-small",
+            input=chunk.page_content
+        )
+        # Extract embeddings from response
+        chunk_embedding = [record.embedding for record in chunk_embeddings.data]
+        embeddings.append({
+            'doc_id': chunk.metadata['source'].split('/')[-1],
+            'embeddings': chunk_embedding[0],
+            'metadata': {'source': chunk.metadata['source'], 'text': chunk.page_content}
+        })
+    
+    print("Complete")
+    return embeddings
+
+def sparse_embed(documents):
+    '''
+    Generate sparse embeddings for a list of documents
+    
+    Args:
+        documents: List of documents to generate sparse embeddings for
+        
+    Returns:
+        List of sparse embeddings in dictionary format
+    '''
+    print("Generating sparse embeddings...")
+    tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')
+    sparse_embeds = []
+
+    for chunk in documents:
+        # Create batch of input_ids
+        inputs = tokenizer(
+            chunk.page_content,
+            padding=True,
+            truncation=True,
+            max_length=512,
+            add_special_tokens=False,
+        )['input_ids']
+
+        # Create sparse dictionaries
+        sparse_embed = build_dict(inputs)
+        sparse_embeds.append(sparse_embed)
+    
+    print("Complete")
+    return sparse_embeds
+
+def build_dict(input_batch):
+    '''
+    Build a dictionary for sparse embeddings
+
+    Args:
+        input_batch: List of embeddings to convert to a dictionary
+
+    Returns:
+        List of sparse embeddings in dictionary format
+    '''
+    sparse_emb = []
+
+    # Iterate through input batch
+    indices = []
+    values = []
+
+    # Convert the input_batch list to a dictionary of key to frequency values
+    freqs = dict(Counter(input_batch))
+    for idx in freqs:
+        indices.append(idx)
+        values.append(float(freqs[idx]))
+    sparse_emb.append({'indices': indices, 'values': values})
+
+    return sparse_emb
+
+def save_embeddings(embeddings, filename):
+    '''
+    Save generated embedding to a json file
+
+    Args:
+        embeddings: List of embeddings to save
+        filename: Name of the file to save the embeddings
+    '''
+    print("Saving embedding...")
+    with open(filename, 'w') as file:
+        json.dump(embeddings, file)
+    print("Complete")
+
+def chunks(iterable):
+    '''
+    Breaks vector list into chunks of BATCH_SIZE for parallel upserts
+
+    Args:
+        iterable: List of vectors to chunk
+    '''
+    print("Chunking")
+    it = iter(iterable)
+    chunk = tuple(itertools.islice(it, BATCH_SIZE))
+    while chunk:
+        yield chunk
+        chunk = tuple(itertools.islice(it, BATCH_SIZE))
+    print("Complete")
+
+def vectorize(dense_embeddings, sparse_embeddings):
+    '''
+    Vectorize embeddings with document ids to prepare for insertion into Pinecone
+
+    Args:
+        embeddings: List of embeddings to vectorize
+
+    Returns:
+        List of vectors with tuples (chunk ids, embeddings)
+    '''
+    print("Vectorizing...")
+    vectors = []
+    for dense, sparse in zip(dense_embeddings, sparse_embeddings):
+        vectors.append({
+            'id': dense['doc_id'],
+            'values': dense['embeddings'],
+            'sparse_values': sparse[0],
+            'metadata': dense['metadata'],
+        })
+    print("Vectorized")
+    return vectors
+
+async def upsert(dense_embeddings, sparse_embeddings):
+    '''
+    Upsert embeddings into pinecone
+    '''
+    index = pc.Index(INDEX_NAME)
+    vectors = vectorize(dense_embeddings, sparse_embeddings)
+
+    # Insert vectors into database in chunks
+    print("Upserting embeddings...")
+    vector_chunks = chunks(vectors)
+    for chunk in vector_chunks:
+        index.upsert(chunk)
+
+    print("Complete")
+
+def move_files():
+    '''
+    Move uploaded files to the previous uploads directory
+    '''
+    print("Moving files...")
+    api = HfApi()
+    for file in os.listdir(NEW_UPLOAD_DIRECTORY):
+        file_path = os.path.join(NEW_UPLOAD_DIRECTORY, file)
+        if (os.path.isfile(file_path)):
+            # Upload file to HuggingFace Datasets
+            api.upload_file(
+                path_or_fileobj = file_path,
+                path_in_repo = file,
+                repo_id = "ryanRocks/FalconOpenAIFiles",
+                repo_type = "dataset",
+            )
+            # Move file to previous uploads directory
+            new_file_path = os.path.join(PREV_UPLOAD_DIRECTORY, file)
+            shutil.move(file_path, new_file_path)
+
+def hybrid_scale(dense, sparse, alpha):
+    print("Hybrid scaling...")
+    # Check alpha value in range 0 to 1
+    if alpha < 0 or alpha > 1:
+        raise ValueError("Alpha must be between 0 and 1")
+    
+    # Scale dense and sparse vectors to create hybrid search vectors
+    hdense = [v * alpha for v in dense]
+    hsparse = {
+        'indices': sparse['indices'],
+        'values': [v * (1 - alpha) for v in sparse['values']],
+    }
+    print("Complete")
+    return hdense, hsparse
+
+def hybrid_query(question, top_k, alpha):
+    try:
+        print("Hybrid querying...")
+
+        # Convert the question into a dense vector
+        print("Converting question to dense vector...")
+        client = OpenAIClient(
+            api_key=os.getenv("OPENAI_API_KEY")
+        )
+        query_embedding = client.embeddings.create(
+                model="text-embedding-3-small",
+                input=question,
+            )
+        dense_vec = [record.embedding for record in query_embedding.data][0]
+        print("Complete")
+
+        # Convert the question into a sparse vector
+        print("Converting question to sparse vector...")
+        tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')
+        inputs = tokenizer(
+            question,
+            padding=True,
+            truncation=True,
+            max_length=512,
+            add_special_tokens=False,
+        )['input_ids']
+        sparse_vec = build_dict(inputs)[0]
+        print("Complete")
+
+        # Scale alpha with hybrid_scale
+        dense_vec, sparse_vec = hybrid_scale(
+            dense_vec, sparse_vec, alpha
+        )
+
+        # Query pinecone with the query parameters
+        print("Querying Pinecone...")
+        index = pc.Index(INDEX_NAME)
+        result = index.query(
+            vector=dense_vec,
+            sparse_vector=sparse_vec,
+            top_k=top_k,
+            include_values=True,
+            include_metadata=True,
+        )
+        print("Complete")
+        
+        # Return search results as json
+        return result
+    except Exception as e:
+        print(f"Error querying Pinecone: {e}")
+
+def prompt(question, history):
+    '''
+    Prompt the language model with user input
+
+    Args:
+        question: User string input to prompt the language model
+    Returns:
+        Language model response to the user
+    '''
+
+    global chatbot_history
+    global chat_history
+
+    # Handle clearing history
+    if len(history) == 0:
+        chatbot_history = []
+        chat_history = []
+        with open(CHAT_HISTORY_FILE, "wb") as f:
+            pickle.dump(chat_history, f)
+
+    # Check for API key
+    if (os.getenv("OPENAI_API_KEY") is None):
+        return "Please set the OPENAI_API_KEY environment variable."
+    
+    # Query database and prompt the language model with results
+    try:
+        # Query database for context
+        results = hybrid_query(question, top_k=5, alpha=0.4)
+        context = ""
+        for match in results['matches']:
+            context += match['metadata']['text'] + "\n"
+        
+        client = OpenAIClient()
+
+        # Prepare prompt with chat history
+        prompt = f"Context:\n{context}\n\nQuestion: {question}\nAnswer:"
+        messages = [{"role": "system", "content": "You are a helpful assistant."}]
+        messages.extend(chat_history)
+        messages.append({"role": "user", "content": prompt})
+
+        time = datetime.now().isoformat()
+        print("Prompting language model...")
+        response = client.chat.completions.create(
+            messages=messages,
+            model="gpt-3.5-turbo",
+        )
+
+        # Extract answer from response
+        answer = response.choices[0].message.content.strip()
+
+        interaction = [
+            {"timestamp": time},
+            {"role": "user", "content": question},
+            {"role": "assistant", "content": answer},
+            {"full_prompt": messages},
+        ]
+
+        with open(f"{time}.json", "w") as f:
+            json.dump(interaction, f)
+
+        # Upload interaction to HuggingFace Datasets
+        api = HfApi()
+        api.upload_file(
+            path_or_fileobj = f"{time}.json",
+            path_in_repo = f"{time}.json",
+            repo_id = INTERACTIONS_DATASET,
+            repo_type = "dataset",
+        )
+
+        # Save chat history
+        chat_history.extend([
+            {"role": "user", "content": question},
+            {"role": "assistant", "content": answer},
+        ])
+
+        with open(CHAT_HISTORY_FILE, "wb") as f:
+            pickle.dump(chat_history, f)
+
+        return answer
+    
+    # Handle exceptions
+    except Exception as e:
+        return "Error: " + str(e)
+
+def get_uploaded_files():
+    '''
+    Get uploaded files in prev_uploads directory
+    '''
+    uploaded_files = []
+
+    for file in os.listdir(PREV_UPLOAD_DIRECTORY):
+        file_path = os.path.join(PREV_UPLOAD_DIRECTORY, file)
+        if (os.path.isfile(file_path)):
+            uploaded_files.append(file_path)
+    return uploaded_files
+
+# Create a Gradio interface
+with gradio.Blocks() as demo:
+    # Load chat history
+    initialize_chat_history()
+
+    # Create chatbot interface
+    chatbot = gradio.Chatbot(value=chat_history, placeholder="What would you like to know?")
+    gradio.ChatInterface(fn=prompt, chatbot=chatbot)
+
+    # Create file upload interface
+    file_output = gradio.File(value=get_uploaded_files())
+    upload_button = gradio.UploadButton("Click to upload a file", file_types=["pdf, docx, txt"], file_count="multiple")
+    upload_button.upload(upload_file, upload_button, file_output)
+
+demo.launch()

README.Docker.md

@@ -0,0 +1,22 @@
+### Building and running your application
+
+When you're ready, start your application by running:
+`docker compose up --build`.
+
+Your application will be available at http://localhost:8000.
+
+### Deploying your application to the cloud
+
+First, build your image, e.g.: `docker build -t myapp .`.
+If your cloud uses a different CPU architecture than your development
+machine (e.g., you are on a Mac M1 and your cloud provider is amd64),
+you'll want to build the image for that platform, e.g.:
+`docker build --platform=linux/amd64 -t myapp .`.
+
+Then, push it to your registry, e.g. `docker push myregistry.com/myapp`.
+
+Consult Docker's [getting started](https://docs.docker.com/go/get-started-sharing/)
+docs for more detail on building and pushing.
+
+### References
+* [Docker's Python guide](https://docs.docker.com/language/python/)

README.md

@@ -1,12 +1,6 @@
 ---
 title: FalconOpenAI
-emoji: 👀
-colorFrom: red
-colorTo: blue
+app_file: OpenAIChat.py
 sdk: gradio
-sdk_version: 4.40.0
-app_file: app.py
-pinned: false
+sdk_version: 4.38.1
 ---
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

compose.yaml

@@ -0,0 +1,52 @@
+# Comments are provided throughout this file to help you get started.
+# If you need more help, visit the Docker Compose reference guide at
+# https://docs.docker.com/go/compose-spec-reference/
+
+# Here the instructions define your application as a service called "server".
+# This service is built from the Dockerfile in the current directory.
+# You can add other services your application may depend on here, such as a
+# database or a cache. For examples, see the Awesome Compose repository:
+# https://github.com/docker/awesome-compose
+services:
+  server:
+    build:
+      context: .
+    ports:
+      - 7860:7860
+    env_file:
+      - .env
+    image: ryanyyc/project-falcon:openai
+
+# The commented out section below is an example of how to define a PostgreSQL
+# database that your application can use. `depends_on` tells Docker Compose to
+# start the database before your application. The `db-data` volume persists the
+# database data between container restarts. The `db-password` secret is used
+# to set the database password. You must create `db/password.txt` and add
+# a password of your choosing to it before running `docker compose up`.
+#     depends_on:
+#       db:
+#         condition: service_healthy
+#   db:
+#     image: postgres
+#     restart: always
+#     user: postgres
+#     secrets:
+#       - db-password
+#     volumes:
+#       - db-data:/var/lib/postgresql/data
+#     environment:
+#       - POSTGRES_DB=example
+#       - POSTGRES_PASSWORD_FILE=/run/secrets/db-password
+#     expose:
+#       - 5432
+#     healthcheck:
+#       test: [ "CMD", "pg_isready" ]
+#       interval: 10s
+#       timeout: 5s
+#       retries: 5
+# volumes:
+#   db-data:
+# secrets:
+#   db-password:
+#     file: db/password.txt
+

requirements.txt

@@ -0,0 +1,193 @@
+aiofiles==23.2.1
+aiohttp==3.9.5
+aiosignal==1.3.1
+altair==5.3.0
+annotated-types==0.7.0
+antlr4-python3-runtime==4.9.3
+anyio==4.4.0
+attrs==23.2.0
+backoff==2.2.1
+beautifulsoup4==4.12.3
+blinker==1.8.2
+cachetools==5.4.0
+certifi==2024.7.4
+cffi==1.16.0
+chardet==5.2.0
+charset-normalizer==3.3.2
+click==8.1.7
+coloredlogs==15.0.1
+contourpy==1.2.1
+cryptography==43.0.0
+cycler==0.12.1
+dataclasses-json==0.6.7
+datasets==2.20.0
+deepdiff==7.0.1
+Deprecated==1.2.14
+dill==0.3.8
+distro==1.9.0
+dnspython==2.6.1
+effdet==0.4.1
+email_validator==2.2.0
+emoji==2.12.1
+et-xmlfile==1.1.0
+fastapi==0.111.0
+fastapi-cli==0.0.4
+ffmpy==0.3.2
+filelock==3.15.4
+filetype==1.2.0
+Flask==3.0.3
+flatbuffers==24.3.25
+fonttools==4.53.1
+frozenlist==1.4.1
+fsspec==2024.5.0
+google-api-core==2.19.1
+google-auth==2.32.0
+google-cloud-vision==3.7.3
+googleapis-common-protos==1.63.2
+gradio==4.38.1
+gradio_client==1.1.0
+grpc-gateway-protoc-gen-openapiv2==0.1.0
+grpcio==1.65.1
+h11==0.14.0
+httpcore==1.0.5
+httptools==0.6.1
+httpx==0.27.0
+huggingface-hub==0.23.4
+humanfriendly==10.0
+idna==3.7
+importlib_resources==6.4.0
+iopath==0.1.10
+itsdangerous==2.2.0
+Jinja2==3.1.4
+joblib==1.4.2
+jsonpatch==1.33
+jsonpath-python==1.0.6
+jsonpointer==3.0.0
+jsonschema==4.23.0
+jsonschema-specifications==2023.12.1
+kiwisolver==1.4.5
+langchain==0.2.7
+langchain-community==0.2.7
+langchain-core==0.2.17
+langchain-openai==0.1.16
+langchain-pinecone==0.1.1
+langchain-text-splitters==0.2.2
+langdetect==1.0.9
+langsmith==0.1.85
+layoutparser==0.3.4
+libmagic==1.0
+lxml==5.2.2
+lz4==4.3.3
+Markdown==3.6
+markdown-it-py==3.0.0
+MarkupSafe==2.1.5
+marshmallow==3.21.3
+matplotlib==3.9.1
+mdurl==0.1.2
+mpmath==1.3.0
+multidict==6.0.5
+multiprocess==0.70.16
+mypy-extensions==1.0.0
+nest-asyncio==1.6.0
+networkx==3.3
+nltk==3.8.1
+numpy==1.26.4
+olefile==0.47
+omegaconf==2.3.0
+onnx==1.16.1
+onnxruntime==1.18.1
+openai==1.35.13
+opencv-python==4.10.0.84
+openpyxl==3.1.5
+ordered-set==4.1.0
+orjson==3.10.6
+packaging==24.1
+pandas==2.2.2
+pdf2image==1.17.0
+pdfminer.six==20231228
+pdfplumber==0.11.2
+pikepdf==9.0.0
+pillow==10.4.0
+pillow_heif==0.17.0
+pinecone==4.0.0
+pinecone-client==3.2.2
+portalocker==2.10.1
+proto-plus==1.24.0
+protobuf==3.20.3
+psutil==6.0.0
+pyarrow==17.0.0
+pyarrow-hotfix==0.6
+pyasn1==0.6.0
+pyasn1_modules==0.4.0
+pycocotools==2.0.8
+pycparser==2.22
+pydantic==2.8.2
+pydantic_core==2.20.1
+pydub==0.25.1
+Pygments==2.18.0
+pypandoc==1.13
+pyparsing==3.1.2
+pypdf==4.3.1
+pypdfium2==4.30.0
+pytesseract==0.3.10
+python-dateutil==2.9.0.post0
+python-docx==1.1.2
+python-dotenv==1.0.1
+python-iso639==2024.4.27
+python-magic==0.4.27
+python-multipart==0.0.9
+python-oxmsg==0.0.1
+python-pptx==0.6.23
+pytz==2024.1
+PyYAML==6.0.1
+rapidfuzz==3.9.4
+referencing==0.35.1
+regex==2024.5.15
+requests==2.32.3
+requests-toolbelt==1.0.0
+rich==13.7.1
+rpds-py==0.19.0
+rsa==4.9
+ruff==0.5.1
+safetensors==0.4.3
+scipy==1.14.0
+semantic-version==2.10.0
+shellingham==1.5.4
+six==1.16.0
+sniffio==1.3.1
+soupsieve==2.5
+SQLAlchemy==2.0.31
+starlette==0.37.2
+sympy==1.13.1
+tabulate==0.9.0
+tenacity==8.5.0
+tesseract==0.1.3
+tiktoken==0.7.0
+timm==1.0.7
+tokenizers==0.19.1
+tomlkit==0.12.0
+toolz==0.12.1
+torch==2.3.1
+torchvision==0.18.1
+tqdm==4.66.4
+transformers==4.42.4
+typer==0.12.3
+typing-inspect==0.9.0
+typing_extensions==4.12.2
+tzdata==2024.1
+ujson==5.10.0
+unstructured==0.15.0
+unstructured-client==0.24.1
+unstructured-inference==0.7.36
+unstructured.pytesseract==0.3.12
+urllib3==2.2.2
+uvicorn==0.30.1
+uvloop==0.19.0
+watchfiles==0.22.0
+websockets==11.0.3
+Werkzeug==3.0.3
+wrapt==1.16.0
+xlrd==2.0.1
+XlsxWriter==3.2.0
+xxhash==3.4.1
+yarl==1.9.4

utils.py

@@ -0,0 +1,23 @@
+import json
+
+def load_embeddings(filename):
+    '''
+    Load embeddings from specified json file
+
+    Args:
+        filename: Name of the json file to load embeddings from
+
+    Returns:
+        List of embeddings loaded from the file
+    '''
+    print("Loading embeddings...")
+    with open(filename, 'r') as file:
+        embeddings = json.load(file)
+    return embeddings
+
+DOC_CHUNK_SIZE = 1000
+DOC_CHUNK_OVERLAP = 40
+DOC_DIRECTORY = './'
+EMBEDDING_FILE = 'embeddings.json'
+BATCH_SIZE = 100
+INDEX_NAME = "project-falcon"