seritwin/Virginia-food-delivery

Model Card for Model ID

Virginia Food Delivery

Model Details

Introduction

Meal recommendations anyone? The original goal of this project was to develop a chatbot that provided meal recommendations for a specific restaurant in Virginia. Due to limited available training data, the scope shifted toward training an LLM to search for restaurants across Virginia that offer food delivery services, various cuisines, and different locations. This capability is important because customers want to search based on price range, menu offerings, and high-level dietary needs or preferences that can be delivered directly to their homes. Current LLMs struggle with this task due to limited and fragmented knowledge of restaurant delivery service offerings.

Data

The RAG dataset used for this project consisted of two CSV files from Kaggle—Uber Eats USA Restaurants and Menus—called restaurant.csv and restaurant-menus.csv. The data preprocessing involved multiple steps prior to training the Large Language Model (LLM). There were 11 columns in the restaurant file {id, position, name, score, ratings, category, price_range, full_address, zip_code, lat, long}. The first step was to separate the full address. The next step aggregated the unique menu items by restaurant ID and identified the maximum and minimum price of food items for each establishment.

Note: The price range in the restaurants file listed price ranges using the “$” symbol, but this column was not used in the RAG dataset. The next step extracted Virginia restaurants (by state), after which the datasets were merged on restaurant ID. Any rows with missing data were removed from the Virginia restaurant dataset.

Prepare data for RAG: A Virginia subset of the dataset was generated using a machine learning package in Python. The data was stratified on the “city” column to preserve the subset proportions relative to the larger dataset. For the RAG dataset, the training size was set to 4,000 entries, which corresponded to 4,000 documents for retrieval.

Note: Some column headers were renamed for clarity.

# Split the data into training and test sets

from sklearn.model_selection import train_test_split

# Stratify by city to keep the same proportion of samples from each city

virginia_subset, _ = train_test_split(
    virginia_restaurants_forRAG,
    train_size=4000,
    stratify=virginia_restaurants_forRAG['city'],   
    random_state=5002
)

Methodology

This project uses the RAG approach, as it was best suited for restaurant data stored in CSV files. Restaurant data is typically structured data in spreadsheet or tabular format. According to Lewis et al. (2020), structured RAG can minimize hallucinations and yield better results from data aggregation compared to traditional RAG. The method involved importing the Virginia restaurant subset data, embedding the subset data, and storing the embeddings in a vector store or vector database.

For this project, each row was embedded as a document. To improve the embeddings, the columns for each row were combined to create text_for_embedding, as outlined by Palakkode (2025) in Build a Simple RAG System with CSV Files: Step-by-Step Guide for Beginners. The code chunk below shows the process used to create the new text_for_embedding column.

# text for embedding column with combined details - it's recommended 
virginia_subset['text_for_embedding'] = (
    "RESTAURANT: " + virginia_subset['name'] + "\n" +
    "LOCATION: " + virginia_subset['street'] + ", " + virginia_subset['city'] + ", " + virginia_subset['zipcode'].astype(str) + "\n" +
    "RESTAURANT CATEGORY: " + virginia_subset['restaurant category'] + "\n" +
    "MENU CATEGORY: " + virginia_subset['menu category'] + "\n" +
    "MENU ITEMS: " + virginia_subset['menu item'] + "\n" +
    "PRICE RANGE: " + virginia_subset['min price'].astype(str) + " to " + virginia_subset['max price'].astype(str)
)

Evaluation

The evaluation metrics selected for this project include Exact Match, F1-Score, ROUGE, and Cosine Similarity.

Exact Match is useful for identifying when a response exactly matches the ground truth. The F1-Score was selected because some menu items contain multi-word phrases, such as side of smashed avocado, and partial matches may still be acceptable. For example, a user searching for smashed avocado may not require it to be listed specifically as a side, as it could be a main dish or snack. The ROUGE metric was selected to measure similarity in phrases, as it captures the longest common subsequence (LCS); this allows for unique and relevant responses that may not be exact matches. Cosine Similarity was selected to capture words or phrases that are close in meaning to one another.

Model	Exact Match	F1-Score	ROUGE	Cosine Similarity
Qwen2.5-1.5B-Instruct	0.33	0.33	0.4580	0.5850
Falcon-H1-1.5B-Instruct	0.00	0.00	0.1453	0.5850
meta-llama/Llama-3.2-3B-Instruct	0.00	0.00	0.1500	0.5850

Usage and Intended Uses

The two code blocks below outline the steps to load the RAG dataset and the Hugging Face model. Note: Other necessary dependencies are omitted from the code chunks.

# Loading the Virginia Subset Dataset
import numpy as np
import pandas as pd

virginia_subset = pd.read_csv('virginia_subset_RAGtraining.csv')
virginia_subset.head()

# Load the Huggingface Model
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")  
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct",
                          device_map ="auto", dtype = torch.float16)

Embed the documents (CSV file)

# Convert each row to it's own document

documents = []

for row in virginia_subset.itertuples(index=False):
    documents.append(
        Document(
            page_content=row.text_for_embedding,  # text for embedding
            metadata={
                "name": row.name,
                "city": row.city,
                "zipcode": row.zipcode,
                "restaurant_category": row.restaurant_category  # clean column name
            }
        )
    )

# Embed each document and store in vector database

hf_embedding_model = HuggingFaceEmbeddings(model_name="Qwen/Qwen3-Embedding-0.6B", 
                                           model_kwargs={"device": "cuda"}
                                           )

vectorstore = FAISS.from_documents(documents, hf_embedding_model) 

Prompt Format - Define User Query

def user_query(query, top_k=1, max_tokens=300, return_documents=False):
    documents = vectorstore.similarity_search_with_score(query, k=top_k)
    
    documents_only = [document[0] for document in documents]
    faiss_distances = [score for _, score in documents]

    # Convert FAISS L2 distance → cosine similarity
    # Assumes embeddings were normalized when added to FAISS
    cosine_scores = [1 - (d / 2) for d in faiss_distances]

    # Combine the context
    context_info = [document.page_content for document, _ in documents]
    context_text = "\n".join(context_info)  


    messages = [
        {"role": "system", 
         "content": ( "You are an assistant specializing in locating Virginia Restaurants "
                      "that offer food delivery. User the context to answer accurately. "
                      "If the context does not contain the answer, say so."
                    )
        },
        {"role": "user", 
         "content": ( 
             f"Context:\n{context_text}\n\n"
             f"Question:{query}"
                )
        }
    ]
    
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True
    )
    model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

    generated_ids = model.generate(
        **model_inputs,
        max_new_tokens=512,
        repetition_penalty=1.1,
        temperature = 0.2,
        do_sample = True,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id
    )

    generated_ids = [
        output_ids[len(input_ids):] 
        for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
    ]

    response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

    if return_documents:
        return response, context_info, faiss_distances, cosine_scores

    return response

Test Prompts using user_query function

# -------------------------
# Step 1: Load evaluation prompts
# -------------------------
# These are your test queries with expected answers

test_data = [
    {
        "prompt": "What restaurants in Richmond, VA serves southern food?",
        "ground_truth": "Big Herm's Kitchen"
    },    
    {
        "prompt": "Which is the price range at Iron Paffles and Coffee located in Charlottesville, VA?",
        "ground_truth": "$1.00 - $15.00"
    }]

# -------------------------
# Step 2: Generate answers using user_query
# -------------------------
results = []
for sample in test_data:
    prompt = sample["prompt"]
    ground_truth = sample["ground_truth"]

    # Call your existing retrieval + generation function
    generated, retrieved_context, faiss_scores,
            cosine_scores = user_query(prompt, top_k=1, max_tokens=300, return_documents=True)

    # Convert FAISS L2 → cosine similarity
    cosine_scores = [1 - (d / 2) for d in faiss_scores]


    results.append({
        "prompt": prompt,
        "ground_truth": ground_truth,
        "generated": generated,
        "retrieved_context": retrieved_context,
        "cosine_similarity": cosine_scores
    })

# -------------------------
# Step 3: Compute metrics
# -------------------------
# User-defined evaluation metrics (e.g., Exact Match, F1, ROUGE, Cosine Similarity)
# Implementation is omitted for flexibility.

# -------------------------
# Step 4: Print results
# -------------------------
# User-defined result formatting and display
# Implementation is omitted.

Expected Output - Format

Sample outputs:
Prompt: What restaurants in Richmond, VA serves southern food?
Ground Truth: Big Herm's Kitchen
Generated: Big Herm's Kitchen

Retrieved Context & Cosine Similarity:
--------------------------------------------------
[Doc 0] Cosine=0.5850
RESTAURANT: Big Herm's Kitchen
LOCATION: 315 N 2Nd St, Richmond, 23219
CATEGORIES: southern, black-owned, american
MENU CATEGORIES: Picked for you, Appetizers, Sides, Sandwiches and Baskets, Burgers, Salads, Homemade Desserts, Beverages
MENU ITEMS: Roasted Corn and Asparagus Salad, Blackened Chicken ...

------------------------------------------------------------
Prompt: Which is the price range at Iron Paffles and Coffee located in Charlottesville, VA?
Ground Truth: $1.00 - $15.00
Generated: The price range at Iron Paffles and Coffee, located in Charlottesville, VA, is $1.00 to $15.00.

Retrieved Context & Cosine Similarity:
--------------------------------------------------
[Doc 0] Cosine=0.7914
RESTAURANT: Iron Paffles and Coffee
LOCATION: 214 Water St W, Charlottesville, 22902
CATEGORIES: american, sandwich, desserts, allergy friendly
MENU CATEGORIES: Picked for you, Savory Paffles, Sweet Paffles, Beverages, Sides, Build your Own, Espresso
MENU ITEMS: Maine Root Root Beer, Iced Mocha, 12o ...

Additional user_query implementation & Output

query = "What restaurants offer delivery for Indian cuisines in Virginia Beach?"
answer = user_query(query, top_k=1)
print("=== Generated Answer ===")
print(answer)

=== Generated Answer ===
Saffron Indian Bistro offers delivery of Indian cuisine in Virginia Beach.


query = "Are there any vegan or vegetarian options in Northern Virginia?"
answer = user_query(query, top_k=1)
print("=== Generated Answer ===")
print(answer)

=== Generated Answer ===
Yes, there are several vegetarian and vegan options available at Biryani Hub in Northern Virginia. Some of these include:

- Channa Masala
- Chilli Garlic Naan
- Murgh Tikka (Chicken) Tandoor
- Gosht Kurma (Goat)
- Butter Naan
- Dal Fry
- Paneer (Cottage Cheese) Butter Masala
- Murgh Saag (Chicken)
- Dum Ka Veg
- Chef's Special Murgh (Chicken)
- Shrimp Apollo
- Kheema (Ground Goat) Biryani

Note: This list is cut short, but there were more results.

Limitations

This model is intended for high-level meal queries and delivery service offerings. The training data has a knowledge cutoff of 2023, and some restaurants may have closed or relocated. There is no guarantee that dietary restrictions or food allergy requests will generate accurate responses; users should exercise their best judgment. However, the universe is the limit when it comes to having fun discovering unique meals that can be delivered to your home.

References

@misc{lewis2020retrievalaugmented,
  title  = {Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks},
  author = {Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio
            and Karpukhin, Vladimir and Goyal, Naman and Küttler, Heinrich and
            Lewis, Mike and Yih, Wen-tau and Rocktäschel, Tim and Riedel, Sebastian
            and Kiela, Douwe},
  year   = {2020},
  note   = {arXiv:2005.11401}
}

@misc{palakkode2025buildcsvrag,
  title        = {Build a Simple RAG System with CSV Files: Step‑by‑Step Guide for Beginners},
  author       = {Palakkode, Abhay},
  year         = {2025},
  howpublished = {\url{https://www.machinelearningplus.com/gen-ai/build-a-simple-rag-system-with-csv-files-step-by-step-guide-for-beginners/}},
}

@misc{sakib2023ubereats,
  author       = {Ahmed Shahriar Sakib},
  title        = {Uber Eats USA Restaurants Menus},
  year         = {2023},
  url          = {https://www.kaggle.com/datasets/ahmedshahriarsakib/uber-eats-usa-restaurants-menus},
  note         = {Accessed: 2023-12-14}
}