README.md

---
license: apache-2.0
language:
- en
pipeline_tag: text-generation
base_model:
- Qwen/Qwen3-4B-Instruct-2507
tags:
- rag
- retrieval-augmented-generation
- boardgame
- boardgame-rules
short_description: RAG Enabled ChatBot for Board Game Rulebooks
---

# Board Game Rules Assistant

## 1. Introduction
Board game rulebooks are often lengthy and complex, with important rules distributed across multiple sections. As a result, learning a new game and explaining its setup
and rules to new players can be a significant barrier to gameplay. While current Large Language Models (LLMs) have likely seen board game rulebooks while training, they
tend to conflate rules from different games or hallucinate rules when asked specified questions about gameplay. To solve this, I developed the **Board Game Rules Assistant**,
a system designed specifically to answer questions about board game set up, general gameplay, and specific rules, grounded only in the official game rulebooks. This project
utilizes a hierarchical parent-child Retrieval-Augmented Generation (RAG) pipeline to ensure response accuracy by retrieving specific rules from official board game rulebook
PDFs and using those rules to answer user questions. In addition to the RAG system itself, this repository includes scripts for parsing PFDs into Markdown files and chunking
Markdown file based on Markdown headings (levels 2-4).

Overall, the results show that grounding responses in source text can improve response accuracy and reduce hallucinations compared with using a generation model alone.

## 2. Data
The knowledge base for this RAG pipeline consists of text extracted from official board game rulebook PDFs using the included parsing script. These rulebooks were obtained
directly from the publishers’ websites. After parsing, the resulting Markdown files were manually validated to ensure the text accurately reflected the original documents.
Other than minor formatting changes made to preserve the structure of the PDFs in Markdown, the text was not edited or rephrased. The current knowledge base contains
rulebooks for the following games: *Betrayal at House on the Hill* (2nd Edition), *Blokus*, *Catan*, *Clue*, *Codenames*, *Hues and Cues*, *Monopoly*, *Really Loud Librarian*,
*Ticket to Ride*, and *Wingspan*. 

**Copyright notice:** The rulebooks are copyrighted by their respective publishers and are included solely as the retrieval corpus for this project. Ownership of the
underlying game content remains with the respective copyright holders.

To evaluate the RAG pipeline, I created a custom evaluation dataset consisting of 50 question-answer pairs derived from the official rule books. The dataset contains five
questions for each game in the knowledge base, covering topics ranging from specific gameplay rules to general setup and gameplay overviews. It also includes unanswerable
questions, whose answers were intentionally absent from the knowledge base, to evaluate the model’s ability to refrain from answering when sufficient information is
unavailable. I authored all evaluation questions and most of the reference answers. For the general gameplay overview questions, I provided the relevant rulebook chunks to
OpenAI's ChatGPT (GPT-5.5) using the prompt: *“Using the provided context only, answer the question: how do I play `<board game name>`?”* All generated responses were
manually reviewed and edited for factual accuracy, clarity, and conciseness before being included in the evaluation dataset.

Due to copyright and data licensing restrictions, the knowledge base and evaluation dataset are not included in this repository. Please see Section 9: Data Licensing for
more information.

## 3. Methodology
This project implements a hierarchical parent-child RAG pipeline to leverage the natural hierarchical structure of board game rulebooks. Each rulebook is first divided into
larger parent chunks based on Markdown headings (levels 2-4). Parent chunks that exceed a predefined size threshold are then recursively split into smaller child chunks
while preserving metadata that links each child to its corresponding parent. 

Only the child chunks are embedded in the vector database. During retrieval, the user query is compared against the child chunk using cosine similarity with k=5 to identify
the most relevant passages. The corresponding parent chunk or chunks are then retrievethrough the metadata links and provided to the generation model as context.

This approach allows semantic search to operate over fine-grained passages while avoiding the computational cost of embedding large parent documents. At the same time, the
generation model receives broader contextual information than would be available from the retrieved child chunks alone, increasing the likelihood of producing coherent and
well-grounded responses.

I selected **[Qwen3-Embedding-4B](https://huggingface.co/Qwen/Qwen3-Embedding-4B)** as the embedding model because I found it achieved higher scores on context relevance,
context recall, and answer relevance compared to the other embedding model I tested.

I selected **[Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507)** as the generation model because during preliminary benchmarking, it outperformed
the larger **[Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct)** model while requiring fewer computational resources.

## 4. Evaluation
To evaluate the retrieval performance and response quality of the RAG pipeline, I utilized three established benchmarks: [RAGBench](https://huggingface.co/datasets/galileo-ai/ragbench),
[LaRA](https://github.com/Alibaba-NLP/LaRA/tree/main), and [RAGTruth](https://github.com/ParticleMedia/RAGTruth), in
addition to my own custom dataset. I selected these datasets because they target common failure reasons of RAG systems: retrieving irrelevant context, failing to retrieve
relevant information, and generating hallucinated or unsupported responses.

 **I. RAGBench:** I used the test split of the DelucionQA dataset. This dataset was developed to evaluate hallucinations and consists of question-answer pairs based on the
 *Jeep Gladiator 2023 Owner’s Manual*. Unlike a traditional RAG benchmark, the dataset provides the relevant context directly in each example rather than as a document
 corpus and does not include ground truth answers. I modified the dataset only by removing columns that were not used during evaluation and renaming columns for consistency
 across benchmarks.

 **II. LaRA:** I used the 32k paper corpus together with all four 32k_paper query files. To evaluate retrieval under the same conditions as my board game knowledge base,
 I processed the document corpus using the chunking strategy described in Section 2. The query files were loaded into Pandas DataFrames, and I added a source_file column
 to preserve the mapping between each query and its original Markdown document.

 **III. RAGTruth:** I used the QA tasks from the source_info.json file. This dataset stores the question and supporting passages together and includes a prompt column
 containing dataset-specific instructions. To use RAGTruth within a consistent evaluation pipeline, I separated the question from the supporting passages and replaced the
 dataset prompt with the same prompt used for the other benchmarks. Like RAGBench, RAGTruth does not include a ground truth response.

All datasets were evaluated using gpt-4o-mini (OpenAI) as the judge for several metrics from the RAGAS framework. 

- Context Relevance: Measures whether the retrieved context is relevant to the user’s query.
- Context Recall: Measures how much of the information required to answer the query is contained in the retrieved context.
- Faithfulness: Measures whether the generated answer is supported by the retrieved context.
- Answer Relevance: Measures how well the generated answer addresses the user's query.
- Answer Correctness: Measures how closely the generated answer matches the reference answer.

Because the three benchmarks differ in the data they provide, not every RAGAS metric was computed for every dataset. Specifically, Answer Correctness was only evaluated on
datasets containing reference answers. Benchmarks such as DelucionQA, which do not provide ground-truth answers, were evaluated using only the metrics that could be computed
from the available date. This ensured that each benchmark was evaluated using metrics appropriate to its design while maintaining a consistent evaluation pipeline.

<table>
  <thead>
    <tr>
      <th>Benchmark</th>
      <th colspan="3">LaRA</th>
      <th colspan="3">Board Game Rules</th>
      <th colspan="3">RAGBench</th>
      <th colspan="3">RAGTruth</th>
    </tr>
    <tr>
      <th>Model</th>
      <th>Qwen</th><th>Llama</th><th>Microsoft</th>
      <th>Qwen</th><th>Llama</th><th>Microsoft</th>
      <th>Qwen</th><th>Llama</th><th>Microsoft</th>
      <th>Qwen</th><th>Llama</th><th>Microsoft</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Faithfulness</td>
      <td>80.08</td><td>75.05</td><td align="center"><b style="color:#000000;">84.68</b></td>
      <td><b style="color:#000000;">89.67</b></td><td>81.79</td><td align="center">78.26</td>
      <td><b style="color:#000000;">93.72</b></td><td>89.39</td><td align="center">92.67</td>
      <td><b style="color:#000000;">91.73</b></td><td>84.62</td><td align="center">90.09</td>
    </tr>
    <tr>
      <td>Answer Relevance</td>
      <td>52.06</td><td>47.98</td><td align="center"><b style="color:#000000;">54.06</b></td>
      <td><b style="color:#000000;">66.99</b></td><td>56.55</td><td align="center">62.09</td>
      <td><b style="color:#000000;">78.12</b></td><td>77.45</td><td align="center">75.79</td>
      <td><b style="color:#000000;">65.51</b></td><td>55.83</td><td align="center">59.40</td>
    </tr>
    <tr>
      <td>Context Relevance</td>
      <td>67.44</td><td><b style="color:#000000;">68.83</b></td><td align="center">66.97</td>
      <td>80.65</td><td><b style="color:#000000;">81.26</b></td><td align="center">79.85</td>
      <td colspan="3" style="background-color:#a6a6a6;"></td>
      <td colspan="3" style="background-color:#a6a6a6;"></td>
    </tr>
    <tr>
      <td>Context Recall</td>
      <td>78.89</td><td><b style="color:#000000;">81.32</b></td><td align="center">79.54</td>
      <td>71.46</td><td><b style="color:#000000;">72.49</b></td><td align="center">71.03</td>
      <td colspan="3" style="background-color:#a6a6a6;"></td>
      <td colspan="3" style="background-color:#a6a6a6;"></td>
    </tr>
    <tr>
      <td>Answer Correctness</td>
      <td>47.17</td><td>47.49</td><td align="center"><b style="color:#000000;">47.99</b></td>
      <td><b style="color:#000000;">47.57</b></td><td>45.18</td><td align="center">38.80</td>
      <td colspan="3" style="background-color:#a6a6a6;"></td>
      <td colspan="3" style="background-color:#a6a6a6;"></td>
    </tr>
  </tbody>
</table>


To further evaluate performance, I compared my primary model, Qwen3-4B-Instruct-2507, against two similarly sized models,
[Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)) and [Phi-3.5-mini-instruct](https://huggingface.co/microsoft/Phi-3.5-mini-instruct).
The results in the table above show all three models performed comparably on retrieval metrics, likely because they used the same embedding model and similarity search
strategy. However, Qwen3-4B-Instruct-2507 consistently performed better on the generation metrics, achieving a higher score for faithfulness, answer relevance, and answer
completeness on most of the benchmarks. 

## 5. Usage and Intended Uses
The intended use case for this model is to assist board game players in understanding specific rules or providing a general overview of the game which can be used to start
the game playing session.

Although the model was designed to be utilized with a document corpus, it does support the ability for a user to enter relevant context as text to be used when generating
the response. In this scenario, the input text is not chunked but supplied directly to the generation model along with the query. This mode is automatically enabled if a
BoardGameRag instance is constructed without children and parent arguments

Below is an example of how the RAG pipeline can be used from parsing to response generation:

```{python}
from pathlib import Path
from parse_rulebooks import parse
from chunk_rulebooks import read_markdown, split_section, split_subsections, split chunks
from boardgame_rag import BoardGameRag

# Parse a rulebook PDF into Markdown
parsed_path = parse(file_name = "catan.pdf", out_name = "catan", ocr = False)

# Chunk the Markdown into parent sections and child chunks
markdown_text = read_markdown("catan.md", parsed_dir = Path("parsed"))
sections = split_sections(markdown_text, game = "Catan", source_file = "catan.md")

parent_docs = []
child_docs = []

for section in sections:
  subections = split.subsections(section)
  parent_docs.extend(subsections):
  for subsection in subsections:
    child_docs.extend(split_chunks(subsection))

# Initialize the RAG pipeline
rag = BoardGameRag(
        model_name = "Qwen/Qwen3-4B-Instruct-2507"),
        tokenizer_name = None,       # uses the same model as model_name
        children = child_docs,
        parents = parent_docs,
        max_new_tokens = 300,
        do_sample = False,
        persist_directory = None
)

# Ask a question
result = rag.generate_answer(
  question = "How many resources do I need to build a settlement?",
  prompt = None        # uses the default rules-QA prompt
)

print(result["answer"])
```

Below is an example of how to generate a response when there is no document corpus

```{python}

# Context is stored as a list of strings
context = ["To build a settlement, a player must pay one brick, one wood, "
            "one wheat, and one sheep. A settlement may only be built on an "
            "unoccupied intersection that is not adjacent to another settlement "
            "or city."
]

# Initialize the RAG pipeline with no children or parents
rag = BoardGameRag(
        model_name = "Qwen/Qwen3-4B-Instruct-2507"),
        tokenizer_name = None,       # uses the same model as model_name
        max_new_tokens = 300,
        do_sample = False,
        persist_directory = None

result = rag.generate_from_context(
  question = "How many resources do I need to build a settlement?"),
  prompt = None,
  context = context
)

print(result)
```

## 6. Prompt Format
This model relies on a formatted user prompt instead of a system prompt. The prompt injects the retrieved RAG context directly into the user message so when generating an
answer, the user only has to provide the text of the prompt
If no prompt is provided during initialization, the system uses the default board game prompt.

```
You are a helpful assistant answering questions about how to play board games. 
Answer the user's question thoroughly using only the provided context. 
If the answer is not directly supported by the context you must say that you cannot answer the question with the information provided. 
If the answer differs across game modes, variants, or optional rules described in the context, summarize the differences and specify which version each applies to. 
Do not provide gameplay tips or strategies unless they are explicitly mentioned in the context.

CONTEXT
{context}

QUESTION
{question}

ANSWER:
```

If a prompt is provided during initialization, the instruction text changes but the context, question, and answer remains the same.

```
You are a helpful assistant answering questions using only the context provided.
If the answer is not directly supported by the context you must say that you cannot answer the question with the information provided.

CONTEXT
{context}

QUESTION
{question}

ANSWER:

```
## 7. Expected Output Format
The model is expected to generate a plain-English response to the user's question using only information retrieved from the knowledge base. If the requested information
is unavailable or falls outside the scope of the knowledge base (e.g., strategy questions), the model should refuse to answer. The exact wording and formatting of
responses may vary because the generation model was not further fine-tuned.

## 8. Limitations
The primary limitation of this RAG pipeline is that it is limited to the board games and rulebook editions included in its knowledge base and cannot answer questions about
games or rules outside that corpus, including unofficial rulings, FAQs, or gameplay strategies. Additionally, while the chunking strategy was designed to leverage the
hierarchical structure of board game rulebooks, not all rulebooks follow the same document structure, so the chunking strategy may not generalize well to other document
collections. Because the system relies on retrieval, retrieval failures may result in incomplete responses or appropriate refusals to answer. Finally, because the
generation model was not fine-tuned for this task, the wording and formatting of responses may vary.

## 9. Data Licensing
This project uses official board game rulebooks obtained from publisher websites. These documents remain the intellectual property of their respective publishers and are
not released under this repository's license. Users are responsible for ensuring they have the appropriate rights to use or redistribute the rulebooks.