READTHIS.md

<<<<<<< HEAD
---
title: RAG Wikipedia
emoji: 😊
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: "5.44.1"
python_version: "3.10"
app_file: app.py
pinned: false
---

# **[RAG](https://www.databricks.com/blog/what-is-retrieval-augmented-generation) (with [Wikipedia Corpus](https://www.wikipedia.org/))**

HuggingFace RAG application that answers questions using retrieved Wikipedia passages (with supporting evidence)

## **ML (model) Stack:**

* **Generator:** [`Qwen/Qwen3-4B-Instruct-2507`](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507)
* **Dataset:** [`rag-datasets/rag-mini-wikipedia`](https://huggingface.co/datasets/rag-datasets/rag-mini-wikipedia)
* **Dataset configuration:** `text-corpus`
* **Dataset split:** `passages`
* **Embedding model:** [`sentence-transformers/all-MiniLM-L6-v2`](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)
* **Keyword retrieval:** [TF-IDF](https://www.geeksforgeeks.org/machine-learning/understanding-tf-idf-term-frequency-inverse-document-frequency/) with [unigram](https://huggingface.co/learn/llm-course/en/chapter6/7) and [bigram](https://www.geeksforgeeks.org/nlp/generate-bigrams-with-nltk/) features
* **Reranker:** [`cross-encoder/ms-marco-MiniLM-L6-v2`](https://huggingface.co/cross-encoder/ms-marco-MiniLM-L6-v2)
* **Generation API:** Hugging Face `InferenceClient`
* **Interface:** [Gradio Blocks](https://gradio.app/docs/gradio/blocks)

## **Features (app workflow)**

* loads the Wikipedia dataset from Hugging Face
* split dataset rows into overlapping text chunks
* creates embeddings for semantic retrieval
* creates a TF-IDF index for keyword retrieval
* combines semantic and keyword scores
* (optionally) reranks passages with a CrossEncoder
* sends the strongest passages to Qwen
* generates answers (with numbered citations)
* displays retrieval and pipeline diagnostics
* includes suggested questions
* caches the dataset, embeddings, vectorizer, and sparse matrix
* supports up to four queued requests at once

## **Interface Features**

### **Retrieved Evidence**

**Retrieved evidence** panel shows the passages used to support the answer

each source card contains:

| Output            | Meaning                                    |
| ----------------- | ------------------------------------------ |
| **Source number** | citation number used in the answer         |
| **Passage title** | show name for the retrieved passage     |
| **Score**         | ranking score from retrieval or reranking  |
| **Passage text**  | exact text retrieved from the dataset      |
| **Source ID**     | original dataset row or passage identifier |

Example:

```text
1
Wikipedia passage 362
9.686

The Legal Tender Act of 1862 established the United States Note,
the first paper currency in United States history. This was done
to increase the money supply to pay for fighting the war

Source ID: 361
```

the answer citation `[1]` refers to this source

## **Retrieval Score (meaning)**

displayed score is used to order passages

higher score means the passage ranked more strongly for the current question

score is not:

* a probability
* a percentage
* a factual confidence score
* a measure of answer correctness

[CrossEncoder](https://sbert.net/examples/cross_encoder/applications/README.html) scores can be positive or negative

negative score does not automatically (always) mean that a passage is invalid **it usually means the passage ranked less strongly than other candidates.**

**scores should only be compared with other passages from the same request and retrieval configuration**

## **Retrieval Controls**

### **Sources Returned**

- Controls how many passages are returned after retrieval and reranking

- Higher values provide more context but may also add unrelated information


| Setting | Value |
| ------- | ----: |
| Default |   `5` |
| Minimum |   `3` |
| Maximum |   `8` |
| Step    |   `1` |


### **Semantic Retrieval Weight**
- controls the balance between semantic retrieval and TF-IDF keyword retrieval


| Setting |  Value |
| ------- | -----: |
| Default | `0.72` |
| Minimum | `0.00` |
| Maximum | `1.00` |
| Step    | `0.05` |


#### **Semantic Retrieval Weight Formula:**
```text
TF-IDF weight = 1 - semantic retrieval weight
```

Examples:

| Semantic weight | TF-IDF weight | Behavior                             |
| --------------: | ------------: | ------------------------------------ |
|          `1.00` |        `0.00` | Semantic retrieval only              |
|          `0.72` |        `0.28` | Default hybrid retrieval             |
|          `0.50` |        `0.50` | Equal semantic and keyword weighting |
|          `0.00` |        `1.00` | TF-IDF keyword retrieval only        |

- Semantic retrieval is useful when the question and passage use different wording

- TF-IDF is useful when the question contains exact names, dates, phrases, or technical terms

### **Use Neural Reranker**
- (when enabled), the CrossEncoder compares the full question with each retrieved candidate and reorders the passages

- note: disabling the reranker can reduce processing time, but retrieval quality may decrease


| Setting             | Default |
| ------------------- | ------- |
| Enabled or disabled | Enabled |


## **Generation Controls**

### **Temperature:**

- controls variation in generated text

| Setting |  Value |
| ------- | -----: |
| Default | `0.20` |
| Minimum | `0.00` |
| Maximum | `1.00` |
| Step    | `0.05` |


|      Temperature | Expected behavior                                           |
| ---------------: | ----------------------------------------------------------- |
| `0.00` to `0.20` | more consistent and direct                                  |
| `0.25` to `0.50` | moderate variation                                          |
|     Above `0.50` | more varied output with greater risk of unsupported content |

- lower values are recommended for factual RAG answers

### **Maximum Output Tokens**

| Setting |  Value |
| ------- | -----: |
| Default |  `700` |
| Minimum |  `200` |
| Maximum | `1200` |
| Step    |   `50` |

- limits the maximum generated answer length

- (this setting does not control the number of retrieved passages)

## **Diagnostics:**

**diagnostics** panel shows JSON returned by the RAG engine

can be used to inspect:

* pipeline status
* retrieval settings
* number of retrieved passages
* retrieval results
* reranker results
* generation settings
* processing details

 exact fields depend on the values returned by `RAGEngine.diagnostics()`


## **App Inputs:**

| Input                     | Data Type    | Default |           Range | Purpose                                       |
| ------------------------- | ------- | ------: | --------------: | --------------------------------------------- |
| Question                  | Text    |   Empty |   Up to 5 lines | question sent to the RAG pipeline             |
| Sources returned          | Integer |     `5` |      `3` to `8` | number of passages returned                   |
| Semantic retrieval weight | Float   |  `0.72` |  `0.0` to `1.0` | balance between semantic retrieval and TF-IDF |
| Use neural reranker       | Boolean |  `True` |       On or off | enables CrossEncoder reranking                |
| Temperature               | Float   |  `0.20` |  `0.0` to `1.0` | controls generation variation                 |
| Maximum output tokens     | Integer |   `700` | `200` to `1200` | limits answer length                          |

## **App Outputs:**

each successful request updates four outputs

### **1. Retrieved Evidence:**

- displays the passages selected by the retrieval pipeline

### **2. Diagnostics:**

- displays pipeline information as JSON

### **3. Question Field:**

- clears the submitted question

## **Request Flow:**

when a question is submitted, the RAG app:

1. removes leading and trailing whitespace

2. checks whether the question is empty

3. initializes the RAG engine if needed

4. reads the current conversation history

5. runs semantic retrieval

6. runs TF-IDF keyword retrieval

7. combines the retrieval scores

8. applies the CrossEncoder reranker when enabled

9. selects the requested number of sources

10. sends the selected passages and chat history to Qwen

11. generates a citation-based answer

12. adds question and answer to the conversation

13. shows the retrieved source cards


## **How Retrieval Works:**

1. the app loads the Wikipedia dataset

2. dataset rows are cleaned and split into chunks

3. MiniLM creates an embedding for each chunk

4. TF-IDF creates a keyword index

5. the user question is compared with both indexes

6. semantic and TF-IDF scores are combined

7. the strongest candidates are selected

8. the CrossEncoder reranks the candidates when enabled

9. the highest-ranked passages are sent to Qwen

10. the generated answer and supporting passages are displayed


## **Limits:**

* wikipedia corpus (small amount of data)

* free HFSpace restarts may require me to manually rebuild the cache

* some dataset text (wikipedia corpus) may have character encoding errors
=======
---
title: RAG Wikipedia
emoji: 🐠
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 6.22.0
python_version: '3.12'
app_file: app.py
pinned: false
license: apache-2.0
short_description: RAG with Wikipedia Corpus
---

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
>>>>>>> space/main