Spaces:
Sleeping
Sleeping
Remove notebook_version folder from git tracking
Browse files
notebook_version/AB_Testing_RAG_Agent.ipynb
DELETED
|
The diff for this file is too large to render.
See raw diff
|
|
|
notebook_version/README.md
DELETED
|
@@ -1,94 +0,0 @@
|
|
| 1 |
-
|
| 2 |
-
<p align = "center" draggable=βfalseβ ><img src="https://github.com/AI-Maker-Space/LLM-Dev-101/assets/37101144/d1343317-fa2f-41e1-8af1-1dbb18399719"
|
| 3 |
-
width="200px"
|
| 4 |
-
height="auto"/>
|
| 5 |
-
</p>
|
| 6 |
-
|
| 7 |
-
## <h1 align="center" id="heading">Session 8: Evaluating RAG with Ragas</h1>
|
| 8 |
-
|
| 9 |
-
| π€ Pre-work | π° Session Sheet | βΊοΈ Recording | πΌοΈ Slides | π¨βπ» Repo | π Homework | π Feedback |
|
| 10 |
-
|:-----------------|:-----------------|:-----------------|:-----------------|:-----------------|:-----------------|:-----------------|
|
| 11 |
-
| [Session 8: Pre-Work](https://www.notion.so/Session-8-RAG-Evaluation-and-Assessment-1c8cd547af3d81d08f7cf5521d0253bb?pvs=4#1c8cd547af3d816583d6c23183b6f87f) | [Session 8: RAG Evaluation and Assessment](https://www.notion.so/Session-8-RAG-Evaluation-and-Assessment-1c8cd547af3d81d08f7cf5521d0253bb) | Coming soon! | [Session 8 Slides](https://www.canva.com/design/DAGjadKGqcw/0Gff9K2EwbOb3lX14un3uw/edit?utm_content=DAGjadKGqcw&utm_campaign=designshare&utm_medium=link2&utm_source=sharebutton) | You are here! | [Session 8: RAG Evaluation and Assessment](https://forms.gle/ujAQLqx2ZHMWTUH79) | [AIE6 Feedback 4/24](https://forms.gle/wA7p89e6svCgjtr58) |
|
| 12 |
-
|
| 13 |
-
In today's assignment, we'll be creating Synthetic Data, and using it to benchmark (and improve) a LCEL RAG Chain.
|
| 14 |
-
|
| 15 |
-
- π€ Breakout Room #1
|
| 16 |
-
1. Task 1: Installing Required Libraries
|
| 17 |
-
2. Task 2: Set Environment Variables
|
| 18 |
-
3. Task 3: Synthetic Dataset Generation for Evaluation using Ragas
|
| 19 |
-
4. Task 4: Evaluating our Pipeline with Ragas
|
| 20 |
-
5. Task 6: Making Adjustments and Re-Evaluating
|
| 21 |
-
|
| 22 |
-
The notebook Colab link is located [here](https://colab.research.google.com/drive/1-t4POIFJI-SWF1lmoBOPETZZqgWCTV4Y?usp=sharing)
|
| 23 |
-
|
| 24 |
-
- π€ Breakout Room #2
|
| 25 |
-
1. Task 1: Building a ReAct Agent with Metal Price Tool
|
| 26 |
-
2. Task 2: Implementing the Agent Graph Structure
|
| 27 |
-
3. Task 3: Converting Agent Messages to Ragas Format
|
| 28 |
-
4. Task 4: Evaluating Agent Performance using Ragas Metrics
|
| 29 |
-
- Tool Call Accuracy
|
| 30 |
-
- Agent Goal Accuracy
|
| 31 |
-
- Topic Adherence
|
| 32 |
-
|
| 33 |
-
The notebook Colab link is located [here](https://colab.research.google.com/drive/1KQm7nA_zTaCyjaAeAacjqanMPv03um7T?usp=sharing)
|
| 34 |
-
|
| 35 |
-
## Ship π’
|
| 36 |
-
|
| 37 |
-
The completed notebook!
|
| 38 |
-
|
| 39 |
-
<details>
|
| 40 |
-
<summary>π§ BONUS CHALLENGE π§ (OPTIONAL)</summary>
|
| 41 |
-
|
| 42 |
-
> NOTE: Completing this challenge will provide full marks on the assignment, regardless of the completion of the notebook. You do not need to complete this in the notebook for full marks.
|
| 43 |
-
|
| 44 |
-
##### **MINIMUM REQUIREMENTS**:
|
| 45 |
-
|
| 46 |
-
1. Baseline `LangGraph RAG` Application using `NAIVE RETRIEVAL`
|
| 47 |
-
2. Baseline Evaluation using `RAGAS METRICS`
|
| 48 |
-
- [Faithfulness](https://docs.ragas.io/en/stable/concepts/metrics/faithfulness.html)
|
| 49 |
-
- [Answer Relevancy](https://docs.ragas.io/en/stable/concepts/metrics/answer_relevance.html)
|
| 50 |
-
- [Context Precision](https://docs.ragas.io/en/stable/concepts/metrics/context_precision.html)
|
| 51 |
-
- [Context Recall](https://docs.ragas.io/en/stable/concepts/metrics/context_recall.html)
|
| 52 |
-
- [Answer Correctness](https://docs.ragas.io/en/stable/concepts/metrics/answer_correctness.html)
|
| 53 |
-
3. Implement a `SEMANTIC CHUNKING STRATEGY`.
|
| 54 |
-
4. Create an `LangGraph RAG` Application using `SEMANTIC CHUNKING` with `NAIVE RETRIEVAL`.
|
| 55 |
-
5. Compare and contrast results.
|
| 56 |
-
|
| 57 |
-
##### **SEMANTIC CHUNKING REQUIREMENTS**:
|
| 58 |
-
|
| 59 |
-
Chunk semantically similar (based on designed threshold) sentences, and then paragraphs, greedily, up to a maximum chunk size. Minimum chunk size is a single sentence.
|
| 60 |
-
|
| 61 |
-
Have fun!
|
| 62 |
-
</details>
|
| 63 |
-
|
| 64 |
-
### Deliverables
|
| 65 |
-
|
| 66 |
-
- A short Loom of the notebook, and a 1min. walkthrough of the application in full
|
| 67 |
-
|
| 68 |
-
## Share π
|
| 69 |
-
|
| 70 |
-
Make a social media post about your final application!
|
| 71 |
-
|
| 72 |
-
### Deliverables
|
| 73 |
-
|
| 74 |
-
- Make a post on any social media platform about what you built!
|
| 75 |
-
|
| 76 |
-
Here's a template to get you started:
|
| 77 |
-
|
| 78 |
-
```
|
| 79 |
-
π Exciting News! π
|
| 80 |
-
|
| 81 |
-
I am thrilled to announce that I have just built and shipped Synthetic Data Generation, benchmarking, and iteration with RAGAS & LangChain! ππ€
|
| 82 |
-
|
| 83 |
-
π Three Key Takeaways:
|
| 84 |
-
1οΈβ£
|
| 85 |
-
2οΈβ£
|
| 86 |
-
3οΈβ£
|
| 87 |
-
|
| 88 |
-
Let's continue pushing the boundaries of what's possible in the world of AI and question-answering. Here's to many more innovations! π
|
| 89 |
-
Shout out to @AIMakerspace !
|
| 90 |
-
|
| 91 |
-
#LangChain #QuestionAnswering #RetrievalAugmented #Innovation #AI #TechMilestone
|
| 92 |
-
|
| 93 |
-
Feel free to reach out if you're curious or would like to collaborate on similar projects! π€π₯
|
| 94 |
-
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
notebook_version/pyproject.toml
DELETED
|
@@ -1,16 +0,0 @@
|
|
| 1 |
-
[project]
|
| 2 |
-
name = "08-evaluating-rag-with-ragas"
|
| 3 |
-
version = "0.1.0"
|
| 4 |
-
description = "Add your description here"
|
| 5 |
-
readme = "README.md"
|
| 6 |
-
requires-python = ">=3.13"
|
| 7 |
-
dependencies = [
|
| 8 |
-
"jupyter>=1.1.1",
|
| 9 |
-
"langchain-community==0.3.14",
|
| 10 |
-
"langchain-openai==0.2.14",
|
| 11 |
-
"langchain-qdrant>=0.2.0",
|
| 12 |
-
"langgraph==0.2.61",
|
| 13 |
-
"numpy>=2.2.2",
|
| 14 |
-
"unstructured>=0.14.8",
|
| 15 |
-
"arxiv>=1.4.0",
|
| 16 |
-
]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
notebook_version/uv.lock
DELETED
|
The diff for this file is too large to render.
See raw diff
|
|
|