Spaces:
Sleeping
Sleeping
File size: 71,877 Bytes
7a56e2a |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 |
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"application/vnd.databricks.v1+cell": {
"cellMetadata": {},
"inputWidgets": {},
"nuid": "42084110-295b-493a-9b3e-5d8d29ff78b3",
"showTitle": false,
"title": ""
}
},
"source": [
"# LLM RAG Evaluation with MLflow Example Notebook\n",
"\n",
"In this notebook, we will demonstrate how to evaluate various a RAG system with MLflow."
]
},
{
"cell_type": "raw",
"metadata": {},
"source": [
"<a href=\"https://raw.githubusercontent.com/mlflow/mlflow/master/docs/source/llms/llm-evaluate/notebooks/rag-evaluation.ipynb\" class=\"notebook-download-btn\"><i class=\"fas fa-download\"></i>Download this Notebook</a>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We need to set our OpenAI API key.\n",
"\n",
"In order to set your private key safely, please be sure to either export your key through a command-line terminal for your current instance, or, for a permanent addition to all user-based sessions, configure your favored environment management configuration file (i.e., .bashrc, .zshrc) to have the following entry:\n",
"\n",
"`OPENAI_API_KEY=<your openai API key>`\n",
"\n",
"If using Azure OpenAI, you will instead need to set\n",
"\n",
"`OPENAI_API_TYPE=\"azure\"`\n",
"\n",
"`OPENAI_API_VERSION=<YYYY-MM-DD>`\n",
"\n",
"`OPENAI_API_KEY=<https://<>.<>.<>.com>`\n",
"\n",
"`OPENAI_API_DEPLOYMENT_NAME=<deployment name>`\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"application/vnd.databricks.v1+cell": {
"cellMetadata": {
"byteLimit": 2048000,
"rowLimit": 10000
},
"inputWidgets": {},
"nuid": "fb946228-62fb-4d68-9732-75935c9cb401",
"showTitle": false,
"title": ""
}
},
"outputs": [],
"source": [
"import pandas as pd\n",
"\n",
"import mlflow\n",
"import os\n",
"os.environ[\"OPENAI_API_KEY\"] =\"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\"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"application/vnd.databricks.v1+cell": {
"cellMetadata": {},
"inputWidgets": {},
"nuid": "273d1345-95d7-435a-a7b6-a5f3dbb3f073",
"showTitle": false,
"title": ""
}
},
"source": [
"## Create a RAG system\n",
"\n",
"Use Langchain and Chroma to create a RAG system that answers questions based on the MLflow documentation."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"application/vnd.databricks.v1+cell": {
"cellMetadata": {
"byteLimit": 2048000,
"rowLimit": 10000
},
"inputWidgets": {},
"nuid": "2c28d0ad-f469-46ab-a2b4-c5e8db50a729",
"showTitle": false,
"title": ""
}
},
"outputs": [],
"source": [
"from langchain.chains import RetrievalQA\n",
"from langchain.document_loaders import WebBaseLoader\n",
"from langchain.embeddings.openai import OpenAIEmbeddings\n",
"from langchain.llms import OpenAI\n",
"from langchain.text_splitter import CharacterTextSplitter\n",
"from langchain.vectorstores import Chroma\n",
"from langchain.chat_models import ChatOpenAI\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"application/vnd.databricks.v1+cell": {
"cellMetadata": {
"byteLimit": 2048000,
"rowLimit": 10000
},
"inputWidgets": {},
"nuid": "83a7e77e-6717-472a-86dc-02e2c356ddef",
"showTitle": false,
"title": ""
}
},
"outputs": [],
"source": [
"loader = WebBaseLoader(\"https://mlflow.org/docs/latest/index.html\")\n",
"\n",
"documents = loader.load()\n",
"text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
"texts = text_splitter.split_documents(documents)\n",
"\n",
"embeddings = OpenAIEmbeddings()\n",
"docsearch = Chroma.from_documents(texts, embeddings)\n",
"\n",
"qa = RetrievalQA.from_chain_type(\n",
" llm=ChatOpenAI(model_name=\"gpt-3.5-turbo-0125\" , temperature=0),\n",
" chain_type=\"stuff\",\n",
" retriever=docsearch.as_retriever(),\n",
" return_source_documents=True,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"application/vnd.databricks.v1+cell": {
"cellMetadata": {},
"inputWidgets": {},
"nuid": "fd70bcf6-7c44-44d3-9435-567b82611e1c",
"showTitle": false,
"title": ""
}
},
"source": [
"## Evaluate the RAG system using `mlflow.evaluate()`"
]
},
{
"cell_type": "markdown",
"metadata": {
"application/vnd.databricks.v1+cell": {
"cellMetadata": {},
"inputWidgets": {},
"nuid": "de1bc359-2e40-459c-bea4-bed35a117988",
"showTitle": false,
"title": ""
}
},
"source": [
"Create a simple function that runs each input through the RAG chain"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"application/vnd.databricks.v1+cell": {
"cellMetadata": {
"byteLimit": 2048000,
"rowLimit": 10000
},
"inputWidgets": {},
"nuid": "667ec809-2bb5-4170-9937-6804386b41ec",
"showTitle": false,
"title": ""
}
},
"outputs": [],
"source": [
"def model(input_df):\n",
" answer = []\n",
" for index, row in input_df.iterrows():\n",
" answer.append(qa(row[\"questions\"]))\n",
"\n",
" return answer"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"application/vnd.databricks.v1+cell": {
"cellMetadata": {},
"inputWidgets": {},
"nuid": "d1064306-b7f3-4b3e-825c-4353d808f21d",
"showTitle": false,
"title": ""
}
},
"source": [
"Create an eval dataset"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"application/vnd.databricks.v1+cell": {
"cellMetadata": {
"byteLimit": 2048000,
"rowLimit": 10000
},
"inputWidgets": {},
"nuid": "a5481491-e4a9-42ea-8a3f-f527faffd04d",
"showTitle": false,
"title": ""
}
},
"outputs": [],
"source": [
"eval_df = pd.DataFrame(\n",
" {\n",
" \"questions\": [\n",
" \"What is MLflow?\",\n",
" \"How to run mlflow.evaluate()?\",\n",
" \"How to log_table()?\",\n",
" \"How to load_table()?\",\n",
" ],\n",
" }\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"application/vnd.databricks.v1+cell": {
"cellMetadata": {},
"inputWidgets": {},
"nuid": "9c3c8023-8feb-427a-b36d-34cd1853a5dc",
"showTitle": false,
"title": ""
}
},
"source": [
"Create a faithfulness metric"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"application/vnd.databricks.v1+cell": {
"cellMetadata": {
"byteLimit": 2048000,
"rowLimit": 10000
},
"inputWidgets": {},
"nuid": "3882b940-9c25-41ce-a301-72d8c0c90aaa",
"showTitle": false,
"title": ""
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"EvaluationMetric(name=faithfulness, greater_is_better=True, long_name=faithfulness, version=v1, metric_details=\n",
"Task:\n",
"You must return the following fields in your response in two lines, one below the other:\n",
"score: Your numerical score for the model's faithfulness based on the rubric\n",
"justification: Your reasoning about the model's faithfulness score\n",
"\n",
"You are an impartial judge. You will be given an input that was sent to a machine\n",
"learning model, and you will be given an output that the model produced. You\n",
"may also be given additional information that was used by the model to generate the output.\n",
"\n",
"Your task is to determine a numerical score called faithfulness based on the input and output.\n",
"A definition of faithfulness and a grading rubric are provided below.\n",
"You must use the grading rubric to determine your score. You must also justify your score.\n",
"\n",
"Examples could be included below for reference. Make sure to use them as references and to\n",
"understand them before completing the task.\n",
"\n",
"Input:\n",
"{input}\n",
"\n",
"Output:\n",
"{output}\n",
"\n",
"{grading_context_columns}\n",
"\n",
"Metric definition:\n",
"Faithfulness is only evaluated with the provided output and provided context, please ignore the provided input entirely when scoring faithfulness. Faithfulness assesses how much of the provided output is factually consistent with the provided context. A higher score indicates that a higher proportion of claims present in the output can be derived from the provided context. Faithfulness does not consider how much extra information from the context is not present in the output.\n",
"\n",
"Grading rubric:\n",
"Faithfulness: Below are the details for different scores:\n",
"- Score 1: None of the claims in the output can be inferred from the provided context.\n",
"- Score 2: Some of the claims in the output can be inferred from the provided context, but the majority of the output is missing from, inconsistent with, or contradictory to the provided context.\n",
"- Score 3: Half or more of the claims in the output can be inferred from the provided context.\n",
"- Score 4: Most of the claims in the output can be inferred from the provided context, with very little information that is not directly supported by the provided context.\n",
"- Score 5: All of the claims in the output are directly supported by the provided context, demonstrating high faithfulness to the provided context.\n",
"\n",
"Examples:\n",
"\n",
"Example Input:\n",
"How do I disable MLflow autologging?\n",
"\n",
"Example Output:\n",
"mlflow.autolog(disable=True) will disable autologging for all functions. In Databricks, autologging is enabled by default. \n",
"\n",
"Additional information used by the model:\n",
"key: context\n",
"value:\n",
"mlflow.autolog(log_input_examples: bool = False, log_model_signatures: bool = True, log_models: bool = True, log_datasets: bool = True, disable: bool = False, exclusive: bool = False, disable_for_unsupported_versions: bool = False, silent: bool = False, extra_tags: Optional[Dict[str, str]] = None) → None[source] Enables (or disables) and configures autologging for all supported integrations. The parameters are passed to any autologging integrations that support them. See the tracking docs for a list of supported autologging integrations. Note that framework-specific configurations set at any point will take precedence over any configurations set by this function.\n",
"\n",
"Example score: 2\n",
"Example justification: The output provides a working solution, using the mlflow.autolog() function that is provided in the context.\n",
" \n",
"\n",
"Example Input:\n",
"How do I disable MLflow autologging?\n",
"\n",
"Example Output:\n",
"mlflow.autolog(disable=True) will disable autologging for all functions.\n",
"\n",
"Additional information used by the model:\n",
"key: context\n",
"value:\n",
"mlflow.autolog(log_input_examples: bool = False, log_model_signatures: bool = True, log_models: bool = True, log_datasets: bool = True, disable: bool = False, exclusive: bool = False, disable_for_unsupported_versions: bool = False, silent: bool = False, extra_tags: Optional[Dict[str, str]] = None) → None[source] Enables (or disables) and configures autologging for all supported integrations. The parameters are passed to any autologging integrations that support them. See the tracking docs for a list of supported autologging integrations. Note that framework-specific configurations set at any point will take precedence over any configurations set by this function.\n",
"\n",
"Example score: 5\n",
"Example justification: The output provides a solution that is using the mlflow.autolog() function that is provided in the context.\n",
" \n",
"\n",
"You must return the following fields in your response in two lines, one below the other:\n",
"score: Your numerical score for the model's faithfulness based on the rubric\n",
"justification: Your reasoning about the model's faithfulness score\n",
"\n",
"Do not add additional new lines. Do not add any other fields.\n",
" )\n"
]
}
],
"source": [
"from mlflow.metrics.genai import EvaluationExample, faithfulness\n",
"\n",
"# Create a good and bad example for faithfulness in the context of this problem\n",
"faithfulness_examples = [\n",
" EvaluationExample(\n",
" input=\"How do I disable MLflow autologging?\",\n",
" output=\"mlflow.autolog(disable=True) will disable autologging for all functions. In Databricks, autologging is enabled by default. \",\n",
" score=2,\n",
" justification=\"The output provides a working solution, using the mlflow.autolog() function that is provided in the context.\",\n",
" grading_context={\n",
" \"context\": \"mlflow.autolog(log_input_examples: bool = False, log_model_signatures: bool = True, log_models: bool = True, log_datasets: bool = True, disable: bool = False, exclusive: bool = False, disable_for_unsupported_versions: bool = False, silent: bool = False, extra_tags: Optional[Dict[str, str]] = None) → None[source] Enables (or disables) and configures autologging for all supported integrations. The parameters are passed to any autologging integrations that support them. See the tracking docs for a list of supported autologging integrations. Note that framework-specific configurations set at any point will take precedence over any configurations set by this function.\"\n",
" },\n",
" ),\n",
" EvaluationExample(\n",
" input=\"How do I disable MLflow autologging?\",\n",
" output=\"mlflow.autolog(disable=True) will disable autologging for all functions.\",\n",
" score=5,\n",
" justification=\"The output provides a solution that is using the mlflow.autolog() function that is provided in the context.\",\n",
" grading_context={\n",
" \"context\": \"mlflow.autolog(log_input_examples: bool = False, log_model_signatures: bool = True, log_models: bool = True, log_datasets: bool = True, disable: bool = False, exclusive: bool = False, disable_for_unsupported_versions: bool = False, silent: bool = False, extra_tags: Optional[Dict[str, str]] = None) → None[source] Enables (or disables) and configures autologging for all supported integrations. The parameters are passed to any autologging integrations that support them. See the tracking docs for a list of supported autologging integrations. Note that framework-specific configurations set at any point will take precedence over any configurations set by this function.\"\n",
" },\n",
" ),\n",
"]\n",
"\n",
"faithfulness_metric = faithfulness(model=\"openai:/gpt-4\", examples=faithfulness_examples)\n",
"print(faithfulness_metric)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Create a relevance metric. You can see the full grading prompt by printing the metric or by accessing the `metric_details` attribute of the metric."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"EvaluationMetric(name=relevance, greater_is_better=True, long_name=relevance, version=v1, metric_details=\n",
"Task:\n",
"You must return the following fields in your response in two lines, one below the other:\n",
"score: Your numerical score for the model's relevance based on the rubric\n",
"justification: Your reasoning about the model's relevance score\n",
"\n",
"You are an impartial judge. You will be given an input that was sent to a machine\n",
"learning model, and you will be given an output that the model produced. You\n",
"may also be given additional information that was used by the model to generate the output.\n",
"\n",
"Your task is to determine a numerical score called relevance based on the input and output.\n",
"A definition of relevance and a grading rubric are provided below.\n",
"You must use the grading rubric to determine your score. You must also justify your score.\n",
"\n",
"Examples could be included below for reference. Make sure to use them as references and to\n",
"understand them before completing the task.\n",
"\n",
"Input:\n",
"{input}\n",
"\n",
"Output:\n",
"{output}\n",
"\n",
"{grading_context_columns}\n",
"\n",
"Metric definition:\n",
"Relevance encompasses the appropriateness, significance, and applicability of the output with respect to both the input and context. Scores should reflect the extent to which the output directly addresses the question provided in the input, given the provided context.\n",
"\n",
"Grading rubric:\n",
"Relevance: Below are the details for different scores:- Score 1: The output doesn't mention anything about the question or is completely irrelevant to the provided context.\n",
"- Score 2: The output provides some relevance to the question and is somehow related to the provided context.\n",
"- Score 3: The output mostly answers the question and is largely consistent with the provided context.\n",
"- Score 4: The output answers the question and is consistent with the provided context.\n",
"- Score 5: The output answers the question comprehensively using the provided context.\n",
"\n",
"Examples:\n",
"\n",
"Example Input:\n",
"How is MLflow related to Databricks?\n",
"\n",
"Example Output:\n",
"Databricks is a data engineering and analytics platform designed to help organizations process and analyze large amounts of data. Databricks is a company specializing in big data and machine learning solutions.\n",
"\n",
"Additional information used by the model:\n",
"key: context\n",
"value:\n",
"MLflow is an open-source platform for managing the end-to-end machine learning (ML) lifecycle. It was developed by Databricks, a company that specializes in big data and machine learning solutions. MLflow is designed to address the challenges that data scientists and machine learning engineers face when developing, training, and deploying machine learning models.\n",
"\n",
"Example score: 2\n",
"Example justification: The output provides relevant information about Databricks, mentioning it as a company specializing in big data and machine learning solutions. However, it doesn't directly address how MLflow is related to Databricks, which is the specific question asked in the input. Therefore, the output is only somewhat related to the provided context.\n",
" \n",
"\n",
"Example Input:\n",
"How is MLflow related to Databricks?\n",
"\n",
"Example Output:\n",
"MLflow is a product created by Databricks to enhance the efficiency of machine learning processes.\n",
"\n",
"Additional information used by the model:\n",
"key: context\n",
"value:\n",
"MLflow is an open-source platform for managing the end-to-end machine learning (ML) lifecycle. It was developed by Databricks, a company that specializes in big data and machine learning solutions. MLflow is designed to address the challenges that data scientists and machine learning engineers face when developing, training, and deploying machine learning models.\n",
"\n",
"Example score: 4\n",
"Example justification: The output provides a relevant and accurate statement about the relationship between MLflow and Databricks. While it doesn't provide extensive detail, it still offers a substantial and meaningful response. To achieve a score of 5, the response could be further improved by providing additional context or details about how MLflow specifically functions within the Databricks ecosystem.\n",
" \n",
"\n",
"You must return the following fields in your response in two lines, one below the other:\n",
"score: Your numerical score for the model's relevance based on the rubric\n",
"justification: Your reasoning about the model's relevance score\n",
"\n",
"Do not add additional new lines. Do not add any other fields.\n",
" )\n"
]
}
],
"source": [
"from mlflow.metrics.genai import EvaluationExample, relevance\n",
"\n",
"relevance_metric = relevance(model=\"openai:/gpt-4\")\n",
"print(relevance_metric)"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
"eval_df_final=eval_df.copy(deep=True)"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [],
"source": [
"aa=model(eval_df)"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'query': 'What is MLflow?',\n",
" 'result': 'MLflow is an open-source platform designed to help machine learning practitioners and teams manage the complexities of the machine learning process. It focuses on the full lifecycle of machine learning projects, making sure that each phase is manageable, traceable, and reproducible.',\n",
" 'source_documents': [Document(page_content='MLflow: A Tool for Managing the Machine Learning Lifecycle \\nMLflow is an open-source platform, purpose-built to assist machine learning practitioners and teams in\\nhandling the complexities of the machine learning process. MLflow focuses on the full lifecycle for\\nmachine learning projects, ensuring that each phase is manageable, traceable, and reproducible.\\nIn each of the sections below, you will find overviews, guides, and step-by-step tutorials to walk you through\\nthe features of MLflow and how they can be leveraged to solve real-world MLOps problems.\\n\\nGetting Started with MLflow \\nIf this is your first time exploring MLflow, the tutorials and guides here are a great place to start. The emphasis in each of these is\\ngetting you up to speed as quickly as possible with the basic functionality, terms, APIs, and general best practices of using MLflow in order to\\nenhance your learning in area-specific guides and tutorials.\\n\\nGetting Started Guides and Quickstarts', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='MLflow: A Tool for Managing the Machine Learning Lifecycle \\nMLflow is an open-source platform, purpose-built to assist machine learning practitioners and teams in\\nhandling the complexities of the machine learning process. MLflow focuses on the full lifecycle for\\nmachine learning projects, ensuring that each phase is manageable, traceable, and reproducible.\\nIn each of the sections below, you will find overviews, guides, and step-by-step tutorials to walk you through\\nthe features of MLflow and how they can be leveraged to solve real-world MLOps problems.\\n\\nGetting Started with MLflow \\nIf this is your first time exploring MLflow, the tutorials and guides here are a great place to start. The emphasis in each of these is\\ngetting you up to speed as quickly as possible with the basic functionality, terms, APIs, and general best practices of using MLflow in order to\\nenhance your learning in area-specific guides and tutorials.\\n\\nGetting Started Guides and Quickstarts', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='MLflow: A Tool for Managing the Machine Learning Lifecycle \\nMLflow is an open-source platform, purpose-built to assist machine learning practitioners and teams in\\nhandling the complexities of the machine learning process. MLflow focuses on the full lifecycle for\\nmachine learning projects, ensuring that each phase is manageable, traceable, and reproducible.\\nIn each of the sections below, you will find overviews, guides, and step-by-step tutorials to walk you through\\nthe features of MLflow and how they can be leveraged to solve real-world MLOps problems.\\n\\nGetting Started with MLflow \\nIf this is your first time exploring MLflow, the tutorials and guides here are a great place to start. The emphasis in each of these is\\ngetting you up to speed as quickly as possible with the basic functionality, terms, APIs, and general best practices of using MLflow in order to\\nenhance your learning in area-specific guides and tutorials.\\n\\nGetting Started Guides and Quickstarts', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation\\n\\n2.12.1\\n\\n\\n MLflow\\n\\nWhat is MLflow?\\nGetting Started with MLflow\\nNew Features\\nLLMs\\nModel Evaluation\\nDeep Learning\\nTraditional ML\\nDeployment\\nMLflow Tracking\\nSystem Metrics\\nMLflow Projects\\nMLflow Models\\nMLflow Model Registry\\nMLflow Recipes\\nMLflow Plugins\\nMLflow Authentication\\nCommand-Line Interface\\nSearch Runs\\nSearch Experiments\\nPython API\\nR API\\nJava API\\nREST API\\nOfficial MLflow Docker Image\\nCommunity Model Flavors\\nTutorials and Examples\\n\\n\\nContribute\\n\\n\\nDocumentation \\nMLflow: A Tool for Managing the Machine Learning Lifecycle', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'})]},\n",
" {'query': 'How to run mlflow.evaluate()?',\n",
" 'result': 'To run `mlflow.evaluate()`, you need to follow these steps:\\n\\n1. Ensure you have MLflow installed in your environment.\\n2. Import the necessary libraries, including `mlflow`.\\n3. Load your model and data.\\n4. Use the `mlflow.evaluate()` function, passing in your model, data, and any other necessary parameters.\\n5. Review the evaluation results and metrics provided by the function.\\n\\nIf you need more specific details or code examples, please refer to the MLflow documentation or tutorials for a step-by-step guide on running `mlflow.evaluate()`.',\n",
" 'source_documents': [Document(page_content='Model Evaluation \\nDive into MLflow’s robust framework for evaluating the performance of your ML models.\\nWith support for traditional ML evaluation (classification and regression tasks), as well as support for evaluating large language models (LLMs),\\nthis suite of APIs offers a simple but powerful automated approach to evaluating the quality of the model development work that you’re doing.\\nIn particular, for LLM evaluation, the mlflow.evaluate() API allows you to validate not only models, but providers and prompts.\\nBy leveraging your own datasets and using the provided default evaluation criteria for tasks such as text summarization and question answering, you can\\nget reliable metrics that allow you to focus on improving the quality of your solution, rather than spending time writing scoring code.\\nVisual insights are also available through the MLflow UI, showcasing logged outputs, auto-generated plots, and model comparison artifacts.', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='Model Evaluation \\nDive into MLflow’s robust framework for evaluating the performance of your ML models.\\nWith support for traditional ML evaluation (classification and regression tasks), as well as support for evaluating large language models (LLMs),\\nthis suite of APIs offers a simple but powerful automated approach to evaluating the quality of the model development work that you’re doing.\\nIn particular, for LLM evaluation, the mlflow.evaluate() API allows you to validate not only models, but providers and prompts.\\nBy leveraging your own datasets and using the provided default evaluation criteria for tasks such as text summarization and question answering, you can\\nget reliable metrics that allow you to focus on improving the quality of your solution, rather than spending time writing scoring code.\\nVisual insights are also available through the MLflow UI, showcasing logged outputs, auto-generated plots, and model comparison artifacts.', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='Model Evaluation \\nDive into MLflow’s robust framework for evaluating the performance of your ML models.\\nWith support for traditional ML evaluation (classification and regression tasks), as well as support for evaluating large language models (LLMs),\\nthis suite of APIs offers a simple but powerful automated approach to evaluating the quality of the model development work that you’re doing.\\nIn particular, for LLM evaluation, the mlflow.evaluate() API allows you to validate not only models, but providers and prompts.\\nBy leveraging your own datasets and using the provided default evaluation criteria for tasks such as text summarization and question answering, you can\\nget reliable metrics that allow you to focus on improving the quality of your solution, rather than spending time writing scoring code.\\nVisual insights are also available through the MLflow UI, showcasing logged outputs, auto-generated plots, and model comparison artifacts.', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='Learn how to evaluate LLMs and LLM-powered solutions with MLflow Evaluate.\\n \\n\\n Using Custom PyFunc with LLMs\\n \\n\\n Explore the nuances of packaging and deploying advanced LLMs in MLflow using custom PyFuncs. This guide delves deep\\n into managing intricate model behaviors, ensuring seamless and efficient LLM deployments.\\n \\n\\n Evaluation for RAG\\n \\n\\n Learn how to evaluate Retrieval Augmented Generation applications by leveraging LLMs to generate a evaluation dataset and evaluate it using the built-in metrics in the MLflow Evaluate API.\\n \\n\\n LLM Tracking with MLflow', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'})]},\n",
" {'query': 'How to log_table()?',\n",
" 'result': \"I don't have information on a specific function called `log_table()` in the context provided. It's possible that it might be a custom function or a feature not explicitly mentioned in the provided context. If you can provide more details or context, I may be able to assist you further.\",\n",
" 'source_documents': [Document(page_content='MLflow Tracking Quickstart\\n \\n\\n A great place to start to learn the fundamentals of MLflow Tracking! Learn in 5 minutes how to log, register, and load a model for inference.\\n \\n\\n Intro to MLflow Tutorial\\n \\n\\n Learn how to get started with the basics of MLflow in a step-by-step instructional tutorial that shows the critical\\n path to logging your first model\\n \\n\\n Autologging Quickstart\\n \\n\\n Short on time? This is a no-frills quickstart that shows how to leverage autologging during training and how to\\n load a model for inference\\n \\n\\n Deployment Quickstart', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='MLflow Tracking Quickstart\\n \\n\\n A great place to start to learn the fundamentals of MLflow Tracking! Learn in 5 minutes how to log, register, and load a model for inference.\\n \\n\\n Intro to MLflow Tutorial\\n \\n\\n Learn how to get started with the basics of MLflow in a step-by-step instructional tutorial that shows the critical\\n path to logging your first model\\n \\n\\n Autologging Quickstart\\n \\n\\n Short on time? This is a no-frills quickstart that shows how to leverage autologging during training and how to\\n load a model for inference\\n \\n\\n Deployment Quickstart', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='MLflow Tracking Quickstart\\n \\n\\n A great place to start to learn the fundamentals of MLflow Tracking! Learn in 5 minutes how to log, register, and load a model for inference.\\n \\n\\n Intro to MLflow Tutorial\\n \\n\\n Learn how to get started with the basics of MLflow in a step-by-step instructional tutorial that shows the critical\\n path to logging your first model\\n \\n\\n Autologging Quickstart\\n \\n\\n Short on time? This is a no-frills quickstart that shows how to leverage autologging during training and how to\\n load a model for inference\\n \\n\\n Deployment Quickstart', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content=\"Dive into the intricacies of MLflow's LLM Tracking system. From capturing prompts to monitoring generated outputs,\\n discover how MLflow provides a holistic solution for managing LLM interactions.\", metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'})]},\n",
" {'query': 'How to load_table()?',\n",
" 'result': \"I'm not sure what `load_table()` refers to in this context. If you can provide more information or context, I might be able to help you better.\",\n",
" 'source_documents': [Document(page_content='MLflow Tracking Quickstart\\n \\n\\n A great place to start to learn the fundamentals of MLflow Tracking! Learn in 5 minutes how to log, register, and load a model for inference.\\n \\n\\n Intro to MLflow Tutorial\\n \\n\\n Learn how to get started with the basics of MLflow in a step-by-step instructional tutorial that shows the critical\\n path to logging your first model\\n \\n\\n Autologging Quickstart\\n \\n\\n Short on time? This is a no-frills quickstart that shows how to leverage autologging during training and how to\\n load a model for inference\\n \\n\\n Deployment Quickstart', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='MLflow Tracking Quickstart\\n \\n\\n A great place to start to learn the fundamentals of MLflow Tracking! Learn in 5 minutes how to log, register, and load a model for inference.\\n \\n\\n Intro to MLflow Tutorial\\n \\n\\n Learn how to get started with the basics of MLflow in a step-by-step instructional tutorial that shows the critical\\n path to logging your first model\\n \\n\\n Autologging Quickstart\\n \\n\\n Short on time? This is a no-frills quickstart that shows how to leverage autologging during training and how to\\n load a model for inference\\n \\n\\n Deployment Quickstart', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='MLflow Tracking Quickstart\\n \\n\\n A great place to start to learn the fundamentals of MLflow Tracking! Learn in 5 minutes how to log, register, and load a model for inference.\\n \\n\\n Intro to MLflow Tutorial\\n \\n\\n Learn how to get started with the basics of MLflow in a step-by-step instructional tutorial that shows the critical\\n path to logging your first model\\n \\n\\n Autologging Quickstart\\n \\n\\n Short on time? This is a no-frills quickstart that shows how to leverage autologging during training and how to\\n load a model for inference\\n \\n\\n Deployment Quickstart', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='This guide showcases the seamless end-to-end process of training a linear regression model, packaging it in a reproducible format,\\n and deploying to a Kubernetes cluster using MLflow. Explore how MLflow simplifies model deployment to production environments.\\n \\n\\n\\nNext \\n\\n\\n © MLflow Project, a Series of LF Projects, LLC. All rights reserved.', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'})]}]"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"aa"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'query': 'What is MLflow?',\n",
" 'result': 'MLflow is an open-source platform designed to help machine learning practitioners and teams manage the complexities of the machine learning process. It focuses on the full lifecycle of machine learning projects, ensuring that each phase is manageable, traceable, and reproducible. It offers features such as tracking, projects, models, model registry, and more to assist in solving real-world MLOps problems.',\n",
" 'source_documents': [Document(page_content='MLflow: A Tool for Managing the Machine Learning Lifecycle \\nMLflow is an open-source platform, purpose-built to assist machine learning practitioners and teams in\\nhandling the complexities of the machine learning process. MLflow focuses on the full lifecycle for\\nmachine learning projects, ensuring that each phase is manageable, traceable, and reproducible.\\nIn each of the sections below, you will find overviews, guides, and step-by-step tutorials to walk you through\\nthe features of MLflow and how they can be leveraged to solve real-world MLOps problems.\\n\\nGetting Started with MLflow \\nIf this is your first time exploring MLflow, the tutorials and guides here are a great place to start. The emphasis in each of these is\\ngetting you up to speed as quickly as possible with the basic functionality, terms, APIs, and general best practices of using MLflow in order to\\nenhance your learning in area-specific guides and tutorials.\\n\\nGetting Started Guides and Quickstarts', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='MLflow: A Tool for Managing the Machine Learning Lifecycle \\nMLflow is an open-source platform, purpose-built to assist machine learning practitioners and teams in\\nhandling the complexities of the machine learning process. MLflow focuses on the full lifecycle for\\nmachine learning projects, ensuring that each phase is manageable, traceable, and reproducible.\\nIn each of the sections below, you will find overviews, guides, and step-by-step tutorials to walk you through\\nthe features of MLflow and how they can be leveraged to solve real-world MLOps problems.\\n\\nGetting Started with MLflow \\nIf this is your first time exploring MLflow, the tutorials and guides here are a great place to start. The emphasis in each of these is\\ngetting you up to speed as quickly as possible with the basic functionality, terms, APIs, and general best practices of using MLflow in order to\\nenhance your learning in area-specific guides and tutorials.\\n\\nGetting Started Guides and Quickstarts', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='MLflow: A Tool for Managing the Machine Learning Lifecycle \\nMLflow is an open-source platform, purpose-built to assist machine learning practitioners and teams in\\nhandling the complexities of the machine learning process. MLflow focuses on the full lifecycle for\\nmachine learning projects, ensuring that each phase is manageable, traceable, and reproducible.\\nIn each of the sections below, you will find overviews, guides, and step-by-step tutorials to walk you through\\nthe features of MLflow and how they can be leveraged to solve real-world MLOps problems.\\n\\nGetting Started with MLflow \\nIf this is your first time exploring MLflow, the tutorials and guides here are a great place to start. The emphasis in each of these is\\ngetting you up to speed as quickly as possible with the basic functionality, terms, APIs, and general best practices of using MLflow in order to\\nenhance your learning in area-specific guides and tutorials.\\n\\nGetting Started Guides and Quickstarts', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation\\n\\n2.12.1\\n\\n\\n MLflow\\n\\nWhat is MLflow?\\nGetting Started with MLflow\\nNew Features\\nLLMs\\nModel Evaluation\\nDeep Learning\\nTraditional ML\\nDeployment\\nMLflow Tracking\\nSystem Metrics\\nMLflow Projects\\nMLflow Models\\nMLflow Model Registry\\nMLflow Recipes\\nMLflow Plugins\\nMLflow Authentication\\nCommand-Line Interface\\nSearch Runs\\nSearch Experiments\\nPython API\\nR API\\nJava API\\nREST API\\nOfficial MLflow Docker Image\\nCommunity Model Flavors\\nTutorials and Examples\\n\\n\\nContribute\\n\\n\\nDocumentation \\nMLflow: A Tool for Managing the Machine Learning Lifecycle', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'})]},\n",
" {'query': 'How to run mlflow.evaluate()?',\n",
" 'result': 'To run `mlflow.evaluate()`, you need to follow these steps:\\n\\n1. Import the necessary libraries:\\n```python\\nimport mlflow\\n```\\n\\n2. Load your model and data:\\n```python\\nmodel = load_model() # Load your model\\ndata = load_data() # Load your evaluation data\\n```\\n\\n3. Use `mlflow.evaluate()` to evaluate your model:\\n```python\\nmlflow.evaluate(model, data)\\n```\\n\\n4. Review the evaluation metrics and visual insights in the MLflow UI.\\n\\nIf you need more specific details or have a particular use case in mind, please provide additional context for a more tailored explanation.',\n",
" 'source_documents': [Document(page_content='Model Evaluation \\nDive into MLflow’s robust framework for evaluating the performance of your ML models.\\nWith support for traditional ML evaluation (classification and regression tasks), as well as support for evaluating large language models (LLMs),\\nthis suite of APIs offers a simple but powerful automated approach to evaluating the quality of the model development work that you’re doing.\\nIn particular, for LLM evaluation, the mlflow.evaluate() API allows you to validate not only models, but providers and prompts.\\nBy leveraging your own datasets and using the provided default evaluation criteria for tasks such as text summarization and question answering, you can\\nget reliable metrics that allow you to focus on improving the quality of your solution, rather than spending time writing scoring code.\\nVisual insights are also available through the MLflow UI, showcasing logged outputs, auto-generated plots, and model comparison artifacts.', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='Model Evaluation \\nDive into MLflow’s robust framework for evaluating the performance of your ML models.\\nWith support for traditional ML evaluation (classification and regression tasks), as well as support for evaluating large language models (LLMs),\\nthis suite of APIs offers a simple but powerful automated approach to evaluating the quality of the model development work that you’re doing.\\nIn particular, for LLM evaluation, the mlflow.evaluate() API allows you to validate not only models, but providers and prompts.\\nBy leveraging your own datasets and using the provided default evaluation criteria for tasks such as text summarization and question answering, you can\\nget reliable metrics that allow you to focus on improving the quality of your solution, rather than spending time writing scoring code.\\nVisual insights are also available through the MLflow UI, showcasing logged outputs, auto-generated plots, and model comparison artifacts.', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='Model Evaluation \\nDive into MLflow’s robust framework for evaluating the performance of your ML models.\\nWith support for traditional ML evaluation (classification and regression tasks), as well as support for evaluating large language models (LLMs),\\nthis suite of APIs offers a simple but powerful automated approach to evaluating the quality of the model development work that you’re doing.\\nIn particular, for LLM evaluation, the mlflow.evaluate() API allows you to validate not only models, but providers and prompts.\\nBy leveraging your own datasets and using the provided default evaluation criteria for tasks such as text summarization and question answering, you can\\nget reliable metrics that allow you to focus on improving the quality of your solution, rather than spending time writing scoring code.\\nVisual insights are also available through the MLflow UI, showcasing logged outputs, auto-generated plots, and model comparison artifacts.', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='Learn how to evaluate LLMs and LLM-powered solutions with MLflow Evaluate.\\n \\n\\n Using Custom PyFunc with LLMs\\n \\n\\n Explore the nuances of packaging and deploying advanced LLMs in MLflow using custom PyFuncs. This guide delves deep\\n into managing intricate model behaviors, ensuring seamless and efficient LLM deployments.\\n \\n\\n Evaluation for RAG\\n \\n\\n Learn how to evaluate Retrieval Augmented Generation applications by leveraging LLMs to generate a evaluation dataset and evaluate it using the built-in metrics in the MLflow Evaluate API.\\n \\n\\n LLM Tracking with MLflow', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'})]},\n",
" {'query': 'How to log_table()?',\n",
" 'result': \"I don't have information on a specific function called `log_table()` in the context provided. It's possible that it might be a custom function or a feature not covered in the provided context. If you can provide more details or context, I may be able to assist you further.\",\n",
" 'source_documents': [Document(page_content='MLflow Tracking Quickstart\\n \\n\\n A great place to start to learn the fundamentals of MLflow Tracking! Learn in 5 minutes how to log, register, and load a model for inference.\\n \\n\\n Intro to MLflow Tutorial\\n \\n\\n Learn how to get started with the basics of MLflow in a step-by-step instructional tutorial that shows the critical\\n path to logging your first model\\n \\n\\n Autologging Quickstart\\n \\n\\n Short on time? This is a no-frills quickstart that shows how to leverage autologging during training and how to\\n load a model for inference\\n \\n\\n Deployment Quickstart', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='MLflow Tracking Quickstart\\n \\n\\n A great place to start to learn the fundamentals of MLflow Tracking! Learn in 5 minutes how to log, register, and load a model for inference.\\n \\n\\n Intro to MLflow Tutorial\\n \\n\\n Learn how to get started with the basics of MLflow in a step-by-step instructional tutorial that shows the critical\\n path to logging your first model\\n \\n\\n Autologging Quickstart\\n \\n\\n Short on time? This is a no-frills quickstart that shows how to leverage autologging during training and how to\\n load a model for inference\\n \\n\\n Deployment Quickstart', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='MLflow Tracking Quickstart\\n \\n\\n A great place to start to learn the fundamentals of MLflow Tracking! Learn in 5 minutes how to log, register, and load a model for inference.\\n \\n\\n Intro to MLflow Tutorial\\n \\n\\n Learn how to get started with the basics of MLflow in a step-by-step instructional tutorial that shows the critical\\n path to logging your first model\\n \\n\\n Autologging Quickstart\\n \\n\\n Short on time? This is a no-frills quickstart that shows how to leverage autologging during training and how to\\n load a model for inference\\n \\n\\n Deployment Quickstart', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content=\"Dive into the intricacies of MLflow's LLM Tracking system. From capturing prompts to monitoring generated outputs,\\n discover how MLflow provides a holistic solution for managing LLM interactions.\", metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'})]},\n",
" {'query': 'How to load_table()?',\n",
" 'result': \"I don't have information on a function called `load_table()` in the context provided. It seems to be outside the scope of the MLflow Tracking, Autologging, and Deployment Quickstart guides. If you can provide more context or clarify where `load_table()` is from, I may be able to assist you further.\",\n",
" 'source_documents': [Document(page_content='MLflow Tracking Quickstart\\n \\n\\n A great place to start to learn the fundamentals of MLflow Tracking! Learn in 5 minutes how to log, register, and load a model for inference.\\n \\n\\n Intro to MLflow Tutorial\\n \\n\\n Learn how to get started with the basics of MLflow in a step-by-step instructional tutorial that shows the critical\\n path to logging your first model\\n \\n\\n Autologging Quickstart\\n \\n\\n Short on time? This is a no-frills quickstart that shows how to leverage autologging during training and how to\\n load a model for inference\\n \\n\\n Deployment Quickstart', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='MLflow Tracking Quickstart\\n \\n\\n A great place to start to learn the fundamentals of MLflow Tracking! Learn in 5 minutes how to log, register, and load a model for inference.\\n \\n\\n Intro to MLflow Tutorial\\n \\n\\n Learn how to get started with the basics of MLflow in a step-by-step instructional tutorial that shows the critical\\n path to logging your first model\\n \\n\\n Autologging Quickstart\\n \\n\\n Short on time? This is a no-frills quickstart that shows how to leverage autologging during training and how to\\n load a model for inference\\n \\n\\n Deployment Quickstart', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='MLflow Tracking Quickstart\\n \\n\\n A great place to start to learn the fundamentals of MLflow Tracking! Learn in 5 minutes how to log, register, and load a model for inference.\\n \\n\\n Intro to MLflow Tutorial\\n \\n\\n Learn how to get started with the basics of MLflow in a step-by-step instructional tutorial that shows the critical\\n path to logging your first model\\n \\n\\n Autologging Quickstart\\n \\n\\n Short on time? This is a no-frills quickstart that shows how to leverage autologging during training and how to\\n load a model for inference\\n \\n\\n Deployment Quickstart', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'}),\n",
" Document(page_content='This guide showcases the seamless end-to-end process of training a linear regression model, packaging it in a reproducible format,\\n and deploying to a Kubernetes cluster using MLflow. Explore how MLflow simplifies model deployment to production environments.\\n \\n\\n\\nNext \\n\\n\\n © MLflow Project, a Series of LF Projects, LLC. All rights reserved.', metadata={'language': 'en', 'source': 'https://mlflow.org/docs/latest/index.html', 'title': 'MLflow: A Tool for Managing the Machine Learning Lifecycle — MLflow 2.12.1 documentation'})]}]"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"eval_df_final[\"results\"]=model(eval_df)"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"application/vnd.databricks.v1+cell": {
"cellMetadata": {
"byteLimit": 2048000,
"rowLimit": 10000
},
"inputWidgets": {},
"nuid": "ea40ce52-6ac7-4c20-9669-d24f80a6cebe",
"showTitle": false,
"title": ""
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/u/marshad/.conda/envs/agllm-env1/lib/python3.9/site-packages/mlflow/data/digest_utils.py:26: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n",
" string_columns = trimmed_df.columns[(df.applymap(type) == str).all(0)]\n",
"/u/marshad/.conda/envs/agllm-env1/lib/python3.9/site-packages/mlflow/models/evaluation/base.py:414: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n",
" data = data.applymap(_hash_array_like_element_as_bytes)\n",
"2024/04/21 17:23:37 INFO mlflow.models.evaluation.base: Evaluating the model with the default evaluator.\n",
"2024/04/21 17:23:37 INFO mlflow.models.evaluation.default_evaluator: Computing model predictions.\n",
"2024/04/21 17:23:44 INFO mlflow.models.evaluation.default_evaluator: Testing metrics on first row...\n",
"2024/04/21 17:23:44 WARNING mlflow.metrics.metric_definitions: Failed to load 'toxicity' metric (error: ModuleNotFoundError(\"No module named 'evaluate'\")), skipping metric logging.\n",
"2024/04/21 17:23:44 WARNING mlflow.metrics.metric_definitions: Failed to load flesch kincaid metric, skipping metric logging.\n",
"2024/04/21 17:23:44 WARNING mlflow.metrics.metric_definitions: Failed to load automated readability index metric, skipping metric logging.\n",
"100%|██████████| 1/1 [00:03<00:00, 3.53s/it]\n",
"100%|██████████| 1/1 [00:03<00:00, 3.72s/it]\n",
"2024/04/21 17:23:51 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: token_count\n",
"2024/04/21 17:23:51 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: toxicity\n",
"2024/04/21 17:23:51 WARNING mlflow.metrics.metric_definitions: Failed to load 'toxicity' metric (error: ModuleNotFoundError(\"No module named 'evaluate'\")), skipping metric logging.\n",
"2024/04/21 17:23:51 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: flesch_kincaid_grade_level\n",
"2024/04/21 17:23:51 WARNING mlflow.metrics.metric_definitions: Failed to load flesch kincaid metric, skipping metric logging.\n",
"2024/04/21 17:23:51 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: ari_grade_level\n",
"2024/04/21 17:23:51 WARNING mlflow.metrics.metric_definitions: Failed to load automated readability index metric, skipping metric logging.\n",
"2024/04/21 17:23:51 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: exact_match\n",
"2024/04/21 17:23:51 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: faithfulness\n",
"100%|██████████| 4/4 [00:03<00:00, 1.02it/s]\n",
"2024/04/21 17:23:55 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: relevance\n",
"100%|██████████| 4/4 [00:04<00:00, 1.20s/it]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'latency/mean': 1.7213420271873474, 'latency/variance': 0.04483020464773446, 'latency/p90': 1.942362928390503, 'faithfulness/v1/mean': 4.75, 'faithfulness/v1/variance': 0.1875, 'faithfulness/v1/p90': 5.0, 'relevance/v1/mean': 3.75, 'relevance/v1/variance': 1.6875, 'relevance/v1/p90': 5.0}\n"
]
}
],
"source": [
"results = mlflow.evaluate(\n",
" model,\n",
" eval_df,\n",
" model_type=\"question-answering\",\n",
" evaluators=\"default\",\n",
" predictions=\"result\",\n",
" extra_metrics=[faithfulness_metric, relevance_metric, mlflow.metrics.latency()],\n",
" evaluator_config={\n",
" \"col_mapping\": {\n",
" \"inputs\": \"questions\",\n",
" \"context\": \"source_documents\",\n",
" }\n",
" },\n",
")\n",
"print(results.metrics)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"application/vnd.databricks.v1+cell": {
"cellMetadata": {},
"inputWidgets": {},
"nuid": "989a0861-5153-44e6-a19d-efcae7fe6cb5",
"showTitle": false,
"title": ""
}
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "747f65b309b94257b396eebffe814fa6",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Downloading artifacts: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>questions</th>\n",
" <th>outputs</th>\n",
" <th>source_documents</th>\n",
" <th>latency</th>\n",
" <th>token_count</th>\n",
" <th>toxicity/v1/score</th>\n",
" <th>flesch_kincaid_grade_level/v1/score</th>\n",
" <th>ari_grade_level/v1/score</th>\n",
" <th>faithfulness/v1/score</th>\n",
" <th>faithfulness/v1/justification</th>\n",
" <th>relevance/v1/score</th>\n",
" <th>relevance/v1/justification</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>What is MLflow?</td>\n",
" <td>MLflow is an open-source platform, purpose-bu...</td>\n",
" <td>[{'lc_attributes': {}, 'lc_namespace': ['langc...</td>\n",
" <td>1.989822</td>\n",
" <td>53</td>\n",
" <td>0.000137</td>\n",
" <td>12.5</td>\n",
" <td>18.4</td>\n",
" <td>5</td>\n",
" <td>The output provided by the model is a direct e...</td>\n",
" <td>5</td>\n",
" <td>The output provides a comprehensive answer to ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>How to run mlflow.evaluate()?</td>\n",
" <td>The mlflow.evaluate() API allows you to valid...</td>\n",
" <td>[{'lc_attributes': {}, 'lc_namespace': ['langc...</td>\n",
" <td>1.945368</td>\n",
" <td>55</td>\n",
" <td>0.000200</td>\n",
" <td>9.1</td>\n",
" <td>12.6</td>\n",
" <td>5</td>\n",
" <td>The output provided by the model is completely...</td>\n",
" <td>4</td>\n",
" <td>The output provides a relevant and accurate ex...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>How to log_table()?</td>\n",
" <td>You can log a table with MLflow using the log...</td>\n",
" <td>[{'lc_attributes': {}, 'lc_namespace': ['langc...</td>\n",
" <td>1.521511</td>\n",
" <td>32</td>\n",
" <td>0.000289</td>\n",
" <td>5.0</td>\n",
" <td>6.8</td>\n",
" <td>1</td>\n",
" <td>The output claims that you can log a table wit...</td>\n",
" <td>5</td>\n",
" <td>The output provides a comprehensive answer to ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>How to load_table()?</td>\n",
" <td>You can't load_table() with MLflow. MLflow is...</td>\n",
" <td>[{'lc_attributes': {}, 'lc_namespace': ['langc...</td>\n",
" <td>1.105279</td>\n",
" <td>27</td>\n",
" <td>0.000279</td>\n",
" <td>5.8</td>\n",
" <td>8.8</td>\n",
" <td>5</td>\n",
" <td>The output claim that \"You can't load_table() ...</td>\n",
" <td>4</td>\n",
" <td>The output provides a relevant and accurate re...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" questions \\\n",
"0 What is MLflow? \n",
"1 How to run mlflow.evaluate()? \n",
"2 How to log_table()? \n",
"3 How to load_table()? \n",
"\n",
" outputs \\\n",
"0 MLflow is an open-source platform, purpose-bu... \n",
"1 The mlflow.evaluate() API allows you to valid... \n",
"2 You can log a table with MLflow using the log... \n",
"3 You can't load_table() with MLflow. MLflow is... \n",
"\n",
" source_documents latency token_count \\\n",
"0 [{'lc_attributes': {}, 'lc_namespace': ['langc... 1.989822 53 \n",
"1 [{'lc_attributes': {}, 'lc_namespace': ['langc... 1.945368 55 \n",
"2 [{'lc_attributes': {}, 'lc_namespace': ['langc... 1.521511 32 \n",
"3 [{'lc_attributes': {}, 'lc_namespace': ['langc... 1.105279 27 \n",
"\n",
" toxicity/v1/score flesch_kincaid_grade_level/v1/score \\\n",
"0 0.000137 12.5 \n",
"1 0.000200 9.1 \n",
"2 0.000289 5.0 \n",
"3 0.000279 5.8 \n",
"\n",
" ari_grade_level/v1/score faithfulness/v1/score \\\n",
"0 18.4 5 \n",
"1 12.6 5 \n",
"2 6.8 1 \n",
"3 8.8 5 \n",
"\n",
" faithfulness/v1/justification relevance/v1/score \\\n",
"0 The output provided by the model is a direct e... 5 \n",
"1 The output provided by the model is completely... 4 \n",
"2 The output claims that you can log a table wit... 5 \n",
"3 The output claim that \"You can't load_table() ... 4 \n",
"\n",
" relevance/v1/justification \n",
"0 The output provides a comprehensive answer to ... \n",
"1 The output provides a relevant and accurate ex... \n",
"2 The output provides a comprehensive answer to ... \n",
"3 The output provides a relevant and accurate re... "
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"results.tables[\"eval_results_table\"]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"application/vnd.databricks.v1+notebook": {
"dashboards": [],
"language": "python",
"notebookMetadata": {
"pythonIndentUnit": 2
},
"notebookName": "LLM Evaluation Examples -- RAG",
"widgets": {}
},
"kernelspec": {
"display_name": "mlflow-dev-env",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.19"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
|