{ "cells": [ { "cell_type": "markdown", "id": "faa97138-7ee4-4aef-942f-961b321f05d7", "metadata": {}, "source": [ "# Evaluating a NeMo checkpoint on an arbitrary task" ] }, { "cell_type": "markdown", "id": "5a4ea639-7a9d-4f06-90cf-be4e9476dacc", "metadata": {}, "source": [ "This notebook demonstrates how to extend the evaluation capabilities within the NeMo Framework container.\n", "It guides you through the advanced configuration of an evaluation job.\n", "\n", "For an introduction to evaluation with NVIDIA Evals Factory and the NeMo Framework, please refer to the tutorial [\"Evaluating a NeMo checkpoint with lm-eval\"](mmlu.ipynb) first.\n", "\n", "In this tutorial, we will evaluate an LLM on the [WikiText-2](https://arxiv.org/abs/1609.07843) benchmark, which is available in the [NVIDIA Evals Factory lm-eval](https://pypi.org/project/nvidia-lm-eval/) package.\n", "The evaluation utilizes the log-probabilities of the context tokens to assess how likely the input text is, according to the model.\n", "\n", "> Note: It is recommended to run this notebook within a [NeMo Framework container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo), as it includes all necessary dependencies." ] }, { "cell_type": "code", "execution_count": null, "id": "23282cea-9b37-465f-a3f9-7e8caf25ce34", "metadata": {}, "outputs": [], "source": [ "import json\n", "import signal\n", "import subprocess\n", "\n", "from nemo.collections.llm import api\n", "from nemo.collections.llm.evaluation.api import EvaluationConfig, EvaluationTarget\n", "from nemo.collections.llm.evaluation.base import list_available_evaluations\n", "from nemo.utils import logging\n", "\n", "logging.setLevel(logging.INFO)" ] }, { "cell_type": "markdown", "id": "60de9e72-96a0-477e-93e3-7e1eb75b4c93", "metadata": {}, "source": [ "## 1. Deploying the model" ] }, { "cell_type": "markdown", "id": "6eecf707-d1ec-4ae4-b8d7-1d248647b520", "metadata": {}, "source": [ "We will start from deploying the model.\n", "\n", "First, you need to prepare a NeMo 2 checkpoint of the model you would like to evaluate.\n", "For the purpose of this tutorial, we will use the Llama 3.2 1B Instruct checkpoint, which you can download from the [NGC Catalog](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/llama-3_2-1b-instruct).\n", "Ensure that you mount the directory containing the checkpoint when starting the container.\n", "In this tutorial, we assume that the checkpoint is accessible under the path `\"/checkpoints/llama-3_2-1b-instruct_v2.0\"`.\n", "\n", "> Note: You can learn more about deployment and available server endpoints from the [\"Evaluating a NeMo checkpoint with lm-eval\"](mmlu.ipynb) tutorial. " ] }, { "cell_type": "code", "execution_count": null, "id": "cf964980-69ba-447d-a6d8-1412726c768a", "metadata": {}, "outputs": [], "source": [ "# modify this variable to point to your checkpoint\n", "# this notebook uses https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/llama-3_2-1b-instruct\n", "CHECKPOINT_PATH = \"/checkpoints/llama-3_2-1b-instruct_v2.0\"\n", "\n", "# if you are not using NeMo FW container, modify this path to point to scripts directory\n", "SCRIPTS_PATH = \"/opt/NeMo/scripts\"\n", "\n", "# modify this path if you would like to save results in a different directory\n", "WORKSPACE = \"/workspace\"" ] }, { "cell_type": "code", "execution_count": null, "id": "dca87531-5e91-4857-a06f-b2cac4b6f61c", "metadata": {}, "outputs": [], "source": [ "deploy_script = f\"{SCRIPTS_PATH}/deploy/nlp/deploy_in_fw_oai_server_eval.py\"" ] }, { "cell_type": "code", "execution_count": null, "id": "e1ae4669-6218-47d9-9a02-70c24fbb25d9", "metadata": {}, "outputs": [], "source": [ "deploy_process = subprocess.Popen(\n", " [\"python\", deploy_script, \"--nemo_checkpoint\", CHECKPOINT_PATH], \n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "78f35cf1-3fa3-4ceb-8162-234edb2f2beb", "metadata": {}, "outputs": [], "source": [ "base_url = \"http://0.0.0.0:8886\"\n", "model_name = \"triton_model\"\n", "\n", "completions_url = f\"{base_url}/v1/completions/\"\n", "chat_url = f\"{base_url}/v1/chat/completions/\"" ] }, { "cell_type": "markdown", "id": "54c72002-36ce-4984-9d68-51a94c949195", "metadata": {}, "source": [ "## 2. Defining a custom evaluation workflow" ] }, { "cell_type": "markdown", "id": "0efea168-8c76-4291-b850-7ec05dd6b7b2", "metadata": {}, "source": [ "NVIDIA Evals Factory packages include pre-defined evaluation configurations.\n", "These configurations represent some of the most commonly used evaluation settings and simplify running the most frequently used benchmarks.\n", "\n", "They can be listed using the `list_available_evaluations` function." ] }, { "cell_type": "code", "execution_count": null, "id": "03d30f41-069c-4156-86f7-010319896bba", "metadata": {}, "outputs": [], "source": [ "list_available_evaluations()" ] }, { "cell_type": "markdown", "id": "1f721bf5-36ae-41c3-b42f-7494b1651b2b", "metadata": {}, "source": [ "However, users are not limited to this short list of benchmarks.\n", "If you would like to evaluate a model on a different task from the underlying evaluation harness, you simply need to specify the full configuration.\n", "\n", "For this tutorial, we will use the `wikitext` task from `lm-evaluation-harness`.\n", "Note that for tasks without a predefined configuration you must specify the type in the `\".\"` format.\n", "\n", "Since this task uses the log-likelihoods of the input texts, we need to specify parameters for loading the tokenizer: `\"tokenizer_backend\"` and `\"tokenizer\"`.\n", "For the model used in this example these are `\"huggingface\"` and `\"/checkpoints/llama-3_2-1b-instruct_v2.0/context/nemo_tokenizer\"`, respectively." ] }, { "cell_type": "code", "execution_count": null, "id": "250733b3-fbe2-4da3-bb18-6f357331c241", "metadata": {}, "outputs": [], "source": [ "target_config = EvaluationTarget(api_endpoint={\"url\": completions_url, \"type\": \"completions\"})\n", "eval_config = EvaluationConfig(\n", " type=\"lm-evaluation-harness.wikitext\",\n", " params={\"extra\": {\n", " \"tokenizer_backend\": \"huggingface\",\n", " \"tokenizer\": f\"{CHECKPOINT_PATH}/context/nemo_tokenizer\"},\n", " },\n", " output_dir=f\"{WORKSPACE}/wikitext_results\",\n", ")\n", "\n", "results = api.evaluate(target_cfg=target_config, eval_cfg=eval_config)" ] }, { "cell_type": "markdown", "id": "757e6d42", "metadata": {}, "source": [ "Finally, we can shut the model server down and inspect evaluation results." ] }, { "cell_type": "code", "execution_count": null, "id": "e55c94d0-f302-48d1-bb40-8cbdea323e32", "metadata": {}, "outputs": [], "source": [ "deploy_process.send_signal(signal.SIGINT)" ] }, { "cell_type": "code", "execution_count": null, "id": "5ac073b9-f581-4df3-8225-92086a2a0962", "metadata": {}, "outputs": [], "source": [ "print(json.dumps(results['tasks'], indent=4))" ] }, { "cell_type": "code", "execution_count": null, "id": "78efa6f2-4dc2-49bd-afa1-9d89e5f09fdf", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" } }, "nbformat": 4, "nbformat_minor": 5 }