Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Evaluate LLMs with Hugging Face Lighteval on Amazon SageMaker","local":"evaluate-llms-with-hugging-face-lighteval-on-amazon-sagemaker","sections":[{"title":"2. Prepare the evaluation configuration","local":"2-prepare-the-evaluation-configuration","sections":[],"depth":2},{"title":"3. Evaluate Zephyr 7B on TruthfulQA on Amazon SageMaker","local":"3-evaluate-zephyr-7b-on-truthfulqa-on-amazon-sagemaker","sections":[],"depth":2}],"depth":1}"/> | |
| <link href="/docs/sagemaker/main/en/_app/immutable/entry/start.Dwx3QL-d.js" rel="modulepreload"> | |
| <link href="/docs/sagemaker/main/en/_app/immutable/chunks/De_Y4Zx4.js" rel="modulepreload"> | |
| <link href="/docs/sagemaker/main/en/_app/immutable/chunks/CVOIDYs5.js" rel="modulepreload"> | |
| <link href="/docs/sagemaker/main/en/_app/immutable/entry/app.CWt0UTTa.js" rel="modulepreload"> | |
| <link href="/docs/sagemaker/main/en/_app/immutable/chunks/tHedSEzA.js" rel="modulepreload"> | |
| <link href="/docs/sagemaker/main/en/_app/immutable/chunks/BiDQlvBp.js" rel="modulepreload"> | |
| <link href="/docs/sagemaker/main/en/_app/immutable/chunks/DsnmJJEf.js" rel="modulepreload"> | |
| <link href="/docs/sagemaker/main/en/_app/immutable/chunks/DdFSaKtu.js" rel="modulepreload"> | |
| <link href="/docs/sagemaker/main/en/_app/immutable/nodes/0.DSQJ7Hj8.js" rel="modulepreload"> | |
| <link href="/docs/sagemaker/main/en/_app/immutable/chunks/DRyGbSf1.js" rel="modulepreload"> | |
| <link href="/docs/sagemaker/main/en/_app/immutable/nodes/5.Cjmj0nVa.js" rel="modulepreload"> | |
| <link href="/docs/sagemaker/main/en/_app/immutable/chunks/BmlbDrdc.js" rel="modulepreload"> | |
| <!--rmxped--><meta name="hf:doc:metadata" content="{"title":"Evaluate LLMs with Hugging Face Lighteval on Amazon SageMaker","local":"evaluate-llms-with-hugging-face-lighteval-on-amazon-sagemaker","sections":[{"title":"2. Prepare the evaluation configuration","local":"2-prepare-the-evaluation-configuration","sections":[],"depth":2},{"title":"3. Evaluate Zephyr 7B on TruthfulQA on Amazon SageMaker","local":"3-evaluate-zephyr-7b-on-truthfulqa-on-amazon-sagemaker","sections":[],"depth":2}],"depth":1}"/><!----> | |
| <link href="/docs/sagemaker/main/en/_app/immutable/assets/0.tn0RQdqM.css" rel="modulepreload"> <!--[--><!--[0--><!--[--><!--[0--><!--[--><p></p> <div class="items-center shrink-0 min-w-[100px] max-sm:min-w-[50px] justify-end ml-auto flex" style="float: right; margin-left: 10px; display: inline-flex; position: relative; z-index: 10;"><div class="inline-flex rounded-md max-sm:rounded-sm"><button class="inline-flex items-center gap-1 h-7 max-sm:h-7 px-2 max-sm:px-1.5 text-sm font-medium text-gray-800 border border-r-0 rounded-l-md max-sm:rounded-l-sm border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-live="polite"><span class="inline-flex items-center justify-center rounded-md p-0.5 max-sm:p-0 hover:text-gray-800 dark:hover:text-gray-200"><svg class="sm:size-3.5 size-3" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----></span> <span>Copy page</span></button> <button class="inline-flex items-center justify-center w-6 max-sm:w-5 h-7 max-sm:h-7 disabled:pointer-events-none text-sm text-gray-500 hover:text-gray-700 dark:hover:text-white rounded-r-md max-sm:rounded-r-sm border border-l transition border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-haspopup="menu" aria-expanded="false" aria-label="Open copy menu"><svg class="transition-transform text-gray-400 overflow-visible sm:size-3.5 size-3 rotate-0" width="1em" height="1em" viewBox="0 0 12 7" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M1 1L6 6L11 1" stroke="currentColor"></path></svg><!----></button></div> <!--[-1--><!--]--></div><!----> <!--[0--><h1 class="relative group"><a id="evaluate-llms-with-hugging-face-lighteval-on-amazon-sagemaker" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#evaluate-llms-with-hugging-face-lighteval-on-amazon-sagemaker"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Evaluate LLMs with Hugging Face Lighteval on Amazon SageMaker</span></h1><!--]--><!----> <p>In this sagemaker example, we are going to learn how to evaluate LLMs using Hugging Face <a href="https://github.com/huggingface/lighteval/tree/main" rel="nofollow">lighteval</a>. LightEval is a lightweight LLM evaluation suite that powers <a href="https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard" rel="nofollow">Hugging Face Open LLM Leaderboard</a>.</p> <p>Evaluating LLMs is crucial for understanding their capabilities and limitations, yet it poses significant challenges due to their complex and opaque nature. LightEval facilitates this evaluation process by enabling LLMs to be assessed on acamedic benchmarks like MMLU or IFEval, providing a structured approach to gauge their performance across diverse tasks.</p> <p>In Detail you will learn how to:</p> <ol><li>Setup Development Environment</li> <li>Prepare the evaluation configuraiton</li> <li>Evaluate Zephyr 7B on TruthfulQA on Amazon SageMaker</li></ol> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-python "><!---->!pip install <span class="hljs-string">'sagemaker>=3.0.0'</span> --upgrade --quiet<!----></pre></div><!----> <blockquote class="tip"><p>This example uses the <a href="https://github.com/aws/sagemaker-python-sdk" rel="nofollow">SageMaker Python SDK v3</a>. v3 introduces a new, framework-agnostic API built around <code>ModelTrainer</code> (training) and <code>ModelBuilder</code> (inference), which replaces the v2 <code>HuggingFace</code> and <code>HuggingFaceModel</code> classes.</p><!----></blockquote><!----> <p>If you are going to use Sagemaker in a local environment. You need access to an IAM Role with the required permissions for Sagemaker. You can find <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html" rel="nofollow">here</a> more about it.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-python "><!----><span class="hljs-keyword">import</span> boto3 | |
| <span class="hljs-keyword">from</span> sagemaker.core.helper.session_helper <span class="hljs-keyword">import</span> Session, get_execution_role | |
| sess = Session() | |
| <span class="hljs-comment"># sagemaker session bucket -> used for uploading data, models and logs</span> | |
| <span class="hljs-comment"># sagemaker will automatically create this bucket if it does not exist</span> | |
| sagemaker_session_bucket = sess.default_bucket() | |
| <span class="hljs-keyword">try</span>: | |
| role = get_execution_role() | |
| <span class="hljs-keyword">except</span> Exception: | |
| iam = boto3.client(<span class="hljs-string">'iam'</span>) | |
| role = iam.get_role(RoleName=<span class="hljs-string">'sagemaker_execution_role'</span>)[<span class="hljs-string">'Role'</span>][<span class="hljs-string">'Arn'</span>] | |
| <span class="hljs-built_in">print</span>(<span class="hljs-string">f"sagemaker role arn: <span class="hljs-subst">{role}</span>"</span>) | |
| <span class="hljs-built_in">print</span>(<span class="hljs-string">f"sagemaker bucket: <span class="hljs-subst">{sess.default_bucket()}</span>"</span>) | |
| <span class="hljs-built_in">print</span>(<span class="hljs-string">f"sagemaker session region: <span class="hljs-subst">{sess.boto_region_name}</span>"</span>)<!----></pre></div><!----> <!--[1--><h2 class="relative group"><a id="2-prepare-the-evaluation-configuration" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#2-prepare-the-evaluation-configuration"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>2. Prepare the evaluation configuration</span></h2><!--]--><!----> <p><a href="https://github.com/huggingface/lighteval/tree/main" rel="nofollow">LightEval</a> lets you evaluate LLMs on common benchmarks like MMLU, TruthfulQA, IFEval, and more. It powers the <a href="https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard" rel="nofollow">Hugging Face Open LLM Leaderboard</a> and is built on top of the <a href="https://github.com/EleutherAI/lm-evaluation-harness" rel="nofollow">Eleuther AI Harness</a> with additional features and improvements.</p> <p>You can find all available benchmarks <a href="https://github.com/huggingface/lighteval/blob/main/examples/tasks/all_tasks.txt" rel="nofollow">here</a>.</p> <p>We are going to use Amazon SageMaker Managed Training to evaluate the model. lighteval is now used through its command-line interface (<code>lighteval accelerate</code>); the older <code>run_evals_accelerate.py</code> script has been removed. The training code lives in the <a href="scripts"><code>scripts</code></a> folder next to this notebook and is uploaded to the training job:</p> <ul><li><a href="scripts/requirements.txt"><code>scripts/requirements.txt</code></a> installs lighteval into the Hugging Face DLC (which does not ship with it).</li> <li><a href="scripts/run_lighteval.py"><code>scripts/run_lighteval.py</code></a> is a small launcher that invokes the <code>lighteval accelerate</code> CLI.</li></ul> <p>In lighteval, an evaluation is launched with the <code>lighteval accelerate</code> command. Tasks are passed as a positional argument using the format <code>suite|task|num_few_shot</code> (for example <code>leaderboard|truthfulqa:mc|0</code>). You can evaluate on several tasks at once by passing a comma-separated list (e.g. <code>leaderboard|truthfulqa:mc|0,leaderboard|gsm8k|5</code>) or by pointing to a tasks file whose path starts with <code>./</code>. You can list every available task with <code>lighteval tasks list</code>.</p> <p>We are going to evaluate the model on the TruthfulQA benchmark with 0 few-shot examples. <a href="https://paperswithcode.com/dataset/truthfulqa" rel="nofollow">TruthfulQA</a> is a benchmark designed to measure whether a language model generates truthful answers to questions, encompassing 817 questions across 38 categories including health, law, finance, and politics.</p> <p>To evaluate a model on all the benchmarks of the Open LLM Leaderboard, you can pass the tasks listed in <a href="https://github.com/huggingface/lighteval/blob/v0.13.0/examples/tasks/open_llm_leaderboard_tasks.txt" rel="nofollow">this file</a>.</p> <!--[1--><h2 class="relative group"><a id="3-evaluate-zephyr-7b-on-truthfulqa-on-amazon-sagemaker" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#3-evaluate-zephyr-7b-on-truthfulqa-on-amazon-sagemaker"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>3. Evaluate Zephyr 7B on TruthfulQA on Amazon SageMaker</span></h2><!--]--><!----> <p>In this example we are going to evaluate <a href="https://huggingface.co/HuggingFaceH4/zephyr-7b-beta" rel="nofollow">HuggingFaceH4/zephyr-7b-beta</a> on the TruthfulQA benchmark, which is part of the Open LLM Leaderboard.</p> <p>In addition to the tasks, the <code>lighteval accelerate</code> command takes:</p> <ul><li><code>model-args</code>: a <code>key=value</code> string describing the model, e.g. <code>model_name=HuggingFaceH4/zephyr-7b-beta,dtype=bfloat16</code>. Useful keys include <code>model_name</code> (Hugging Face Model ID or path) and <code>dtype</code> (<code>bfloat16</code>, <code>float16</code> or <code>float32</code>).</li> <li><code>--output-dir</code>: the directory where the evaluation results are saved. We use <code>/opt/ml/model</code> so that SageMaker uploads the results to S3 once the job finishes.</li></ul> <p>You can see all available options with <code>lighteval accelerate --help</code>.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-python "><!----><span class="hljs-keyword">from</span> sagemaker.train.model_trainer <span class="hljs-keyword">import</span> ModelTrainer | |
| <span class="hljs-keyword">from</span> sagemaker.train.configs <span class="hljs-keyword">import</span> SourceCode, Compute, OutputDataConfig | |
| <span class="hljs-keyword">from</span> sagemaker.core <span class="hljs-keyword">import</span> image_uris | |
| <span class="hljs-comment"># evaluation configuration</span> | |
| model_id = <span class="hljs-string">"HuggingFaceH4/zephyr-7b-beta"</span> <span class="hljs-comment"># Hugging Face Model ID to evaluate</span> | |
| task = <span class="hljs-string">"leaderboard|truthfulqa:mc|0"</span> <span class="hljs-comment"># suite|task|num_few_shot (comma-separate for multiple tasks)</span> | |
| model_dtype = <span class="hljs-string">"bfloat16"</span> <span class="hljs-comment"># torch dtype used to load the model weights</span> | |
| output_dir = <span class="hljs-string">"/opt/ml/model"</span> <span class="hljs-comment"># SageMaker uploads this directory to S3 after the job</span> | |
| instance_type = <span class="hljs-string">"ml.g5.4xlarge"</span> | |
| <span class="hljs-comment"># Retrieve the Hugging Face PyTorch training DLC image URI</span> | |
| training_image = image_uris.retrieve( | |
| framework=<span class="hljs-string">"huggingface"</span>, | |
| region=sess.boto_region_name, | |
| version=<span class="hljs-string">"4.56.2"</span>, | |
| base_framework_version=<span class="hljs-string">"pytorch2.8.0"</span>, | |
| py_version=<span class="hljs-string">"py312"</span>, | |
| image_scope=<span class="hljs-string">"training"</span>, | |
| instance_type=instance_type, | |
| ) | |
| <span class="hljs-comment"># lighteval is invoked through its CLI. SageMaker installs requirements.txt first, then runs this command.</span> | |
| <span class="hljs-comment"># The model-args and task are single-quoted so the shell keeps the `,` and `|` characters intact.</span> | |
| command = ( | |
| <span class="hljs-string">"python run_lighteval.py accelerate "</span> | |
| <span class="hljs-string">f"'model_name=<span class="hljs-subst">{model_id}</span>,dtype=<span class="hljs-subst">{model_dtype}</span>' "</span> | |
| <span class="hljs-string">f"'<span class="hljs-subst">{task}</span>' "</span> | |
| <span class="hljs-string">f"--output-dir <span class="hljs-subst">{output_dir}</span>"</span> | |
| ) | |
| <span class="hljs-comment"># create the ModelTrainer</span> | |
| huggingface_estimator = ModelTrainer( | |
| sagemaker_session=sess, | |
| role=role, | |
| base_job_name=<span class="hljs-string">"lighteval"</span>, <span class="hljs-comment"># the name of the training job</span> | |
| training_image=training_image, | |
| source_code=SourceCode( | |
| source_dir=<span class="hljs-string">"scripts"</span>, <span class="hljs-comment"># directory uploaded to the job (contains requirements.txt)</span> | |
| requirements=<span class="hljs-string">"requirements.txt"</span>, <span class="hljs-comment"># dependencies installed before running the command</span> | |
| command=command, <span class="hljs-comment"># lighteval CLI invocation</span> | |
| ), | |
| compute=Compute( | |
| instance_type=instance_type, <span class="hljs-comment"># instance type used for the evaluation job</span> | |
| instance_count=<span class="hljs-number">1</span>, <span class="hljs-comment"># the number of instances used</span> | |
| volume_size_in_gb=<span class="hljs-number">300</span>, <span class="hljs-comment"># the size of the EBS volume in GB</span> | |
| ), | |
| output_data_config=OutputDataConfig( | |
| s3_output_path=<span class="hljs-string">f"s3://<span class="hljs-subst">{sess.default_bucket()}</span>/lighteval/output"</span>, | |
| ), | |
| environment={ | |
| <span class="hljs-string">"HUGGINGFACE_HUB_CACHE"</span>: <span class="hljs-string">"/tmp/.cache"</span>, | |
| <span class="hljs-comment"># "HF_TOKEN": "REPLACE_WITH_YOUR_TOKEN" # needed for gated/private models</span> | |
| }, <span class="hljs-comment"># set env variable to cache models in /tmp</span> | |
| )<!----></pre></div><!----> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-python "><!----><span class="hljs-comment"># start the evaluation job</span> | |
| huggingface_estimator.train(wait=<span class="hljs-literal">True</span>)<!----></pre></div><!----> <p>After the evaluation job is finished, we can download the evaluation results from the S3 bucket. Lighteval will save the results and generations in the <code>output_dir</code>. The results are savedas json and include detailed information about each task and the model’s performance. The results are available in the <code>results</code> key.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-python "><!----><span class="hljs-keyword">import</span> tarfile | |
| <span class="hljs-keyword">import</span> json | |
| <span class="hljs-keyword">import</span> io | |
| <span class="hljs-keyword">import</span> os | |
| <span class="hljs-keyword">import</span> boto3 | |
| <span class="hljs-keyword">from</span> urllib.parse <span class="hljs-keyword">import</span> urlparse | |
| <span class="hljs-comment"># get the S3 URI of the uploaded output artifacts (model.tar.gz)</span> | |
| training_job = huggingface_estimator._latest_training_job | |
| model_data = training_job.model_artifacts.s3_model_artifacts | |
| <span class="hljs-comment"># download the artifacts from s3</span> | |
| parsed = urlparse(model_data) | |
| s3 = boto3.client(<span class="hljs-string">"s3"</span>) | |
| results_tar = s3.get_object(Bucket=parsed.netloc, Key=parsed.path.lstrip(<span class="hljs-string">"/"</span>))[<span class="hljs-string">"Body"</span>].read() | |
| result = {} | |
| <span class="hljs-comment"># lighteval writes the scores to results/<model_id>/results_<timestamp>.json</span> | |
| <span class="hljs-keyword">with</span> tarfile.<span class="hljs-built_in">open</span>(fileobj=io.BytesIO(results_tar), mode=<span class="hljs-string">"r:gz"</span>) <span class="hljs-keyword">as</span> tar: | |
| <span class="hljs-keyword">for</span> member <span class="hljs-keyword">in</span> tar.getmembers(): | |
| <span class="hljs-keyword">if</span> os.path.join(<span class="hljs-string">"results"</span>, model_id) <span class="hljs-keyword">in</span> member.name <span class="hljs-keyword">and</span> member.name.endswith(<span class="hljs-string">".json"</span>): | |
| f = tar.extractfile(member) | |
| <span class="hljs-keyword">if</span> f <span class="hljs-keyword">is</span> <span class="hljs-keyword">not</span> <span class="hljs-literal">None</span>: | |
| result = json.loads(f.read()) | |
| <span class="hljs-keyword">break</span> | |
| <span class="hljs-comment"># print results</span> | |
| <span class="hljs-built_in">print</span>(result[<span class="hljs-string">"results"</span>]) | |
| <span class="hljs-comment"># {'leaderboard|truthfulqa:mc|0': {'truthfulqa_mc1': 0.406, 'truthfulqa_mc2': 0.575, ...}, 'all': {...}}</span><!----></pre></div><!----> <hr/> <blockquote class="tip"><p>📍 Find the complete example on GitHub <a href="https://github.com/huggingface/hub-docs/tree/main/docs/sagemaker/notebooks/sagemaker-sdk/evaluate-llm-lighteval/sagemaker-notebook.ipynb" rel="nofollow">here</a>!</p><!----></blockquote><!----> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/hub-docs/blob/main/docs/sagemaker/source/examples/sagemaker-sdk-evaluate-llm-lighteval.mdx" target="_blank"><svg class="mr-1" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M31,16l-7,7l-1.41-1.41L28.17,16l-5.58-5.59L24,9l7,7z"></path><path d="M1,16l7-7l1.41,1.41L3.83,16l5.58,5.59L8,23l-7-7z"></path><path d="M12.419,25.484L17.639,6.552l1.932,0.518L14.351,26.002z"></path></svg><!----> <span><span class="underline">Update</span> on GitHub</span></a><!----> <p></p><!--]--><!----><!--]--><!--]--><!--]--> <!--[-1--><!--]--><!--]--> | |
| <script> | |
| { | |
| __sveltekit_142mtmn = { | |
| base: "/docs/sagemaker/main/en", | |
| assets: "/docs/sagemaker/main/en" | |
| }; | |
| const element = document.currentScript.parentElement; | |
| Promise.all([ | |
| import("/docs/sagemaker/main/en/_app/immutable/entry/start.Dwx3QL-d.js"), | |
| import("/docs/sagemaker/main/en/_app/immutable/entry/app.CWt0UTTa.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 5], | |
| data: [null,null], | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 29.7 kB
- Xet hash:
- 47602e2339366eac0855abd447d512934e183857302c24822721890cb2ad8aef
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.