Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"LoRA for token classification","local":"lora-for-token-classification","sections":[{"title":"Setup","local":"setup","sections":[],"depth":2},{"title":"Load dataset and metric","local":"load-dataset-and-metric","sections":[],"depth":2},{"title":"Preprocess dataset","local":"preprocess-dataset","sections":[],"depth":2},{"title":"Train","local":"train","sections":[],"depth":2},{"title":"Share model","local":"share-model","sections":[],"depth":2},{"title":"Inference","local":"inference","sections":[],"depth":2}],"depth":1}"> | |
| <link href="/docs/peft/main/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/entry/start.f1bde2f6.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/chunks/scheduler.d627b047.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/chunks/singletons.eea5c746.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/chunks/index.a57a1c33.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/chunks/paths.e43091d4.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/entry/app.182b7f64.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/chunks/index.d48c4817.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/nodes/0.8a16be44.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/chunks/each.e59479a4.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/nodes/40.9927d2a1.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/chunks/Tip.9bd3babf.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/chunks/CodeBlock.5da89496.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/chunks/Heading.47e562a9.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{"title":"LoRA for token classification","local":"lora-for-token-classification","sections":[{"title":"Setup","local":"setup","sections":[],"depth":2},{"title":"Load dataset and metric","local":"load-dataset-and-metric","sections":[],"depth":2},{"title":"Preprocess dataset","local":"preprocess-dataset","sections":[],"depth":2},{"title":"Train","local":"train","sections":[],"depth":2},{"title":"Share model","local":"share-model","sections":[],"depth":2},{"title":"Inference","local":"inference","sections":[],"depth":2}],"depth":1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <h1 class="relative group"><a id="lora-for-token-classification" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#lora-for-token-classification"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>LoRA for token classification</span></h1> <p data-svelte-h="svelte-353m1i">Low-Rank Adaptation (LoRA) is a reparametrization method that aims to reduce the number of trainable parameters with low-rank representations. The weight matrix is broken down into low-rank matrices that are trained and updated. All the pretrained model parameters remain frozen. After training, the low-rank matrices are added back to the original weights. This makes it more efficient to store and train a LoRA model because there are significantly fewer parameters.</p> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-kt6e6m">💡 Read <a href="https://arxiv.org/abs/2106.09685" rel="nofollow">LoRA: Low-Rank Adaptation of Large Language Models</a> to learn more about LoRA.</p></div> <p data-svelte-h="svelte-1hsuxns">This guide will show you how to train a <a href="https://huggingface.co/roberta-large" rel="nofollow"><code>roberta-large</code></a> model with LoRA on the <a href="https://huggingface.co/datasets/tner/bionlp2004" rel="nofollow">BioNLP2004</a> dataset for token classification.</p> <p data-svelte-h="svelte-1c9nexd">Before you begin, make sure you have all the necessary libraries installed:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->!pip install -q peft transformers datasets evaluate seqeval<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="setup" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#setup"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Setup</span></h2> <p data-svelte-h="svelte-1quxkd3">Let’s start by importing all the necessary libraries you’ll need:</p> <ul data-svelte-h="svelte-1vdqu50"><li>🤗 Transformers for loading the base <code>roberta-large</code> model and tokenizer, and handling the training loop</li> <li>🤗 Datasets for loading and preparing the <code>bionlp2004</code> dataset for training</li> <li>🤗 Evaluate for evaluating the model’s performance</li> <li>🤗 PEFT for setting up the LoRA configuration and creating the PEFT model</li></ul> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> datasets <span class="hljs-keyword">import</span> load_dataset | |
| <span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> ( | |
| AutoModelForTokenClassification, | |
| AutoTokenizer, | |
| DataCollatorForTokenClassification, | |
| TrainingArguments, | |
| Trainer, | |
| ) | |
| <span class="hljs-keyword">from</span> peft <span class="hljs-keyword">import</span> get_peft_config, PeftModel, PeftConfig, get_peft_model, LoraConfig, TaskType | |
| <span class="hljs-keyword">import</span> evaluate | |
| <span class="hljs-keyword">import</span> torch | |
| <span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np | |
| model_checkpoint = <span class="hljs-string">"roberta-large"</span> | |
| lr = <span class="hljs-number">1e-3</span> | |
| batch_size = <span class="hljs-number">16</span> | |
| num_epochs = <span class="hljs-number">10</span><!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="load-dataset-and-metric" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#load-dataset-and-metric"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Load dataset and metric</span></h2> <p data-svelte-h="svelte-1qbccnw">The <a href="https://huggingface.co/datasets/tner/bionlp2004" rel="nofollow">BioNLP2004</a> dataset includes tokens and tags for biological structures like DNA, RNA and proteins. Load the dataset:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->bionlp = load_dataset(<span class="hljs-string">"tner/bionlp2004"</span>) | |
| bionlp[<span class="hljs-string">"train"</span>][<span class="hljs-number">0</span>] | |
| { | |
| <span class="hljs-string">"tokens"</span>: [ | |
| <span class="hljs-string">"Since"</span>, | |
| <span class="hljs-string">"HUVECs"</span>, | |
| <span class="hljs-string">"released"</span>, | |
| <span class="hljs-string">"superoxide"</span>, | |
| <span class="hljs-string">"anions"</span>, | |
| <span class="hljs-string">"in"</span>, | |
| <span class="hljs-string">"response"</span>, | |
| <span class="hljs-string">"to"</span>, | |
| <span class="hljs-string">"TNF"</span>, | |
| <span class="hljs-string">","</span>, | |
| <span class="hljs-string">"and"</span>, | |
| <span class="hljs-string">"H2O2"</span>, | |
| <span class="hljs-string">"induces"</span>, | |
| <span class="hljs-string">"VCAM-1"</span>, | |
| <span class="hljs-string">","</span>, | |
| <span class="hljs-string">"PDTC"</span>, | |
| <span class="hljs-string">"may"</span>, | |
| <span class="hljs-string">"act"</span>, | |
| <span class="hljs-string">"as"</span>, | |
| <span class="hljs-string">"a"</span>, | |
| <span class="hljs-string">"radical"</span>, | |
| <span class="hljs-string">"scavenger"</span>, | |
| <span class="hljs-string">"."</span>, | |
| ], | |
| <span class="hljs-string">"tags"</span>: [<span class="hljs-number">0</span>, <span class="hljs-number">7</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">3</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>], | |
| }<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1i5fqx3">The <code>tags</code> values are defined in the label ids <a href="https://huggingface.co/datasets/tner/bionlp2004#label-id" rel="nofollow">dictionary</a>. The letter that prefixes each label indicates the token position: <code>B</code> is for the first token of an entity, <code>I</code> is for a token inside the entity, and <code>0</code> is for a token that is not part of an entity.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->{ | |
| <span class="hljs-string">"O"</span>: <span class="hljs-number">0</span>, | |
| <span class="hljs-string">"B-DNA"</span>: <span class="hljs-number">1</span>, | |
| <span class="hljs-string">"I-DNA"</span>: <span class="hljs-number">2</span>, | |
| <span class="hljs-string">"B-protein"</span>: <span class="hljs-number">3</span>, | |
| <span class="hljs-string">"I-protein"</span>: <span class="hljs-number">4</span>, | |
| <span class="hljs-string">"B-cell_type"</span>: <span class="hljs-number">5</span>, | |
| <span class="hljs-string">"I-cell_type"</span>: <span class="hljs-number">6</span>, | |
| <span class="hljs-string">"B-cell_line"</span>: <span class="hljs-number">7</span>, | |
| <span class="hljs-string">"I-cell_line"</span>: <span class="hljs-number">8</span>, | |
| <span class="hljs-string">"B-RNA"</span>: <span class="hljs-number">9</span>, | |
| <span class="hljs-string">"I-RNA"</span>: <span class="hljs-number">10</span>, | |
| }<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1sgw51e">Then load the <a href="https://huggingface.co/spaces/evaluate-metric/seqeval" rel="nofollow"><code>seqeval</code></a> framework which includes several metrics - precision, accuracy, F1, and recall - for evaluating sequence labeling tasks.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->seqeval = evaluate.load(<span class="hljs-string">"seqeval"</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-96unc4">Now you can write an evaluation function to compute the metrics from the model predictions and labels, and return the precision, recall, F1, and accuracy scores:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->label_list = [ | |
| <span class="hljs-string">"O"</span>, | |
| <span class="hljs-string">"B-DNA"</span>, | |
| <span class="hljs-string">"I-DNA"</span>, | |
| <span class="hljs-string">"B-protein"</span>, | |
| <span class="hljs-string">"I-protein"</span>, | |
| <span class="hljs-string">"B-cell_type"</span>, | |
| <span class="hljs-string">"I-cell_type"</span>, | |
| <span class="hljs-string">"B-cell_line"</span>, | |
| <span class="hljs-string">"I-cell_line"</span>, | |
| <span class="hljs-string">"B-RNA"</span>, | |
| <span class="hljs-string">"I-RNA"</span>, | |
| ] | |
| <span class="hljs-keyword">def</span> <span class="hljs-title function_">compute_metrics</span>(<span class="hljs-params">p</span>): | |
| predictions, labels = p | |
| predictions = np.argmax(predictions, axis=<span class="hljs-number">2</span>) | |
| true_predictions = [ | |
| [label_list[p] <span class="hljs-keyword">for</span> (p, l) <span class="hljs-keyword">in</span> <span class="hljs-built_in">zip</span>(prediction, label) <span class="hljs-keyword">if</span> l != -<span class="hljs-number">100</span>] | |
| <span class="hljs-keyword">for</span> prediction, label <span class="hljs-keyword">in</span> <span class="hljs-built_in">zip</span>(predictions, labels) | |
| ] | |
| true_labels = [ | |
| [label_list[l] <span class="hljs-keyword">for</span> (p, l) <span class="hljs-keyword">in</span> <span class="hljs-built_in">zip</span>(prediction, label) <span class="hljs-keyword">if</span> l != -<span class="hljs-number">100</span>] | |
| <span class="hljs-keyword">for</span> prediction, label <span class="hljs-keyword">in</span> <span class="hljs-built_in">zip</span>(predictions, labels) | |
| ] | |
| results = seqeval.compute(predictions=true_predictions, references=true_labels) | |
| <span class="hljs-keyword">return</span> { | |
| <span class="hljs-string">"precision"</span>: results[<span class="hljs-string">"overall_precision"</span>], | |
| <span class="hljs-string">"recall"</span>: results[<span class="hljs-string">"overall_recall"</span>], | |
| <span class="hljs-string">"f1"</span>: results[<span class="hljs-string">"overall_f1"</span>], | |
| <span class="hljs-string">"accuracy"</span>: results[<span class="hljs-string">"overall_accuracy"</span>], | |
| }<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="preprocess-dataset" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#preprocess-dataset"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Preprocess dataset</span></h2> <p data-svelte-h="svelte-7igb10">Initialize a tokenizer and make sure you set <code>is_split_into_words=True</code> because the text sequence has already been split into words. However, this doesn’t mean it is tokenized yet (even though it may look like it!), and you’ll need to further tokenize the words into subwords.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, add_prefix_space=<span class="hljs-literal">True</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-12p7a1j">You’ll also need to write a function to:</p> <ol data-svelte-h="svelte-dy65zz"><li>Map each token to their respective word with the <a href="https://huggingface.co/docs/transformers/main/en/main_classes/tokenizer#transformers.BatchEncoding.word_ids" rel="nofollow">word_ids</a> method.</li> <li>Ignore the special tokens by setting them to <code>-100</code>.</li> <li>Label the first token of a given entity.</li></ol> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">def</span> <span class="hljs-title function_">tokenize_and_align_labels</span>(<span class="hljs-params">examples</span>): | |
| tokenized_inputs = tokenizer(examples[<span class="hljs-string">"tokens"</span>], truncation=<span class="hljs-literal">True</span>, is_split_into_words=<span class="hljs-literal">True</span>) | |
| labels = [] | |
| <span class="hljs-keyword">for</span> i, label <span class="hljs-keyword">in</span> <span class="hljs-built_in">enumerate</span>(examples[<span class="hljs-string">f"tags"</span>]): | |
| word_ids = tokenized_inputs.word_ids(batch_index=i) | |
| previous_word_idx = <span class="hljs-literal">None</span> | |
| label_ids = [] | |
| <span class="hljs-keyword">for</span> word_idx <span class="hljs-keyword">in</span> word_ids: | |
| <span class="hljs-keyword">if</span> word_idx <span class="hljs-keyword">is</span> <span class="hljs-literal">None</span>: | |
| label_ids.append(-<span class="hljs-number">100</span>) | |
| <span class="hljs-keyword">elif</span> word_idx != previous_word_idx: | |
| label_ids.append(label[word_idx]) | |
| <span class="hljs-keyword">else</span>: | |
| label_ids.append(-<span class="hljs-number">100</span>) | |
| previous_word_idx = word_idx | |
| labels.append(label_ids) | |
| tokenized_inputs[<span class="hljs-string">"labels"</span>] = labels | |
| <span class="hljs-keyword">return</span> tokenized_inputs<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-shbkgz">Use <a href="https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.map" rel="nofollow">map</a> to apply the <code>tokenize_and_align_labels</code> function to the dataset:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->tokenized_bionlp = bionlp.<span class="hljs-built_in">map</span>(tokenize_and_align_labels, batched=<span class="hljs-literal">True</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-yafogm">Finally, create a data collator to pad the examples to the longest length in a batch:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer)<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="train" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#train"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Train</span></h2> <p data-svelte-h="svelte-189x64j">Now you’re ready to create a <a href="/docs/peft/main/en/package_reference/peft_model#peft.PeftModel">PeftModel</a>. Start by loading the base <code>roberta-large</code> model, the number of expected labels, and the <code>id2label</code> and <code>label2id</code> dictionaries:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->id2label = { | |
| <span class="hljs-number">0</span>: <span class="hljs-string">"O"</span>, | |
| <span class="hljs-number">1</span>: <span class="hljs-string">"B-DNA"</span>, | |
| <span class="hljs-number">2</span>: <span class="hljs-string">"I-DNA"</span>, | |
| <span class="hljs-number">3</span>: <span class="hljs-string">"B-protein"</span>, | |
| <span class="hljs-number">4</span>: <span class="hljs-string">"I-protein"</span>, | |
| <span class="hljs-number">5</span>: <span class="hljs-string">"B-cell_type"</span>, | |
| <span class="hljs-number">6</span>: <span class="hljs-string">"I-cell_type"</span>, | |
| <span class="hljs-number">7</span>: <span class="hljs-string">"B-cell_line"</span>, | |
| <span class="hljs-number">8</span>: <span class="hljs-string">"I-cell_line"</span>, | |
| <span class="hljs-number">9</span>: <span class="hljs-string">"B-RNA"</span>, | |
| <span class="hljs-number">10</span>: <span class="hljs-string">"I-RNA"</span>, | |
| } | |
| label2id = { | |
| <span class="hljs-string">"O"</span>: <span class="hljs-number">0</span>, | |
| <span class="hljs-string">"B-DNA"</span>: <span class="hljs-number">1</span>, | |
| <span class="hljs-string">"I-DNA"</span>: <span class="hljs-number">2</span>, | |
| <span class="hljs-string">"B-protein"</span>: <span class="hljs-number">3</span>, | |
| <span class="hljs-string">"I-protein"</span>: <span class="hljs-number">4</span>, | |
| <span class="hljs-string">"B-cell_type"</span>: <span class="hljs-number">5</span>, | |
| <span class="hljs-string">"I-cell_type"</span>: <span class="hljs-number">6</span>, | |
| <span class="hljs-string">"B-cell_line"</span>: <span class="hljs-number">7</span>, | |
| <span class="hljs-string">"I-cell_line"</span>: <span class="hljs-number">8</span>, | |
| <span class="hljs-string">"B-RNA"</span>: <span class="hljs-number">9</span>, | |
| <span class="hljs-string">"I-RNA"</span>: <span class="hljs-number">10</span>, | |
| } | |
| model = AutoModelForTokenClassification.from_pretrained( | |
| model_checkpoint, num_labels=<span class="hljs-number">11</span>, id2label=id2label, label2id=label2id | |
| )<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1nj5zad">Define the <a href="/docs/peft/main/en/package_reference/lora#peft.LoraConfig">LoraConfig</a> with:</p> <ul data-svelte-h="svelte-y18ek6"><li><code>task_type</code>, token classification (<code>TaskType.TOKEN_CLS</code>)</li> <li><code>r</code>, the dimension of the low-rank matrices</li> <li><code>lora_alpha</code>, scaling factor for the weight matrices</li> <li><code>lora_dropout</code>, dropout probability of the LoRA layers</li> <li><code>bias</code>, set to <code>all</code> to train all bias parameters</li></ul> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-wrjzuj">💡 The weight matrix is scaled by <code>lora_alpha/r</code>, and a higher <code>lora_alpha</code> value assigns more weight to the LoRA activations. For performance, we recommend setting <code>bias</code> to <code>None</code> first, and then <code>lora_only</code>, before trying <code>all</code>.</p></div> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->peft_config = LoraConfig( | |
| task_type=TaskType.TOKEN_CLS, inference_mode=<span class="hljs-literal">False</span>, r=<span class="hljs-number">16</span>, lora_alpha=<span class="hljs-number">16</span>, lora_dropout=<span class="hljs-number">0.1</span>, bias=<span class="hljs-string">"all"</span> | |
| )<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1un2a94">Pass the base model and <code>peft_config</code> to the <a href="/docs/peft/main/en/package_reference/peft_model#peft.get_peft_model">get_peft_model()</a> function to create a <a href="/docs/peft/main/en/package_reference/peft_model#peft.PeftModel">PeftModel</a>. You can check out how much more efficient training the <a href="/docs/peft/main/en/package_reference/peft_model#peft.PeftModel">PeftModel</a> is compared to fully training the base model by printing out the trainable parameters:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->model = get_peft_model(model, peft_config) | |
| model.print_trainable_parameters() | |
| <span class="hljs-string">"trainable params: 1855499 || all params: 355894283 || trainable%: 0.5213624069370061"</span><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1yqceda">From the 🤗 Transformers library, create a <a href="https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments" rel="nofollow">TrainingArguments</a> class and specify where you want to save the model to, the training hyperparameters, how to evaluate the model, and when to save the checkpoints:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->training_args = TrainingArguments( | |
| output_dir=<span class="hljs-string">"roberta-large-lora-token-classification"</span>, | |
| learning_rate=lr, | |
| per_device_train_batch_size=batch_size, | |
| per_device_eval_batch_size=batch_size, | |
| num_train_epochs=num_epochs, | |
| weight_decay=<span class="hljs-number">0.01</span>, | |
| evaluation_strategy=<span class="hljs-string">"epoch"</span>, | |
| save_strategy=<span class="hljs-string">"epoch"</span>, | |
| load_best_model_at_end=<span class="hljs-literal">True</span>, | |
| )<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-4kkvlh">Pass the model, <code>TrainingArguments</code>, datasets, tokenizer, data collator and evaluation function to the <a href="https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer" rel="nofollow">Trainer</a> class. The <code>Trainer</code> handles the training loop for you, and when you’re ready, call <a href="https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer.train" rel="nofollow">train</a> to begin!</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->trainer = Trainer( | |
| model=model, | |
| args=training_args, | |
| train_dataset=tokenized_bionlp[<span class="hljs-string">"train"</span>], | |
| eval_dataset=tokenized_bionlp[<span class="hljs-string">"validation"</span>], | |
| tokenizer=tokenizer, | |
| data_collator=data_collator, | |
| compute_metrics=compute_metrics, | |
| ) | |
| trainer.train()<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="share-model" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#share-model"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Share model</span></h2> <p data-svelte-h="svelte-1vig879">Once training is complete, you can store and share your model on the Hub if you’d like. Log in to your Hugging Face account and enter your token when prompted:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> huggingface_hub <span class="hljs-keyword">import</span> notebook_login | |
| notebook_login()<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-hu6253">Upload the model to a specific model repository on the Hub with the <a href="https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.push_to_hub" rel="nofollow">push_to_hub</a> method:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->model.push_to_hub(<span class="hljs-string">"your-name/roberta-large-lora-token-classification"</span>)<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="inference" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#inference"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Inference</span></h2> <p data-svelte-h="svelte-1aj5h8f">To use your model for inference, load the configuration and model:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->peft_model_id = <span class="hljs-string">"stevhliu/roberta-large-lora-token-classification"</span> | |
| config = PeftConfig.from_pretrained(peft_model_id) | |
| inference_model = AutoModelForTokenClassification.from_pretrained( | |
| config.base_model_name_or_path, num_labels=<span class="hljs-number">11</span>, id2label=id2label, label2id=label2id | |
| ) | |
| tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path) | |
| model = PeftModel.from_pretrained(inference_model, peft_model_id)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-4ied6n">Get some text to tokenize:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->text = <span class="hljs-string">"The activation of IL-2 gene expression and NF-kappa B through CD28 requires reactive oxygen production by 5-lipoxygenase."</span> | |
| inputs = tokenizer(text, return_tensors=<span class="hljs-string">"pt"</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-179hhpe">Pass the inputs to the model, and print out the model prediction for each token:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">with</span> torch.no_grad(): | |
| logits = model(**inputs).logits | |
| tokens = inputs.tokens() | |
| predictions = torch.argmax(logits, dim=<span class="hljs-number">2</span>) | |
| <span class="hljs-keyword">for</span> token, prediction <span class="hljs-keyword">in</span> <span class="hljs-built_in">zip</span>(tokens, predictions[<span class="hljs-number">0</span>].numpy()): | |
| <span class="hljs-built_in">print</span>((token, model.config.id2label[prediction])) | |
| (<span class="hljs-string">"<s>"</span>, <span class="hljs-string">"O"</span>) | |
| (<span class="hljs-string">"The"</span>, <span class="hljs-string">"O"</span>) | |
| (<span class="hljs-string">"Ġactivation"</span>, <span class="hljs-string">"O"</span>) | |
| (<span class="hljs-string">"Ġof"</span>, <span class="hljs-string">"O"</span>) | |
| (<span class="hljs-string">"ĠIL"</span>, <span class="hljs-string">"B-DNA"</span>) | |
| (<span class="hljs-string">"-"</span>, <span class="hljs-string">"O"</span>) | |
| (<span class="hljs-string">"2"</span>, <span class="hljs-string">"I-DNA"</span>) | |
| (<span class="hljs-string">"Ġgene"</span>, <span class="hljs-string">"O"</span>) | |
| (<span class="hljs-string">"Ġexpression"</span>, <span class="hljs-string">"O"</span>) | |
| (<span class="hljs-string">"Ġand"</span>, <span class="hljs-string">"O"</span>) | |
| (<span class="hljs-string">"ĠNF"</span>, <span class="hljs-string">"B-protein"</span>) | |
| (<span class="hljs-string">"-"</span>, <span class="hljs-string">"O"</span>) | |
| (<span class="hljs-string">"k"</span>, <span class="hljs-string">"I-protein"</span>) | |
| (<span class="hljs-string">"appa"</span>, <span class="hljs-string">"I-protein"</span>) | |
| (<span class="hljs-string">"ĠB"</span>, <span class="hljs-string">"I-protein"</span>) | |
| (<span class="hljs-string">"Ġthrough"</span>, <span class="hljs-string">"O"</span>) | |
| (<span class="hljs-string">"ĠCD"</span>, <span class="hljs-string">"B-protein"</span>) | |
| (<span class="hljs-string">"28"</span>, <span class="hljs-string">"I-protein"</span>) | |
| (<span class="hljs-string">"Ġrequires"</span>, <span class="hljs-string">"O"</span>) | |
| (<span class="hljs-string">"Ġreactive"</span>, <span class="hljs-string">"O"</span>) | |
| (<span class="hljs-string">"Ġoxygen"</span>, <span class="hljs-string">"O"</span>) | |
| (<span class="hljs-string">"Ġproduction"</span>, <span class="hljs-string">"O"</span>) | |
| (<span class="hljs-string">"Ġby"</span>, <span class="hljs-string">"O"</span>) | |
| (<span class="hljs-string">"Ġ5"</span>, <span class="hljs-string">"B-protein"</span>) | |
| (<span class="hljs-string">"-"</span>, <span class="hljs-string">"O"</span>) | |
| (<span class="hljs-string">"lip"</span>, <span class="hljs-string">"I-protein"</span>) | |
| (<span class="hljs-string">"oxy"</span>, <span class="hljs-string">"I-protein"</span>) | |
| (<span class="hljs-string">"gen"</span>, <span class="hljs-string">"I-protein"</span>) | |
| (<span class="hljs-string">"ase"</span>, <span class="hljs-string">"I-protein"</span>) | |
| (<span class="hljs-string">"."</span>, <span class="hljs-string">"O"</span>) | |
| (<span class="hljs-string">"</s>"</span>, <span class="hljs-string">"O"</span>)<!-- HTML_TAG_END --></pre></div> <p></p> | |
| <script> | |
| { | |
| __sveltekit_c8wad6 = { | |
| assets: "/docs/peft/main/en", | |
| base: "/docs/peft/main/en", | |
| env: {} | |
| }; | |
| const element = document.currentScript.parentElement; | |
| const data = [null,null]; | |
| Promise.all([ | |
| import("/docs/peft/main/en/_app/immutable/entry/start.f1bde2f6.js"), | |
| import("/docs/peft/main/en/_app/immutable/entry/app.182b7f64.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 40], | |
| data, | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 61.2 kB
- Xet hash:
- d67de8982904068bba7fcc1cace3d342172cfe4f8bd9153f173802094bfacb44
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.