Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Benchmarks","local":"benchmarks","sections":[{"title":"How to benchmark 🤗 Transformers models","local":"how-to-benchmark--transformers-models","sections":[],"depth":2},{"title":"Benchmark best practices","local":"benchmark-best-practices","sections":[],"depth":2},{"title":"Sharing your benchmark","local":"sharing-your-benchmark","sections":[],"depth":2}],"depth":1}"> | |
| <link href="/docs/transformers/pr_33913/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/entry/start.b67f883f.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/scheduler.25b97de1.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/singletons.62a184e0.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/index.e188933d.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/paths.51881b9e.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/entry/app.e436b1f2.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/index.d9030fc9.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/nodes/0.05e395f5.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/each.e59479a4.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/nodes/9.9af70843.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/Tip.baa67368.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/CodeBlock.e6cd0d95.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/DocNotebookDropdown.5ea6cb78.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/globals.7f7f1b26.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/Markdown.7217f838.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/stores.c3f24f16.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/EditOnGithub.91d95064.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{"title":"Benchmarks","local":"benchmarks","sections":[{"title":"How to benchmark 🤗 Transformers models","local":"how-to-benchmark--transformers-models","sections":[],"depth":2},{"title":"Benchmark best practices","local":"benchmark-best-practices","sections":[],"depth":2},{"title":"Sharing your benchmark","local":"sharing-your-benchmark","sections":[],"depth":2}],"depth":1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <h1 class="relative group"><a id="benchmarks" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#benchmarks"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Benchmarks</span></h1> <div class="course-tip course-tip-orange bg-gradient-to-br dark:bg-gradient-to-r before:border-orange-500 dark:before:border-orange-800 from-orange-50 dark:from-gray-900 to-white dark:to-gray-950 border border-orange-50 text-orange-700 dark:text-gray-400"><p data-svelte-h="svelte-ef5x4f">Hugging Face’s Benchmarking tools are deprecated and it is advised to use external Benchmarking libraries to measure the speed | |
| and memory complexity of Transformer models.</p></div> <div class="flex space-x-1 absolute z-10 right-0 top-0"> <div class="relative colab-dropdown "> <button class=" " type="button"> <img alt="Open In Colab" class="!m-0" src="https://colab.research.google.com/assets/colab-badge.svg"> </button> </div> <div class="relative colab-dropdown "> <button class=" " type="button"> <img alt="Open In Studio Lab" class="!m-0" src="https://studiolab.sagemaker.aws/studiolab.svg"> </button> </div></div> <p data-svelte-h="svelte-13lcah">Let’s take a look at how 🤗 Transformers models can be benchmarked, best practices, and already available benchmarks.</p> <p data-svelte-h="svelte-r2a0pt">A notebook explaining in more detail how to benchmark 🤗 Transformers models can be found <a href="https://github.com/huggingface/notebooks/tree/main/examples/benchmark.ipynb" rel="nofollow">here</a>.</p> <h2 class="relative group"><a id="how-to-benchmark--transformers-models" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#how-to-benchmark--transformers-models"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>How to benchmark 🤗 Transformers models</span></h2> <p data-svelte-h="svelte-1flsss8">The classes <code>PyTorchBenchmark</code> and <code>TensorFlowBenchmark</code> allow to flexibly benchmark 🤗 Transformers models. The benchmark classes allow us to measure the <em>peak memory usage</em> and <em>required time</em> for both <em>inference</em> and <em>training</em>.</p> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-1cagnud">Here, <em>inference</em> is defined by a single forward pass, and <em>training</em> is defined by a single forward pass and | |
| backward pass.</p></div> <p data-svelte-h="svelte-1ehry66">The benchmark classes <code>PyTorchBenchmark</code> and <code>TensorFlowBenchmark</code> expect an object of type <code>PyTorchBenchmarkArguments</code> and | |
| <code>TensorFlowBenchmarkArguments</code>, respectively, for instantiation. <code>PyTorchBenchmarkArguments</code> and <code>TensorFlowBenchmarkArguments</code> are data classes and contain all relevant configurations for their corresponding benchmark class. In the following example, it is shown how a BERT model of type <em>bert-base-cased</em> can be benchmarked.</p> <div class="space-y-10 py-6 2xl:py-8 2xl:-mx-4"> <div class="border border-gray-200 rounded-xl px-4 relative"><div class="flex h-[22px] mt-[-12.5px] justify-between leading-none"><div class="flex px-1 items-center space-x-1 bg-white dark:bg-gray-950"><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><defs><clipPath id="a"><rect x="3.05" y="0.5" width="25.73" height="31" fill="none"></rect></clipPath></defs><g clip-path="url(#a)"><path d="M24.94,9.51a12.81,12.81,0,0,1,0,18.16,12.68,12.68,0,0,1-18,0,12.81,12.81,0,0,1,0-18.16l9-9V5l-.84.83-6,6a9.58,9.58,0,1,0,13.55,0ZM20.44,9a1.68,1.68,0,1,1,1.67-1.67A1.68,1.68,0,0,1,20.44,9Z" fill="#ee4c2c"></path></g></svg> <span>Pytorch</span></div> <div class="cursor-pointer flex items-center justify-center space-x-1 text-sm px-2 bg-white dark:bg-gray-950 hover:underline leading-none"><svg class="" width="0.9em" height="0.9em" viewBox="0 0 10 9" fill="currentColor" xmlns="http://www.w3.org/2000/svg"><path d="M1.39125 1.9725L0.0883333 0.669997L0.677917 0.0804138L8.9275 8.33041L8.33792 8.91958L6.95875 7.54041C6.22592 8.00523 5.37572 8.25138 4.50792 8.25C2.26125 8.25 0.392083 6.63333 0 4.5C0.179179 3.52946 0.667345 2.64287 1.39167 1.9725H1.39125ZM5.65667 6.23833L5.04667 5.62833C4.81335 5.73996 4.55116 5.77647 4.29622 5.73282C4.04129 5.68918 3.80617 5.56752 3.62328 5.38463C3.44039 5.20175 3.31874 4.96663 3.27509 4.71169C3.23144 4.45676 3.26795 4.19456 3.37958 3.96125L2.76958 3.35125C2.50447 3.75187 2.38595 4.2318 2.4341 4.70978C2.48225 5.18777 2.6941 5.63442 3.0338 5.97411C3.37349 6.31381 3.82015 6.52567 4.29813 6.57382C4.77611 6.62197 5.25605 6.50345 5.65667 6.23833ZM2.83042 1.06666C3.35 0.862497 3.91625 0.749997 4.50792 0.749997C6.75458 0.749997 8.62375 2.36666 9.01583 4.5C8.88816 5.19404 8.60119 5.84899 8.1775 6.41333L6.56917 4.805C6.61694 4.48317 6.58868 4.15463 6.48664 3.84569C6.3846 3.53675 6.21162 3.256 5.98156 3.02594C5.7515 2.79588 5.47075 2.6229 5.16181 2.52086C4.85287 2.41882 4.52433 2.39056 4.2025 2.43833L2.83042 1.06708V1.06666Z" fill="currentColor"></path></svg> <span>Hide Pytorch content</span></div></div> <div class="framework-content"> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> PyTorchBenchmark, PyTorchBenchmarkArguments | |
| <span class="hljs-meta">>>> </span>args = PyTorchBenchmarkArguments(models=[<span class="hljs-string">"google-bert/bert-base-uncased"</span>], batch_sizes=[<span class="hljs-number">8</span>], sequence_lengths=[<span class="hljs-number">8</span>, <span class="hljs-number">32</span>, <span class="hljs-number">128</span>, <span class="hljs-number">512</span>]) | |
| <span class="hljs-meta">>>> </span>benchmark = PyTorchBenchmark(args)<!-- HTML_TAG_END --></pre></div></div></div> <div class="border border-gray-200 rounded-xl px-4 relative"><div class="flex h-[22px] mt-[-12.5px] justify-between leading-none"><div class="flex px-1 items-center space-x-1 bg-white dark:bg-gray-950"><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" focusable="false" role="img" width="0.94em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 274"><path d="M145.726 42.065v42.07l72.861 42.07v-42.07l-72.86-42.07zM0 84.135v42.07l36.43 21.03V105.17L0 84.135zm109.291 21.035l-36.43 21.034v126.2l36.43 21.035v-84.135l36.435 21.035v-42.07l-36.435-21.034V105.17z" fill="#E55B2D"></path><path d="M145.726 42.065L36.43 105.17v42.065l72.861-42.065v42.065l36.435-21.03v-84.14zM255.022 63.1l-36.435 21.035v42.07l36.435-21.035V63.1zm-72.865 84.135l-36.43 21.035v42.07l36.43-21.036v-42.07zm-36.43 63.104l-36.436-21.035v84.135l36.435-21.035V210.34z" fill="#ED8E24"></path><path d="M145.726 0L0 84.135l36.43 21.035l109.296-63.105l72.861 42.07L255.022 63.1L145.726 0zm0 126.204l-36.435 21.03l36.435 21.036l36.43-21.035l-36.43-21.03z" fill="#F8BF3C"></path></svg> <span>TensorFlow</span></div> <div class="cursor-pointer flex items-center justify-center space-x-1 text-sm px-2 bg-white dark:bg-gray-950 hover:underline leading-none"><svg class="" width="0.9em" height="0.9em" viewBox="0 0 10 9" fill="currentColor" xmlns="http://www.w3.org/2000/svg"><path d="M1.39125 1.9725L0.0883333 0.669997L0.677917 0.0804138L8.9275 8.33041L8.33792 8.91958L6.95875 7.54041C6.22592 8.00523 5.37572 8.25138 4.50792 8.25C2.26125 8.25 0.392083 6.63333 0 4.5C0.179179 3.52946 0.667345 2.64287 1.39167 1.9725H1.39125ZM5.65667 6.23833L5.04667 5.62833C4.81335 5.73996 4.55116 5.77647 4.29622 5.73282C4.04129 5.68918 3.80617 5.56752 3.62328 5.38463C3.44039 5.20175 3.31874 4.96663 3.27509 4.71169C3.23144 4.45676 3.26795 4.19456 3.37958 3.96125L2.76958 3.35125C2.50447 3.75187 2.38595 4.2318 2.4341 4.70978C2.48225 5.18777 2.6941 5.63442 3.0338 5.97411C3.37349 6.31381 3.82015 6.52567 4.29813 6.57382C4.77611 6.62197 5.25605 6.50345 5.65667 6.23833ZM2.83042 1.06666C3.35 0.862497 3.91625 0.749997 4.50792 0.749997C6.75458 0.749997 8.62375 2.36666 9.01583 4.5C8.88816 5.19404 8.60119 5.84899 8.1775 6.41333L6.56917 4.805C6.61694 4.48317 6.58868 4.15463 6.48664 3.84569C6.3846 3.53675 6.21162 3.256 5.98156 3.02594C5.7515 2.79588 5.47075 2.6229 5.16181 2.52086C4.85287 2.41882 4.52433 2.39056 4.2025 2.43833L2.83042 1.06708V1.06666Z" fill="currentColor"></path></svg> <span>Hide TensorFlow content</span></div></div> <div class="framework-content"> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> TensorFlowBenchmark, TensorFlowBenchmarkArguments | |
| <span class="hljs-meta">>>> </span>args = TensorFlowBenchmarkArguments( | |
| <span class="hljs-meta">... </span> models=[<span class="hljs-string">"google-bert/bert-base-uncased"</span>], batch_sizes=[<span class="hljs-number">8</span>], sequence_lengths=[<span class="hljs-number">8</span>, <span class="hljs-number">32</span>, <span class="hljs-number">128</span>, <span class="hljs-number">512</span>] | |
| <span class="hljs-meta">... </span>) | |
| <span class="hljs-meta">>>> </span>benchmark = TensorFlowBenchmark(args)<!-- HTML_TAG_END --></pre></div> </div></div> </div> <p data-svelte-h="svelte-1eubt8z">Here, three arguments are given to the benchmark argument data classes, namely <code>models</code>, <code>batch_sizes</code>, and | |
| <code>sequence_lengths</code>. The argument <code>models</code> is required and expects a <code>list</code> of model identifiers from the | |
| <a href="https://huggingface.co/models" rel="nofollow">model hub</a> The <code>list</code> arguments <code>batch_sizes</code> and <code>sequence_lengths</code> define | |
| the size of the <code>input_ids</code> on which the model is benchmarked. There are many more parameters that can be configured | |
| via the benchmark argument data classes. For more detail on these one can either directly consult the files | |
| <code>src/transformers/benchmark/benchmark_args_utils.py</code>, <code>src/transformers/benchmark/benchmark_args.py</code> (for PyTorch) | |
| and <code>src/transformers/benchmark/benchmark_args_tf.py</code> (for Tensorflow). Alternatively, running the following shell | |
| commands from root will print out a descriptive list of all configurable parameters for PyTorch and Tensorflow | |
| respectively.</p> <div class="space-y-10 py-6 2xl:py-8 2xl:-mx-4"> <div class="border border-gray-200 rounded-xl px-4 relative"><div class="flex h-[22px] mt-[-12.5px] justify-between leading-none"><div class="flex px-1 items-center space-x-1 bg-white dark:bg-gray-950"><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><defs><clipPath id="a"><rect x="3.05" y="0.5" width="25.73" height="31" fill="none"></rect></clipPath></defs><g clip-path="url(#a)"><path d="M24.94,9.51a12.81,12.81,0,0,1,0,18.16,12.68,12.68,0,0,1-18,0,12.81,12.81,0,0,1,0-18.16l9-9V5l-.84.83-6,6a9.58,9.58,0,1,0,13.55,0ZM20.44,9a1.68,1.68,0,1,1,1.67-1.67A1.68,1.68,0,0,1,20.44,9Z" fill="#ee4c2c"></path></g></svg> <span>Pytorch</span></div> <div class="cursor-pointer flex items-center justify-center space-x-1 text-sm px-2 bg-white dark:bg-gray-950 hover:underline leading-none"><svg class="" width="0.9em" height="0.9em" viewBox="0 0 10 9" fill="currentColor" xmlns="http://www.w3.org/2000/svg"><path d="M1.39125 1.9725L0.0883333 0.669997L0.677917 0.0804138L8.9275 8.33041L8.33792 8.91958L6.95875 7.54041C6.22592 8.00523 5.37572 8.25138 4.50792 8.25C2.26125 8.25 0.392083 6.63333 0 4.5C0.179179 3.52946 0.667345 2.64287 1.39167 1.9725H1.39125ZM5.65667 6.23833L5.04667 5.62833C4.81335 5.73996 4.55116 5.77647 4.29622 5.73282C4.04129 5.68918 3.80617 5.56752 3.62328 5.38463C3.44039 5.20175 3.31874 4.96663 3.27509 4.71169C3.23144 4.45676 3.26795 4.19456 3.37958 3.96125L2.76958 3.35125C2.50447 3.75187 2.38595 4.2318 2.4341 4.70978C2.48225 5.18777 2.6941 5.63442 3.0338 5.97411C3.37349 6.31381 3.82015 6.52567 4.29813 6.57382C4.77611 6.62197 5.25605 6.50345 5.65667 6.23833ZM2.83042 1.06666C3.35 0.862497 3.91625 0.749997 4.50792 0.749997C6.75458 0.749997 8.62375 2.36666 9.01583 4.5C8.88816 5.19404 8.60119 5.84899 8.1775 6.41333L6.56917 4.805C6.61694 4.48317 6.58868 4.15463 6.48664 3.84569C6.3846 3.53675 6.21162 3.256 5.98156 3.02594C5.7515 2.79588 5.47075 2.6229 5.16181 2.52086C4.85287 2.41882 4.52433 2.39056 4.2025 2.43833L2.83042 1.06708V1.06666Z" fill="currentColor"></path></svg> <span>Hide Pytorch content</span></div></div> <div class="framework-content"> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->python examples/pytorch/benchmarking/run_benchmark.py --<span class="hljs-built_in">help</span><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-ersiz3">An instantiated benchmark object can then simply be run by calling <code>benchmark.run()</code>.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span>results = benchmark.run() | |
| <span class="hljs-meta">>>> </span><span class="hljs-built_in">print</span>(results) | |
| ==================== INFERENCE - SPEED - RESULT ==================== | |
| -------------------------------------------------------------------------------- | |
| Model Name Batch Size Seq Length Time <span class="hljs-keyword">in</span> s | |
| -------------------------------------------------------------------------------- | |
| google-bert/bert-base-uncased <span class="hljs-number">8</span> <span class="hljs-number">8</span> <span class="hljs-number">0.006</span> | |
| google-bert/bert-base-uncased <span class="hljs-number">8</span> <span class="hljs-number">32</span> <span class="hljs-number">0.006</span> | |
| google-bert/bert-base-uncased <span class="hljs-number">8</span> <span class="hljs-number">128</span> <span class="hljs-number">0.018</span> | |
| google-bert/bert-base-uncased <span class="hljs-number">8</span> <span class="hljs-number">512</span> <span class="hljs-number">0.088</span> | |
| -------------------------------------------------------------------------------- | |
| ==================== INFERENCE - MEMORY - RESULT ==================== | |
| -------------------------------------------------------------------------------- | |
| Model Name Batch Size Seq Length Memory <span class="hljs-keyword">in</span> MB | |
| -------------------------------------------------------------------------------- | |
| google-bert/bert-base-uncased <span class="hljs-number">8</span> <span class="hljs-number">8</span> <span class="hljs-number">1227</span> | |
| google-bert/bert-base-uncased <span class="hljs-number">8</span> <span class="hljs-number">32</span> <span class="hljs-number">1281</span> | |
| google-bert/bert-base-uncased <span class="hljs-number">8</span> <span class="hljs-number">128</span> <span class="hljs-number">1307</span> | |
| google-bert/bert-base-uncased <span class="hljs-number">8</span> <span class="hljs-number">512</span> <span class="hljs-number">1539</span> | |
| -------------------------------------------------------------------------------- | |
| ==================== ENVIRONMENT INFORMATION ==================== | |
| - transformers_version: <span class="hljs-number">2.11</span><span class="hljs-number">.0</span> | |
| - framework: PyTorch | |
| - use_torchscript: <span class="hljs-literal">False</span> | |
| - framework_version: <span class="hljs-number">1.4</span><span class="hljs-number">.0</span> | |
| - python_version: <span class="hljs-number">3.6</span><span class="hljs-number">.10</span> | |
| - system: Linux | |
| - cpu: x86_64 | |
| - architecture: 64bit | |
| - date: <span class="hljs-number">2020</span>-06-<span class="hljs-number">29</span> | |
| - time: 08:<span class="hljs-number">58</span>:<span class="hljs-number">43.371351</span> | |
| - fp16: <span class="hljs-literal">False</span> | |
| - use_multiprocessing: <span class="hljs-literal">True</span> | |
| - only_pretrain_model: <span class="hljs-literal">False</span> | |
| - cpu_ram_mb: <span class="hljs-number">32088</span> | |
| - use_gpu: <span class="hljs-literal">True</span> | |
| - num_gpus: <span class="hljs-number">1</span> | |
| - gpu: TITAN RTX | |
| - gpu_ram_mb: <span class="hljs-number">24217</span> | |
| - gpu_power_watts: <span class="hljs-number">280.0</span> | |
| - gpu_performance_state: <span class="hljs-number">2</span> | |
| - use_tpu: <span class="hljs-literal">False</span><!-- HTML_TAG_END --></pre></div></div></div> <div class="border border-gray-200 rounded-xl px-4 relative"><div class="flex h-[22px] mt-[-12.5px] justify-between leading-none"><div class="flex px-1 items-center space-x-1 bg-white dark:bg-gray-950"><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" focusable="false" role="img" width="0.94em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 274"><path d="M145.726 42.065v42.07l72.861 42.07v-42.07l-72.86-42.07zM0 84.135v42.07l36.43 21.03V105.17L0 84.135zm109.291 21.035l-36.43 21.034v126.2l36.43 21.035v-84.135l36.435 21.035v-42.07l-36.435-21.034V105.17z" fill="#E55B2D"></path><path d="M145.726 42.065L36.43 105.17v42.065l72.861-42.065v42.065l36.435-21.03v-84.14zM255.022 63.1l-36.435 21.035v42.07l36.435-21.035V63.1zm-72.865 84.135l-36.43 21.035v42.07l36.43-21.036v-42.07zm-36.43 63.104l-36.436-21.035v84.135l36.435-21.035V210.34z" fill="#ED8E24"></path><path d="M145.726 0L0 84.135l36.43 21.035l109.296-63.105l72.861 42.07L255.022 63.1L145.726 0zm0 126.204l-36.435 21.03l36.435 21.036l36.43-21.035l-36.43-21.03z" fill="#F8BF3C"></path></svg> <span>TensorFlow</span></div> <div class="cursor-pointer flex items-center justify-center space-x-1 text-sm px-2 bg-white dark:bg-gray-950 hover:underline leading-none"><svg class="" width="0.9em" height="0.9em" viewBox="0 0 10 9" fill="currentColor" xmlns="http://www.w3.org/2000/svg"><path d="M1.39125 1.9725L0.0883333 0.669997L0.677917 0.0804138L8.9275 8.33041L8.33792 8.91958L6.95875 7.54041C6.22592 8.00523 5.37572 8.25138 4.50792 8.25C2.26125 8.25 0.392083 6.63333 0 4.5C0.179179 3.52946 0.667345 2.64287 1.39167 1.9725H1.39125ZM5.65667 6.23833L5.04667 5.62833C4.81335 5.73996 4.55116 5.77647 4.29622 5.73282C4.04129 5.68918 3.80617 5.56752 3.62328 5.38463C3.44039 5.20175 3.31874 4.96663 3.27509 4.71169C3.23144 4.45676 3.26795 4.19456 3.37958 3.96125L2.76958 3.35125C2.50447 3.75187 2.38595 4.2318 2.4341 4.70978C2.48225 5.18777 2.6941 5.63442 3.0338 5.97411C3.37349 6.31381 3.82015 6.52567 4.29813 6.57382C4.77611 6.62197 5.25605 6.50345 5.65667 6.23833ZM2.83042 1.06666C3.35 0.862497 3.91625 0.749997 4.50792 0.749997C6.75458 0.749997 8.62375 2.36666 9.01583 4.5C8.88816 5.19404 8.60119 5.84899 8.1775 6.41333L6.56917 4.805C6.61694 4.48317 6.58868 4.15463 6.48664 3.84569C6.3846 3.53675 6.21162 3.256 5.98156 3.02594C5.7515 2.79588 5.47075 2.6229 5.16181 2.52086C4.85287 2.41882 4.52433 2.39056 4.2025 2.43833L2.83042 1.06708V1.06666Z" fill="currentColor"></path></svg> <span>Hide TensorFlow content</span></div></div> <div class="framework-content"> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->python examples/tensorflow/benchmarking/run_benchmark_tf.py --<span class="hljs-built_in">help</span><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-ersiz3">An instantiated benchmark object can then simply be run by calling <code>benchmark.run()</code>.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span>results = benchmark.run() | |
| <span class="hljs-meta">>>> </span><span class="hljs-built_in">print</span>(results) | |
| <span class="hljs-meta">>>> </span>results = benchmark.run() | |
| <span class="hljs-meta">>>> </span><span class="hljs-built_in">print</span>(results) | |
| ==================== INFERENCE - SPEED - RESULT ==================== | |
| -------------------------------------------------------------------------------- | |
| Model Name Batch Size Seq Length Time <span class="hljs-keyword">in</span> s | |
| -------------------------------------------------------------------------------- | |
| google-bert/bert-base-uncased <span class="hljs-number">8</span> <span class="hljs-number">8</span> <span class="hljs-number">0.005</span> | |
| google-bert/bert-base-uncased <span class="hljs-number">8</span> <span class="hljs-number">32</span> <span class="hljs-number">0.008</span> | |
| google-bert/bert-base-uncased <span class="hljs-number">8</span> <span class="hljs-number">128</span> <span class="hljs-number">0.022</span> | |
| google-bert/bert-base-uncased <span class="hljs-number">8</span> <span class="hljs-number">512</span> <span class="hljs-number">0.105</span> | |
| -------------------------------------------------------------------------------- | |
| ==================== INFERENCE - MEMORY - RESULT ==================== | |
| -------------------------------------------------------------------------------- | |
| Model Name Batch Size Seq Length Memory <span class="hljs-keyword">in</span> MB | |
| -------------------------------------------------------------------------------- | |
| google-bert/bert-base-uncased <span class="hljs-number">8</span> <span class="hljs-number">8</span> <span class="hljs-number">1330</span> | |
| google-bert/bert-base-uncased <span class="hljs-number">8</span> <span class="hljs-number">32</span> <span class="hljs-number">1330</span> | |
| google-bert/bert-base-uncased <span class="hljs-number">8</span> <span class="hljs-number">128</span> <span class="hljs-number">1330</span> | |
| google-bert/bert-base-uncased <span class="hljs-number">8</span> <span class="hljs-number">512</span> <span class="hljs-number">1770</span> | |
| -------------------------------------------------------------------------------- | |
| ==================== ENVIRONMENT INFORMATION ==================== | |
| - transformers_version: <span class="hljs-number">2.11</span><span class="hljs-number">.0</span> | |
| - framework: Tensorflow | |
| - use_xla: <span class="hljs-literal">False</span> | |
| - framework_version: <span class="hljs-number">2.2</span><span class="hljs-number">.0</span> | |
| - python_version: <span class="hljs-number">3.6</span><span class="hljs-number">.10</span> | |
| - system: Linux | |
| - cpu: x86_64 | |
| - architecture: 64bit | |
| - date: <span class="hljs-number">2020</span>-06-<span class="hljs-number">29</span> | |
| - time: 09:<span class="hljs-number">26</span>:<span class="hljs-number">35.617317</span> | |
| - fp16: <span class="hljs-literal">False</span> | |
| - use_multiprocessing: <span class="hljs-literal">True</span> | |
| - only_pretrain_model: <span class="hljs-literal">False</span> | |
| - cpu_ram_mb: <span class="hljs-number">32088</span> | |
| - use_gpu: <span class="hljs-literal">True</span> | |
| - num_gpus: <span class="hljs-number">1</span> | |
| - gpu: TITAN RTX | |
| - gpu_ram_mb: <span class="hljs-number">24217</span> | |
| - gpu_power_watts: <span class="hljs-number">280.0</span> | |
| - gpu_performance_state: <span class="hljs-number">2</span> | |
| - use_tpu: <span class="hljs-literal">False</span><!-- HTML_TAG_END --></pre></div> </div></div> </div> <p data-svelte-h="svelte-44z1ex">By default, the <em>time</em> and the <em>required memory</em> for <em>inference</em> are benchmarked. In the example output above the first | |
| two sections show the result corresponding to <em>inference time</em> and <em>inference memory</em>. In addition, all relevant | |
| information about the computing environment, <em>e.g.</em> the GPU type, the system, the library versions, etc… are printed | |
| out in the third section under <em>ENVIRONMENT INFORMATION</em>. This information can optionally be saved in a <em>.csv</em> file | |
| when adding the argument <code>save_to_csv=True</code> to <code>PyTorchBenchmarkArguments</code> and | |
| <code>TensorFlowBenchmarkArguments</code> respectively. In this case, every section is saved in a separate | |
| <em>.csv</em> file. The path to each <em>.csv</em> file can optionally be defined via the argument data classes.</p> <p data-svelte-h="svelte-1r6j6dx">Instead of benchmarking pre-trained models via their model identifier, <em>e.g.</em> <code>google-bert/bert-base-uncased</code>, the user can | |
| alternatively benchmark an arbitrary configuration of any available model class. In this case, a <code>list</code> of | |
| configurations must be inserted with the benchmark args as follows.</p> <div class="space-y-10 py-6 2xl:py-8 2xl:-mx-4"> <div class="border border-gray-200 rounded-xl px-4 relative"><div class="flex h-[22px] mt-[-12.5px] justify-between leading-none"><div class="flex px-1 items-center space-x-1 bg-white dark:bg-gray-950"><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><defs><clipPath id="a"><rect x="3.05" y="0.5" width="25.73" height="31" fill="none"></rect></clipPath></defs><g clip-path="url(#a)"><path d="M24.94,9.51a12.81,12.81,0,0,1,0,18.16,12.68,12.68,0,0,1-18,0,12.81,12.81,0,0,1,0-18.16l9-9V5l-.84.83-6,6a9.58,9.58,0,1,0,13.55,0ZM20.44,9a1.68,1.68,0,1,1,1.67-1.67A1.68,1.68,0,0,1,20.44,9Z" fill="#ee4c2c"></path></g></svg> <span>Pytorch</span></div> <div class="cursor-pointer flex items-center justify-center space-x-1 text-sm px-2 bg-white dark:bg-gray-950 hover:underline leading-none"><svg class="" width="0.9em" height="0.9em" viewBox="0 0 10 9" fill="currentColor" xmlns="http://www.w3.org/2000/svg"><path d="M1.39125 1.9725L0.0883333 0.669997L0.677917 0.0804138L8.9275 8.33041L8.33792 8.91958L6.95875 7.54041C6.22592 8.00523 5.37572 8.25138 4.50792 8.25C2.26125 8.25 0.392083 6.63333 0 4.5C0.179179 3.52946 0.667345 2.64287 1.39167 1.9725H1.39125ZM5.65667 6.23833L5.04667 5.62833C4.81335 5.73996 4.55116 5.77647 4.29622 5.73282C4.04129 5.68918 3.80617 5.56752 3.62328 5.38463C3.44039 5.20175 3.31874 4.96663 3.27509 4.71169C3.23144 4.45676 3.26795 4.19456 3.37958 3.96125L2.76958 3.35125C2.50447 3.75187 2.38595 4.2318 2.4341 4.70978C2.48225 5.18777 2.6941 5.63442 3.0338 5.97411C3.37349 6.31381 3.82015 6.52567 4.29813 6.57382C4.77611 6.62197 5.25605 6.50345 5.65667 6.23833ZM2.83042 1.06666C3.35 0.862497 3.91625 0.749997 4.50792 0.749997C6.75458 0.749997 8.62375 2.36666 9.01583 4.5C8.88816 5.19404 8.60119 5.84899 8.1775 6.41333L6.56917 4.805C6.61694 4.48317 6.58868 4.15463 6.48664 3.84569C6.3846 3.53675 6.21162 3.256 5.98156 3.02594C5.7515 2.79588 5.47075 2.6229 5.16181 2.52086C4.85287 2.41882 4.52433 2.39056 4.2025 2.43833L2.83042 1.06708V1.06666Z" fill="currentColor"></path></svg> <span>Hide Pytorch content</span></div></div> <div class="framework-content"> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> PyTorchBenchmark, PyTorchBenchmarkArguments, BertConfig | |
| <span class="hljs-meta">>>> </span>args = PyTorchBenchmarkArguments( | |
| <span class="hljs-meta">... </span> models=[<span class="hljs-string">"bert-base"</span>, <span class="hljs-string">"bert-384-hid"</span>, <span class="hljs-string">"bert-6-lay"</span>], batch_sizes=[<span class="hljs-number">8</span>], sequence_lengths=[<span class="hljs-number">8</span>, <span class="hljs-number">32</span>, <span class="hljs-number">128</span>, <span class="hljs-number">512</span>] | |
| <span class="hljs-meta">... </span>) | |
| <span class="hljs-meta">>>> </span>config_base = BertConfig() | |
| <span class="hljs-meta">>>> </span>config_384_hid = BertConfig(hidden_size=<span class="hljs-number">384</span>) | |
| <span class="hljs-meta">>>> </span>config_6_lay = BertConfig(num_hidden_layers=<span class="hljs-number">6</span>) | |
| <span class="hljs-meta">>>> </span>benchmark = PyTorchBenchmark(args, configs=[config_base, config_384_hid, config_6_lay]) | |
| <span class="hljs-meta">>>> </span>benchmark.run() | |
| ==================== INFERENCE - SPEED - RESULT ==================== | |
| -------------------------------------------------------------------------------- | |
| Model Name Batch Size Seq Length Time <span class="hljs-keyword">in</span> s | |
| -------------------------------------------------------------------------------- | |
| bert-base <span class="hljs-number">8</span> <span class="hljs-number">128</span> <span class="hljs-number">0.006</span> | |
| bert-base <span class="hljs-number">8</span> <span class="hljs-number">512</span> <span class="hljs-number">0.006</span> | |
| bert-base <span class="hljs-number">8</span> <span class="hljs-number">128</span> <span class="hljs-number">0.018</span> | |
| bert-base <span class="hljs-number">8</span> <span class="hljs-number">512</span> <span class="hljs-number">0.088</span> | |
| bert-<span class="hljs-number">384</span>-hid <span class="hljs-number">8</span> <span class="hljs-number">8</span> <span class="hljs-number">0.006</span> | |
| bert-<span class="hljs-number">384</span>-hid <span class="hljs-number">8</span> <span class="hljs-number">32</span> <span class="hljs-number">0.006</span> | |
| bert-<span class="hljs-number">384</span>-hid <span class="hljs-number">8</span> <span class="hljs-number">128</span> <span class="hljs-number">0.011</span> | |
| bert-<span class="hljs-number">384</span>-hid <span class="hljs-number">8</span> <span class="hljs-number">512</span> <span class="hljs-number">0.054</span> | |
| bert-<span class="hljs-number">6</span>-lay <span class="hljs-number">8</span> <span class="hljs-number">8</span> <span class="hljs-number">0.003</span> | |
| bert-<span class="hljs-number">6</span>-lay <span class="hljs-number">8</span> <span class="hljs-number">32</span> <span class="hljs-number">0.004</span> | |
| bert-<span class="hljs-number">6</span>-lay <span class="hljs-number">8</span> <span class="hljs-number">128</span> <span class="hljs-number">0.009</span> | |
| bert-<span class="hljs-number">6</span>-lay <span class="hljs-number">8</span> <span class="hljs-number">512</span> <span class="hljs-number">0.044</span> | |
| -------------------------------------------------------------------------------- | |
| ==================== INFERENCE - MEMORY - RESULT ==================== | |
| -------------------------------------------------------------------------------- | |
| Model Name Batch Size Seq Length Memory <span class="hljs-keyword">in</span> MB | |
| -------------------------------------------------------------------------------- | |
| bert-base <span class="hljs-number">8</span> <span class="hljs-number">8</span> <span class="hljs-number">1277</span> | |
| bert-base <span class="hljs-number">8</span> <span class="hljs-number">32</span> <span class="hljs-number">1281</span> | |
| bert-base <span class="hljs-number">8</span> <span class="hljs-number">128</span> <span class="hljs-number">1307</span> | |
| bert-base <span class="hljs-number">8</span> <span class="hljs-number">512</span> <span class="hljs-number">1539</span> | |
| bert-<span class="hljs-number">384</span>-hid <span class="hljs-number">8</span> <span class="hljs-number">8</span> <span class="hljs-number">1005</span> | |
| bert-<span class="hljs-number">384</span>-hid <span class="hljs-number">8</span> <span class="hljs-number">32</span> <span class="hljs-number">1027</span> | |
| bert-<span class="hljs-number">384</span>-hid <span class="hljs-number">8</span> <span class="hljs-number">128</span> <span class="hljs-number">1035</span> | |
| bert-<span class="hljs-number">384</span>-hid <span class="hljs-number">8</span> <span class="hljs-number">512</span> <span class="hljs-number">1255</span> | |
| bert-<span class="hljs-number">6</span>-lay <span class="hljs-number">8</span> <span class="hljs-number">8</span> <span class="hljs-number">1097</span> | |
| bert-<span class="hljs-number">6</span>-lay <span class="hljs-number">8</span> <span class="hljs-number">32</span> <span class="hljs-number">1101</span> | |
| bert-<span class="hljs-number">6</span>-lay <span class="hljs-number">8</span> <span class="hljs-number">128</span> <span class="hljs-number">1127</span> | |
| bert-<span class="hljs-number">6</span>-lay <span class="hljs-number">8</span> <span class="hljs-number">512</span> <span class="hljs-number">1359</span> | |
| -------------------------------------------------------------------------------- | |
| ==================== ENVIRONMENT INFORMATION ==================== | |
| - transformers_version: <span class="hljs-number">2.11</span><span class="hljs-number">.0</span> | |
| - framework: PyTorch | |
| - use_torchscript: <span class="hljs-literal">False</span> | |
| - framework_version: <span class="hljs-number">1.4</span><span class="hljs-number">.0</span> | |
| - python_version: <span class="hljs-number">3.6</span><span class="hljs-number">.10</span> | |
| - system: Linux | |
| - cpu: x86_64 | |
| - architecture: 64bit | |
| - date: <span class="hljs-number">2020</span>-06-<span class="hljs-number">29</span> | |
| - time: 09:<span class="hljs-number">35</span>:<span class="hljs-number">25.143267</span> | |
| - fp16: <span class="hljs-literal">False</span> | |
| - use_multiprocessing: <span class="hljs-literal">True</span> | |
| - only_pretrain_model: <span class="hljs-literal">False</span> | |
| - cpu_ram_mb: <span class="hljs-number">32088</span> | |
| - use_gpu: <span class="hljs-literal">True</span> | |
| - num_gpus: <span class="hljs-number">1</span> | |
| - gpu: TITAN RTX | |
| - gpu_ram_mb: <span class="hljs-number">24217</span> | |
| - gpu_power_watts: <span class="hljs-number">280.0</span> | |
| - gpu_performance_state: <span class="hljs-number">2</span> | |
| - use_tpu: <span class="hljs-literal">False</span><!-- HTML_TAG_END --></pre></div></div></div> <div class="border border-gray-200 rounded-xl px-4 relative"><div class="flex h-[22px] mt-[-12.5px] justify-between leading-none"><div class="flex px-1 items-center space-x-1 bg-white dark:bg-gray-950"><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" focusable="false" role="img" width="0.94em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 274"><path d="M145.726 42.065v42.07l72.861 42.07v-42.07l-72.86-42.07zM0 84.135v42.07l36.43 21.03V105.17L0 84.135zm109.291 21.035l-36.43 21.034v126.2l36.43 21.035v-84.135l36.435 21.035v-42.07l-36.435-21.034V105.17z" fill="#E55B2D"></path><path d="M145.726 42.065L36.43 105.17v42.065l72.861-42.065v42.065l36.435-21.03v-84.14zM255.022 63.1l-36.435 21.035v42.07l36.435-21.035V63.1zm-72.865 84.135l-36.43 21.035v42.07l36.43-21.036v-42.07zm-36.43 63.104l-36.436-21.035v84.135l36.435-21.035V210.34z" fill="#ED8E24"></path><path d="M145.726 0L0 84.135l36.43 21.035l109.296-63.105l72.861 42.07L255.022 63.1L145.726 0zm0 126.204l-36.435 21.03l36.435 21.036l36.43-21.035l-36.43-21.03z" fill="#F8BF3C"></path></svg> <span>TensorFlow</span></div> <div class="cursor-pointer flex items-center justify-center space-x-1 text-sm px-2 bg-white dark:bg-gray-950 hover:underline leading-none"><svg class="" width="0.9em" height="0.9em" viewBox="0 0 10 9" fill="currentColor" xmlns="http://www.w3.org/2000/svg"><path d="M1.39125 1.9725L0.0883333 0.669997L0.677917 0.0804138L8.9275 8.33041L8.33792 8.91958L6.95875 7.54041C6.22592 8.00523 5.37572 8.25138 4.50792 8.25C2.26125 8.25 0.392083 6.63333 0 4.5C0.179179 3.52946 0.667345 2.64287 1.39167 1.9725H1.39125ZM5.65667 6.23833L5.04667 5.62833C4.81335 5.73996 4.55116 5.77647 4.29622 5.73282C4.04129 5.68918 3.80617 5.56752 3.62328 5.38463C3.44039 5.20175 3.31874 4.96663 3.27509 4.71169C3.23144 4.45676 3.26795 4.19456 3.37958 3.96125L2.76958 3.35125C2.50447 3.75187 2.38595 4.2318 2.4341 4.70978C2.48225 5.18777 2.6941 5.63442 3.0338 5.97411C3.37349 6.31381 3.82015 6.52567 4.29813 6.57382C4.77611 6.62197 5.25605 6.50345 5.65667 6.23833ZM2.83042 1.06666C3.35 0.862497 3.91625 0.749997 4.50792 0.749997C6.75458 0.749997 8.62375 2.36666 9.01583 4.5C8.88816 5.19404 8.60119 5.84899 8.1775 6.41333L6.56917 4.805C6.61694 4.48317 6.58868 4.15463 6.48664 3.84569C6.3846 3.53675 6.21162 3.256 5.98156 3.02594C5.7515 2.79588 5.47075 2.6229 5.16181 2.52086C4.85287 2.41882 4.52433 2.39056 4.2025 2.43833L2.83042 1.06708V1.06666Z" fill="currentColor"></path></svg> <span>Hide TensorFlow content</span></div></div> <div class="framework-content"> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> TensorFlowBenchmark, TensorFlowBenchmarkArguments, BertConfig | |
| <span class="hljs-meta">>>> </span>args = TensorFlowBenchmarkArguments( | |
| <span class="hljs-meta">... </span> models=[<span class="hljs-string">"bert-base"</span>, <span class="hljs-string">"bert-384-hid"</span>, <span class="hljs-string">"bert-6-lay"</span>], batch_sizes=[<span class="hljs-number">8</span>], sequence_lengths=[<span class="hljs-number">8</span>, <span class="hljs-number">32</span>, <span class="hljs-number">128</span>, <span class="hljs-number">512</span>] | |
| <span class="hljs-meta">... </span>) | |
| <span class="hljs-meta">>>> </span>config_base = BertConfig() | |
| <span class="hljs-meta">>>> </span>config_384_hid = BertConfig(hidden_size=<span class="hljs-number">384</span>) | |
| <span class="hljs-meta">>>> </span>config_6_lay = BertConfig(num_hidden_layers=<span class="hljs-number">6</span>) | |
| <span class="hljs-meta">>>> </span>benchmark = TensorFlowBenchmark(args, configs=[config_base, config_384_hid, config_6_lay]) | |
| <span class="hljs-meta">>>> </span>benchmark.run() | |
| ==================== INFERENCE - SPEED - RESULT ==================== | |
| -------------------------------------------------------------------------------- | |
| Model Name Batch Size Seq Length Time <span class="hljs-keyword">in</span> s | |
| -------------------------------------------------------------------------------- | |
| bert-base <span class="hljs-number">8</span> <span class="hljs-number">8</span> <span class="hljs-number">0.005</span> | |
| bert-base <span class="hljs-number">8</span> <span class="hljs-number">32</span> <span class="hljs-number">0.008</span> | |
| bert-base <span class="hljs-number">8</span> <span class="hljs-number">128</span> <span class="hljs-number">0.022</span> | |
| bert-base <span class="hljs-number">8</span> <span class="hljs-number">512</span> <span class="hljs-number">0.106</span> | |
| bert-<span class="hljs-number">384</span>-hid <span class="hljs-number">8</span> <span class="hljs-number">8</span> <span class="hljs-number">0.005</span> | |
| bert-<span class="hljs-number">384</span>-hid <span class="hljs-number">8</span> <span class="hljs-number">32</span> <span class="hljs-number">0.007</span> | |
| bert-<span class="hljs-number">384</span>-hid <span class="hljs-number">8</span> <span class="hljs-number">128</span> <span class="hljs-number">0.018</span> | |
| bert-<span class="hljs-number">384</span>-hid <span class="hljs-number">8</span> <span class="hljs-number">512</span> <span class="hljs-number">0.064</span> | |
| bert-<span class="hljs-number">6</span>-lay <span class="hljs-number">8</span> <span class="hljs-number">8</span> <span class="hljs-number">0.002</span> | |
| bert-<span class="hljs-number">6</span>-lay <span class="hljs-number">8</span> <span class="hljs-number">32</span> <span class="hljs-number">0.003</span> | |
| bert-<span class="hljs-number">6</span>-lay <span class="hljs-number">8</span> <span class="hljs-number">128</span> <span class="hljs-number">0.0011</span> | |
| bert-<span class="hljs-number">6</span>-lay <span class="hljs-number">8</span> <span class="hljs-number">512</span> <span class="hljs-number">0.074</span> | |
| -------------------------------------------------------------------------------- | |
| ==================== INFERENCE - MEMORY - RESULT ==================== | |
| -------------------------------------------------------------------------------- | |
| Model Name Batch Size Seq Length Memory <span class="hljs-keyword">in</span> MB | |
| -------------------------------------------------------------------------------- | |
| bert-base <span class="hljs-number">8</span> <span class="hljs-number">8</span> <span class="hljs-number">1330</span> | |
| bert-base <span class="hljs-number">8</span> <span class="hljs-number">32</span> <span class="hljs-number">1330</span> | |
| bert-base <span class="hljs-number">8</span> <span class="hljs-number">128</span> <span class="hljs-number">1330</span> | |
| bert-base <span class="hljs-number">8</span> <span class="hljs-number">512</span> <span class="hljs-number">1770</span> | |
| bert-<span class="hljs-number">384</span>-hid <span class="hljs-number">8</span> <span class="hljs-number">8</span> <span class="hljs-number">1330</span> | |
| bert-<span class="hljs-number">384</span>-hid <span class="hljs-number">8</span> <span class="hljs-number">32</span> <span class="hljs-number">1330</span> | |
| bert-<span class="hljs-number">384</span>-hid <span class="hljs-number">8</span> <span class="hljs-number">128</span> <span class="hljs-number">1330</span> | |
| bert-<span class="hljs-number">384</span>-hid <span class="hljs-number">8</span> <span class="hljs-number">512</span> <span class="hljs-number">1540</span> | |
| bert-<span class="hljs-number">6</span>-lay <span class="hljs-number">8</span> <span class="hljs-number">8</span> <span class="hljs-number">1330</span> | |
| bert-<span class="hljs-number">6</span>-lay <span class="hljs-number">8</span> <span class="hljs-number">32</span> <span class="hljs-number">1330</span> | |
| bert-<span class="hljs-number">6</span>-lay <span class="hljs-number">8</span> <span class="hljs-number">128</span> <span class="hljs-number">1330</span> | |
| bert-<span class="hljs-number">6</span>-lay <span class="hljs-number">8</span> <span class="hljs-number">512</span> <span class="hljs-number">1540</span> | |
| -------------------------------------------------------------------------------- | |
| ==================== ENVIRONMENT INFORMATION ==================== | |
| - transformers_version: <span class="hljs-number">2.11</span><span class="hljs-number">.0</span> | |
| - framework: Tensorflow | |
| - use_xla: <span class="hljs-literal">False</span> | |
| - framework_version: <span class="hljs-number">2.2</span><span class="hljs-number">.0</span> | |
| - python_version: <span class="hljs-number">3.6</span><span class="hljs-number">.10</span> | |
| - system: Linux | |
| - cpu: x86_64 | |
| - architecture: 64bit | |
| - date: <span class="hljs-number">2020</span>-06-<span class="hljs-number">29</span> | |
| - time: 09:<span class="hljs-number">38</span>:<span class="hljs-number">15.487125</span> | |
| - fp16: <span class="hljs-literal">False</span> | |
| - use_multiprocessing: <span class="hljs-literal">True</span> | |
| - only_pretrain_model: <span class="hljs-literal">False</span> | |
| - cpu_ram_mb: <span class="hljs-number">32088</span> | |
| - use_gpu: <span class="hljs-literal">True</span> | |
| - num_gpus: <span class="hljs-number">1</span> | |
| - gpu: TITAN RTX | |
| - gpu_ram_mb: <span class="hljs-number">24217</span> | |
| - gpu_power_watts: <span class="hljs-number">280.0</span> | |
| - gpu_performance_state: <span class="hljs-number">2</span> | |
| - use_tpu: <span class="hljs-literal">False</span><!-- HTML_TAG_END --></pre></div> </div></div> </div> <p data-svelte-h="svelte-ztfcrc">Again, <em>inference time</em> and <em>required memory</em> for <em>inference</em> are measured, but this time for customized configurations | |
| of the <code>BertModel</code> class. This feature can especially be helpful when deciding for which configuration the model | |
| should be trained.</p> <h2 class="relative group"><a id="benchmark-best-practices" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#benchmark-best-practices"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Benchmark best practices</span></h2> <p data-svelte-h="svelte-c3uya">This section lists a couple of best practices one should be aware of when benchmarking a model.</p> <ul data-svelte-h="svelte-zn5q06"><li>Currently, only single device benchmarking is supported. When benchmarking on GPU, it is recommended that the user | |
| specifies on which device the code should be run by setting the <code>CUDA_VISIBLE_DEVICES</code> environment variable in the | |
| shell, <em>e.g.</em> <code>export CUDA_VISIBLE_DEVICES=0</code> before running the code.</li> <li>The option <code>no_multi_processing</code> should only be set to <code>True</code> for testing and debugging. To ensure accurate | |
| memory measurement it is recommended to run each memory benchmark in a separate process by making sure | |
| <code>no_multi_processing</code> is set to <code>True</code>.</li> <li>One should always state the environment information when sharing the results of a model benchmark. Results can vary | |
| heavily between different GPU devices, library versions, etc., as a consequence, benchmark results on their own are not very | |
| useful for the community.</li></ul> <h2 class="relative group"><a id="sharing-your-benchmark" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#sharing-your-benchmark"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Sharing your benchmark</span></h2> <p data-svelte-h="svelte-wholqk">Previously all available core models (10 at the time) have been benchmarked for <em>inference time</em>, across many different | |
| settings: using PyTorch, with and without TorchScript, using TensorFlow, with and without XLA. All of those tests were | |
| done across CPUs (except for TensorFlow XLA) and GPUs.</p> <p data-svelte-h="svelte-mk4cui">The approach is detailed in the <a href="https://medium.com/huggingface/benchmarking-transformers-pytorch-and-tensorflow-e2917fb891c2" rel="nofollow">following blogpost</a> and the results are | |
| available <a href="https://docs.google.com/spreadsheets/d/1sryqufw2D0XlUH4sq3e9Wnxu5EAQkaohzrJbd5HdQ_w/edit?usp=sharing" rel="nofollow">here</a>.</p> <p data-svelte-h="svelte-15vqju9">With the new <em>benchmark</em> tools, it is easier than ever to share your benchmark results with the community</p> <ul data-svelte-h="svelte-1q9j466"><li><a href="https://github.com/huggingface/transformers/tree/main/examples/pytorch/benchmarking/README.md" rel="nofollow">PyTorch Benchmarking Results</a>.</li> <li><a href="https://github.com/huggingface/transformers/tree/main/examples/tensorflow/benchmarking/README.md" rel="nofollow">TensorFlow Benchmarking Results</a>.</li></ul> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/transformers/blob/main/docs/source/en/benchmarks.md" target="_blank"><span data-svelte-h="svelte-1kd6by1"><</span> <span data-svelte-h="svelte-x0xyl0">></span> <span data-svelte-h="svelte-1dajgef"><span class="underline ml-1.5">Update</span> on GitHub</span></a> <p></p> | |
| <script> | |
| { | |
| __sveltekit_z647wz = { | |
| assets: "/docs/transformers/pr_33913/en", | |
| base: "/docs/transformers/pr_33913/en", | |
| env: {} | |
| }; | |
| const element = document.currentScript.parentElement; | |
| const data = [null,null]; | |
| Promise.all([ | |
| import("/docs/transformers/pr_33913/en/_app/immutable/entry/start.b67f883f.js"), | |
| import("/docs/transformers/pr_33913/en/_app/immutable/entry/app.e436b1f2.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 9], | |
| data, | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 65.9 kB
- Xet hash:
- 0da977573b82f2803963fe1f1b0e308c9da80afd8988a6379fad4654a5816b26
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.