Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Use with PyArrow","local":"use-with-pyarrow","sections":[{"title":"Dataset format","local":"dataset-format","sections":[],"depth":2},{"title":"Process data","local":"process-data","sections":[],"depth":2},{"title":"Import or Export from PyArrow","local":"import-or-export-from-pyarrow","sections":[],"depth":2}],"depth":1}"> | |
| <link href="/docs/datasets/pr_7489/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/entry/start.e78fe3cb.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/chunks/scheduler.bdbef820.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/chunks/singletons.de3d48f2.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/chunks/index.8a885b74.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/chunks/paths.8ce597b3.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/entry/app.ec9204ff.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/chunks/index.c0aea24a.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/nodes/0.06409290.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/chunks/each.e59479a4.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/nodes/54.56065af6.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/chunks/CodeBlock.e814ab8d.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/chunks/index.6bcf9ddd.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{"title":"Use with PyArrow","local":"use-with-pyarrow","sections":[{"title":"Dataset format","local":"dataset-format","sections":[],"depth":2},{"title":"Process data","local":"process-data","sections":[],"depth":2},{"title":"Import or Export from PyArrow","local":"import-or-export-from-pyarrow","sections":[],"depth":2}],"depth":1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <h1 class="relative group"><a id="use-with-pyarrow" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#use-with-pyarrow"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Use with PyArrow</span></h1> <p data-svelte-h="svelte-o6efn3">This document is a quick introduction to using <code>datasets</code> with PyArrow, with a particular focus on how to process | |
| datasets using Arrow compute functions, and how to convert a dataset to PyArrow or from PyArrow.</p> <p data-svelte-h="svelte-syebji">This is particularly useful as it allows fast zero-copy operations, since <code>datasets</code> uses PyArrow under the hood.</p> <h2 class="relative group"><a id="dataset-format" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#dataset-format"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Dataset format</span></h2> <p data-svelte-h="svelte-ej8pz8">By default, datasets return regular Python objects: integers, floats, strings, lists, etc.</p> <p data-svelte-h="svelte-wng2v6">To get PyArrow Tables or Arrays instead, you can set the format of the dataset to <code>pyarrow</code> using <a href="/docs/datasets/pr_7489/en/package_reference/main_classes#datasets.Dataset.with_format">Dataset.with_format()</a>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> datasets <span class="hljs-keyword">import</span> Dataset | |
| <span class="hljs-meta">>>> </span>data = {<span class="hljs-string">"col_0"</span>: [<span class="hljs-string">"a"</span>, <span class="hljs-string">"b"</span>, <span class="hljs-string">"c"</span>, <span class="hljs-string">"d"</span>], <span class="hljs-string">"col_1"</span>: [<span class="hljs-number">0.</span>, <span class="hljs-number">0.</span>, <span class="hljs-number">1.</span>, <span class="hljs-number">1.</span>]} | |
| <span class="hljs-meta">>>> </span>ds = Dataset.from_dict(data) | |
| <span class="hljs-meta">>>> </span>ds = ds.with_format(<span class="hljs-string">"arrow"</span>) | |
| <span class="hljs-meta">>>> </span>ds[<span class="hljs-number">0</span>] <span class="hljs-comment"># pa.Table</span> | |
| pyarrow.Table | |
| col_0: string | |
| col_1: double | |
| ---- | |
| col_0: [[<span class="hljs-string">"a"</span>]] | |
| col_1: [[<span class="hljs-number">0</span>]] | |
| <span class="hljs-meta">>>> </span>ds[:<span class="hljs-number">2</span>] <span class="hljs-comment"># pa.Table</span> | |
| pyarrow.Table | |
| col_0: string | |
| col_1: double | |
| ---- | |
| col_0: [[<span class="hljs-string">"a"</span>,<span class="hljs-string">"b"</span>]] | |
| col_1: [[<span class="hljs-number">0</span>,<span class="hljs-number">0</span>]] | |
| <span class="hljs-meta">>>> </span>ds[<span class="hljs-string">"data"</span>] <span class="hljs-comment"># pa.array</span> | |
| <pyarrow.lib.ChunkedArray <span class="hljs-built_in">object</span> at <span class="hljs-number">0x1394312a0</span>> | |
| [ | |
| [ | |
| <span class="hljs-string">"a"</span>, | |
| <span class="hljs-string">"b"</span>, | |
| <span class="hljs-string">"c"</span>, | |
| <span class="hljs-string">"d"</span> | |
| ] | |
| ]<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-2qljfh">This also works for <code>IterableDataset</code> objects obtained e.g. using <code>load_dataset(..., streaming=True)</code>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span>ds = ds.with_format(<span class="hljs-string">"arrow"</span>) | |
| <span class="hljs-meta">>>> </span><span class="hljs-keyword">for</span> table <span class="hljs-keyword">in</span> ds.<span class="hljs-built_in">iter</span>(batch_size=<span class="hljs-number">2</span>): | |
| <span class="hljs-meta">... </span> <span class="hljs-built_in">print</span>(table) | |
| <span class="hljs-meta">... </span> <span class="hljs-keyword">break</span> | |
| pyarrow.Table | |
| col_0: string | |
| col_1: double | |
| ---- | |
| col_0: [[<span class="hljs-string">"a"</span>,<span class="hljs-string">"b"</span>]] | |
| col_1: [[<span class="hljs-number">0</span>,<span class="hljs-number">0</span>]]<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="process-data" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#process-data"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Process data</span></h2> <p data-svelte-h="svelte-1q0ocif">PyArrow functions are generally faster than regular hand-written python functions, and therefore they are a good option to optimize data processing. You can use Arrow compute functions to process a dataset in <a href="/docs/datasets/pr_7489/en/package_reference/main_classes#datasets.Dataset.map">Dataset.map()</a> or <a href="/docs/datasets/pr_7489/en/package_reference/main_classes#datasets.Dataset.filter">Dataset.filter()</a>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">import</span> pyarrow.compute <span class="hljs-keyword">as</span> pc | |
| <span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> datasets <span class="hljs-keyword">import</span> Dataset | |
| <span class="hljs-meta">>>> </span>data = {<span class="hljs-string">"col_0"</span>: [<span class="hljs-string">"a"</span>, <span class="hljs-string">"b"</span>, <span class="hljs-string">"c"</span>, <span class="hljs-string">"d"</span>], <span class="hljs-string">"col_1"</span>: [<span class="hljs-number">0.</span>, <span class="hljs-number">0.</span>, <span class="hljs-number">1.</span>, <span class="hljs-number">1.</span>]} | |
| <span class="hljs-meta">>>> </span>ds = Dataset.from_dict(data) | |
| <span class="hljs-meta">>>> </span>ds = ds.with_format(<span class="hljs-string">"arrow"</span>) | |
| <span class="hljs-meta">>>> </span>ds = ds.<span class="hljs-built_in">map</span>(<span class="hljs-keyword">lambda</span> t: t.append_column(<span class="hljs-string">"col_2"</span>, pc.add(t[<span class="hljs-string">"col_1"</span>], <span class="hljs-number">1</span>)), batched=<span class="hljs-literal">True</span>) | |
| <span class="hljs-meta">>>> </span>ds[:<span class="hljs-number">2</span>] | |
| pyarrow.Table | |
| col_0: string | |
| col_1: double | |
| col_2: double | |
| ---- | |
| col_0: [[<span class="hljs-string">"a"</span>,<span class="hljs-string">"b"</span>]] | |
| col_1: [[<span class="hljs-number">0</span>,<span class="hljs-number">0</span>]] | |
| col_2: [[<span class="hljs-number">1</span>,<span class="hljs-number">1</span>]] | |
| <span class="hljs-meta">>>> </span>ds = ds.<span class="hljs-built_in">filter</span>(<span class="hljs-keyword">lambda</span> t: pc.equal(t[<span class="hljs-string">"col_0"</span>], <span class="hljs-string">"b"</span>), batched=<span class="hljs-literal">True</span>) | |
| <span class="hljs-meta">>>> </span>ds[<span class="hljs-number">0</span>] | |
| pyarrow.Table | |
| col_0: string | |
| col_1: double | |
| col_2: double | |
| ---- | |
| col_0: [[<span class="hljs-string">"b"</span>]] | |
| col_1: [[<span class="hljs-number">0</span>]] | |
| col_2: [[<span class="hljs-number">1</span>]]<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-4b4wdc">We use <code>batched=True</code> because it is faster to process batches of data in PyArrow rather than row by row. It’s also possible to use <code>batch_size=</code> in <code>map()</code> to set the size of each <code>table</code>.</p> <p data-svelte-h="svelte-1cbojbz">This also works for <a href="/docs/datasets/pr_7489/en/package_reference/main_classes#datasets.IterableDataset.map">IterableDataset.map()</a> and <a href="/docs/datasets/pr_7489/en/package_reference/main_classes#datasets.IterableDataset.filter">IterableDataset.filter()</a>.</p> <h2 class="relative group"><a id="import-or-export-from-pyarrow" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#import-or-export-from-pyarrow"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Import or Export from PyArrow</span></h2> <p data-svelte-h="svelte-qkc3iv">A <a href="/docs/datasets/pr_7489/en/package_reference/main_classes#datasets.Dataset">Dataset</a> is a wrapper of a PyArrow Table, you can instantiate a Dataset directly from the Table:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->ds = Dataset(table)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1rbmfxv">You can access the PyArrow Table of a dataset using <a href="/docs/datasets/pr_7489/en/package_reference/main_classes#datasets.Dataset.data">Dataset.data</a>, which returns a <code>MemoryMappedTable</code> or a <code>InMemoryTable</code> or a <code>ConcatenationTable</code>, depending on the origin of the Arrow data and the operations that were applied.</p> <p data-svelte-h="svelte-1olec7b">Those objects wrap the underlying PyArrow table accessible at <code>Dataset.data.table</code>. This table contains all the data of the dataset, but there might also be an indices mapping at <code>Dataset._indices</code> which maps the dataset rows indices to the PyArrow Table rows indices. This can happen if the dataset has been shuffled with <a href="/docs/datasets/pr_7489/en/package_reference/main_classes#datasets.Dataset.shuffle">Dataset.shuffle()</a> or if only a subset of the rows are used (e.g. after a <a href="/docs/datasets/pr_7489/en/package_reference/main_classes#datasets.Dataset.select">Dataset.select()</a>).</p> <p data-svelte-h="svelte-oy3ykb">In the general case, you can export a dataset to a PyArrow Table using <code>table = ds.with_format("arrow")[:]</code>.</p> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/datasets/blob/main/docs/source/use_with_pyarrow.mdx" target="_blank"><span data-svelte-h="svelte-1kd6by1"><</span> <span data-svelte-h="svelte-x0xyl0">></span> <span data-svelte-h="svelte-1dajgef"><span class="underline ml-1.5">Update</span> on GitHub</span></a> <p></p> | |
| <script> | |
| { | |
| __sveltekit_maf72q = { | |
| assets: "/docs/datasets/pr_7489/en", | |
| base: "/docs/datasets/pr_7489/en", | |
| env: {} | |
| }; | |
| const element = document.currentScript.parentElement; | |
| const data = [null,null]; | |
| Promise.all([ | |
| import("/docs/datasets/pr_7489/en/_app/immutable/entry/start.e78fe3cb.js"), | |
| import("/docs/datasets/pr_7489/en/_app/immutable/entry/app.ec9204ff.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 54], | |
| data, | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 21.4 kB
- Xet hash:
- c92a5f7d92e8f0f9aab70b0bff2d4c2481ff90d58956eb0d8604eff58cfbecde
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.