Buckets:
| <meta charset="utf-8" /><meta http-equiv="content-security-policy" content=""><meta name="hf:doc:metadata" content="{"local":"cloud-storage","sections":[{"local":"listing-datasets","sections":[{"local":"google-cloud-storage","title":"Google Cloud Storage"}],"title":"Listing datasets"},{"local":"saving-datasets","title":"Saving datasets"},{"local":"loading-datasets","title":"Loading datasets"}],"title":"Cloud storage"}" data-svelte="svelte-1phssyn"> | |
| <link rel="modulepreload" href="/docs/datasets/v2.2.2/en/_app/assets/pages/__layout.svelte-efc77dbd.css"> | |
| <link rel="modulepreload" href="/docs/datasets/v2.2.2/en/_app/start-0f8c1da7.js"> | |
| <link rel="modulepreload" href="/docs/datasets/v2.2.2/en/_app/chunks/vendor-8138ceec.js"> | |
| <link rel="modulepreload" href="/docs/datasets/v2.2.2/en/_app/chunks/paths-4b3c6e7e.js"> | |
| <link rel="modulepreload" href="/docs/datasets/v2.2.2/en/_app/pages/__layout.svelte-efb8e839.js"> | |
| <link rel="modulepreload" href="/docs/datasets/v2.2.2/en/_app/pages/filesystems.mdx-a1b23374.js"> | |
| <link rel="modulepreload" href="/docs/datasets/v2.2.2/en/_app/chunks/Tip-12722dfc.js"> | |
| <link rel="modulepreload" href="/docs/datasets/v2.2.2/en/_app/chunks/IconCopyLink-2dd3a6ac.js"> | |
| <link rel="modulepreload" href="/docs/datasets/v2.2.2/en/_app/chunks/CodeBlock-fc89709f.js"> | |
| <h1 class="relative group"><a id="cloud-storage" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#cloud-storage"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> | |
| <span>Cloud storage | |
| </span></h1> | |
| <p>🤗 Datasets supports access to cloud storage providers through a S3 filesystem implementation: <a href="/docs/datasets/v2.2.2/en/package_reference/main_classes#datasets.filesystems.S3FileSystem">filesystems.S3FileSystem</a>. You can save and load datasets from your Amazon S3 bucket in a Pythonic way. Take a look at the following table for other supported cloud storage providers:</p> | |
| <table><thead><tr><th>Storage provider</th> | |
| <th>Filesystem implementation</th></tr></thead> | |
| <tbody><tr><td>Amazon S3</td> | |
| <td><a href="https://s3fs.readthedocs.io/en/latest/" rel="nofollow">s3fs</a></td></tr> | |
| <tr><td>Google Cloud Storage</td> | |
| <td><a href="https://gcsfs.readthedocs.io/en/latest/" rel="nofollow">gcsfs</a></td></tr> | |
| <tr><td>Azure DataLake</td> | |
| <td><a href="https://github.com/dask/adlfs" rel="nofollow">adl</a></td></tr> | |
| <tr><td>Azure Blob</td> | |
| <td><a href="https://github.com/dask/adlfs" rel="nofollow">abfs</a></td></tr> | |
| <tr><td>Dropbox</td> | |
| <td><a href="https://github.com/MarineChap/dropboxdrivefs" rel="nofollow">dropboxdrivefs</a></td></tr> | |
| <tr><td>Google Drive</td> | |
| <td><a href="https://github.com/intake/gdrivefs" rel="nofollow">gdrivefs</a></td></tr></tbody></table> | |
| <p>This guide will show you how to save and load datasets with <strong>s3fs</strong> to a S3 bucket, but other filesystem implementations can be used similarly.</p> | |
| <h2 class="relative group"><a id="listing-datasets" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#listing-datasets"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> | |
| <span>Listing datasets | |
| </span></h2> | |
| <ol><li>Install the S3 dependency with 🤗 Datasets:</li></ol> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START -->>>> pip <span class="hljs-keyword">install </span>datasets[<span class="hljs-built_in">s3</span>]<!-- HTML_TAG_END --></pre></div> | |
| <ol start="2"><li>List files from a public S3 bucket with <code>s3.ls</code>:</li></ol> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">import</span> datasets | |
| <span class="hljs-meta">>>> </span>s3 = datasets.filesystems.S3FileSystem(anon=<span class="hljs-literal">True</span>) | |
| <span class="hljs-meta">>>> </span>s3.ls(<span class="hljs-string">'public-datasets/imdb/train'</span>) | |
| [<span class="hljs-string">'dataset_info.json.json'</span>,<span class="hljs-string">'dataset.arrow'</span>,<span class="hljs-string">'state.json'</span>]<!-- HTML_TAG_END --></pre></div> | |
| <p>Access a private S3 bucket by entering your <code>aws_access_key_id</code> and <code>aws_secret_access_key</code>:</p> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">import</span> datasets | |
| <span class="hljs-meta">>>> </span>s3 = datasets.filesystems.S3FileSystem(key=aws_access_key_id, secret=aws_secret_access_key) | |
| <span class="hljs-meta">>>> </span>s3.ls(<span class="hljs-string">'my-private-datasets/imdb/train'</span>) | |
| [<span class="hljs-string">'dataset_info.json.json'</span>,<span class="hljs-string">'dataset.arrow'</span>,<span class="hljs-string">'state.json'</span>]<!-- HTML_TAG_END --></pre></div> | |
| <h3 class="relative group"><a id="google-cloud-storage" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#google-cloud-storage"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> | |
| <span>Google Cloud Storage | |
| </span></h3> | |
| <p>Other filesystem implementations, like Google Cloud Storage, are used similarly:</p> | |
| <ol><li>Install the Google Cloud Storage implementation:</li></ol> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START --><span class="hljs-meta">>>></span> <span class="language-python">conda install -c conda-forge gcsfs</span> | |
| # or install with pip | |
| <span class="hljs-meta">>>></span> <span class="language-python">pip install gcsfs</span><!-- HTML_TAG_END --></pre></div> | |
| <ol start="2"><li>Load your dataset:</li></ol> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">import</span> gcsfs | |
| <span class="hljs-meta">>>> </span>gcs = gcsfs.GCSFileSystem(project=<span class="hljs-string">'my-google-project'</span>) | |
| <span class="hljs-meta">>>> </span><span class="hljs-comment"># saves encoded_dataset to your s3 bucket</span> | |
| <span class="hljs-meta">>>> </span>encoded_dataset.save_to_disk(<span class="hljs-string">'gcs://my-private-datasets/imdb/train'</span>, fs=gcs)<!-- HTML_TAG_END --></pre></div> | |
| <h2 class="relative group"><a id="saving-datasets" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#saving-datasets"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> | |
| <span>Saving datasets | |
| </span></h2> | |
| <p>After you have processed your dataset, you can save it to S3 with <a href="/docs/datasets/v2.2.2/en/package_reference/main_classes#datasets.Dataset.save_to_disk">Dataset.save_to_disk()</a>:</p> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> datasets.filesystems <span class="hljs-keyword">import</span> S3FileSystem | |
| <span class="hljs-meta">>>> </span><span class="hljs-comment"># create S3FileSystem instance</span> | |
| <span class="hljs-meta">>>> </span>s3 = S3FileSystem(anon=<span class="hljs-literal">True</span>) | |
| <span class="hljs-meta">>>> </span><span class="hljs-comment"># saves encoded_dataset to your s3 bucket</span> | |
| <span class="hljs-meta">>>> </span>encoded_dataset.save_to_disk(<span class="hljs-string">'s3://my-private-datasets/imdb/train'</span>, fs=s3)<!-- HTML_TAG_END --></pre></div> | |
| <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p>Remember to include your <code>aws_access_key_id</code> and <code>aws_secret_access_key</code> whenever you are interacting with a private S3 bucket.</p></div> | |
| <p>Save your dataset with <code>botocore.session.Session</code> and a custom AWS profile:</p> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">import</span> botocore | |
| <span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> datasets.filesystems <span class="hljs-keyword">import</span> S3FileSystem | |
| <span class="hljs-meta">>>> </span><span class="hljs-comment"># creates a botocore session with the provided AWS profile</span> | |
| <span class="hljs-meta">>>> </span>s3_session = botocore.session.Session(profile=<span class="hljs-string">'my_profile_name'</span>) | |
| <span class="hljs-meta">>>> </span><span class="hljs-comment"># create S3FileSystem instance with s3_session</span> | |
| <span class="hljs-meta">>>> </span>s3 = S3FileSystem(session=s3_session) | |
| <span class="hljs-meta">>>> </span><span class="hljs-comment"># saves encoded_dataset to your s3 bucket</span> | |
| <span class="hljs-meta">>>> </span>encoded_dataset.save_to_disk(<span class="hljs-string">'s3://my-private-datasets/imdb/train'</span>,fs=s3)<!-- HTML_TAG_END --></pre></div> | |
| <h2 class="relative group"><a id="loading-datasets" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#loading-datasets"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> | |
| <span>Loading datasets | |
| </span></h2> | |
| <p>When you are ready to use your dataset again, reload it with <a href="/docs/datasets/v2.2.2/en/package_reference/main_classes#datasets.Dataset.load_from_disk">Dataset.load_from_disk()</a>:</p> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> datasets <span class="hljs-keyword">import</span> load_from_disk | |
| <span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> datasets.filesystems <span class="hljs-keyword">import</span> S3FileSystem | |
| <span class="hljs-meta">>>> </span><span class="hljs-comment"># create S3FileSystem without credentials</span> | |
| <span class="hljs-meta">>>> </span>s3 = S3FileSystem(anon=<span class="hljs-literal">True</span>) | |
| <span class="hljs-meta">>>> </span><span class="hljs-comment"># load encoded_dataset to from s3 bucket</span> | |
| <span class="hljs-meta">>>> </span>dataset = load_from_disk(<span class="hljs-string">'s3://a-public-datasets/imdb/train'</span>,fs=s3) | |
| <span class="hljs-meta">>>> </span><span class="hljs-built_in">print</span>(<span class="hljs-built_in">len</span>(dataset)) | |
| <span class="hljs-meta">>>> </span><span class="hljs-comment"># 25000</span><!-- HTML_TAG_END --></pre></div> | |
| <p>Load with <code>botocore.session.Session</code> and custom AWS profile:</p> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">import</span> botocore | |
| <span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> datasets.filesystems <span class="hljs-keyword">import</span> S3FileSystem | |
| <span class="hljs-meta">>>> </span><span class="hljs-comment"># create S3FileSystem instance with aws_access_key_id and aws_secret_access_key</span> | |
| <span class="hljs-meta">>>> </span>s3_session = botocore.session.Session(profile=<span class="hljs-string">'my_profile_name'</span>) | |
| <span class="hljs-meta">>>> </span><span class="hljs-comment"># create S3FileSystem instance with s3_session</span> | |
| <span class="hljs-meta">>>> </span>s3 = S3FileSystem(session=s3_session) | |
| <span class="hljs-meta">>>> </span><span class="hljs-comment"># load encoded_dataset to from s3 bucket</span> | |
| <span class="hljs-meta">>>> </span>dataset = load_from_disk(<span class="hljs-string">'s3://my-private-datasets/imdb/train'</span>,fs=s3) | |
| <span class="hljs-meta">>>> </span><span class="hljs-built_in">print</span>(<span class="hljs-built_in">len</span>(dataset)) | |
| <span class="hljs-meta">>>> </span><span class="hljs-comment"># 25000</span><!-- HTML_TAG_END --></pre></div> | |
| <script type="module" data-hydrate="vvbjx6"> | |
| import { start } from "/docs/datasets/v2.2.2/en/_app/start-0f8c1da7.js"; | |
| start({ | |
| target: document.querySelector('[data-hydrate="vvbjx6"]').parentNode, | |
| paths: {"base":"/docs/datasets/v2.2.2/en","assets":"/docs/datasets/v2.2.2/en"}, | |
| session: {}, | |
| route: false, | |
| spa: false, | |
| trailing_slash: "never", | |
| hydrate: { | |
| status: 200, | |
| error: null, | |
| nodes: [ | |
| import("/docs/datasets/v2.2.2/en/_app/pages/__layout.svelte-efb8e839.js"), | |
| import("/docs/datasets/v2.2.2/en/_app/pages/filesystems.mdx-a1b23374.js") | |
| ], | |
| params: {} | |
| } | |
| }); | |
| </script> | |
Xet Storage Details
- Size:
- 27.3 kB
- Xet hash:
- 42a98629835cace967340b0ea0d9604b6e52ad64ebc624a822cee580a3b1ba52
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.