Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Image Dataset","local":"image-dataset","sections":[{"title":"Only images","local":"only-images","sections":[],"depth":2},{"title":"Additional columns","local":"additional-columns","sections":[],"depth":2},{"title":"Relative paths","local":"relative-paths","sections":[],"depth":2},{"title":"Image classification","local":"image-classification","sections":[],"depth":2},{"title":"Large scale datasets","local":"large-scale-datasets","sections":[{"title":"WebDataset format","local":"webdataset-format","sections":[],"depth":3},{"title":"Parquet format","local":"parquet-format","sections":[],"depth":3}],"depth":2}],"depth":1}"> | |
| <link href="/docs/hub/pr_2437/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload"> | |
| <link rel="modulepreload" href="/docs/hub/pr_2437/en/_app/immutable/entry/start.5e3ed1fd.js"> | |
| <link rel="modulepreload" href="/docs/hub/pr_2437/en/_app/immutable/chunks/scheduler.258d2a4d.js"> | |
| <link rel="modulepreload" href="/docs/hub/pr_2437/en/_app/immutable/chunks/singletons.2d0c91e1.js"> | |
| <link rel="modulepreload" href="/docs/hub/pr_2437/en/_app/immutable/chunks/index.c8b82093.js"> | |
| <link rel="modulepreload" href="/docs/hub/pr_2437/en/_app/immutable/chunks/paths.d360121b.js"> | |
| <link rel="modulepreload" href="/docs/hub/pr_2437/en/_app/immutable/entry/app.93e7704b.js"> | |
| <link rel="modulepreload" href="/docs/hub/pr_2437/en/_app/immutable/chunks/preload-helper.a4507a26.js"> | |
| <link rel="modulepreload" href="/docs/hub/pr_2437/en/_app/immutable/chunks/index.421344fd.js"> | |
| <link rel="modulepreload" href="/docs/hub/pr_2437/en/_app/immutable/nodes/0.bedbef6e.js"> | |
| <link rel="modulepreload" href="/docs/hub/pr_2437/en/_app/immutable/chunks/each.e59479a4.js"> | |
| <link rel="modulepreload" href="/docs/hub/pr_2437/en/_app/immutable/nodes/46.9f085fea.js"> | |
| <link rel="modulepreload" href="/docs/hub/pr_2437/en/_app/immutable/chunks/CopyLLMTxtMenu.9c2a67a1.js"> | |
| <link rel="modulepreload" href="/docs/hub/pr_2437/en/_app/immutable/chunks/MermaidChart.svelte_svelte_type_style_lang.9bb92958.js"> | |
| <link rel="modulepreload" href="/docs/hub/pr_2437/en/_app/immutable/chunks/CodeBlock.619ec4e3.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{"title":"Image Dataset","local":"image-dataset","sections":[{"title":"Only images","local":"only-images","sections":[],"depth":2},{"title":"Additional columns","local":"additional-columns","sections":[],"depth":2},{"title":"Relative paths","local":"relative-paths","sections":[],"depth":2},{"title":"Image classification","local":"image-classification","sections":[],"depth":2},{"title":"Large scale datasets","local":"large-scale-datasets","sections":[{"title":"WebDataset format","local":"webdataset-format","sections":[],"depth":3},{"title":"Parquet format","local":"parquet-format","sections":[],"depth":3}],"depth":2}],"depth":1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <div class="items-center shrink-0 min-w-[100px] max-sm:min-w-[50px] justify-end ml-auto flex" style="float: right; margin-left: 10px; display: inline-flex; position: relative; z-index: 10;"><div class="inline-flex rounded-md max-sm:rounded-sm"><button class="inline-flex items-center gap-1 h-7 max-sm:h-7 px-2 max-sm:px-1.5 text-sm font-medium text-gray-800 border border-r-0 rounded-l-md max-sm:rounded-l-sm border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-live="polite"><span class="inline-flex items-center justify-center rounded-md p-0.5 max-sm:p-0 hover:text-gray-800 dark:hover:text-gray-200"><svg class="sm:size-3.5 size-3" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg></span> <span>Copy page</span></button> <button class="inline-flex items-center justify-center w-6 max-sm:w-5 h-7 max-sm:h-7 disabled:pointer-events-none text-sm text-gray-500 hover:text-gray-700 dark:hover:text-white rounded-r-md max-sm:rounded-r-sm border border-l transition border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-haspopup="menu" aria-expanded="false" aria-label="Open copy menu"><svg class="transition-transform text-gray-400 overflow-visible sm:size-3.5 size-3 rotate-0" width="1em" height="1em" viewBox="0 0 12 7" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M1 1L6 6L11 1" stroke="currentColor"></path></svg></button></div> </div> <h1 class="relative group"><a id="image-dataset" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#image-dataset"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Image Dataset</span></h1> <p data-svelte-h="svelte-v7rbuf">This guide will show you how to configure your dataset repository with image files. You can find accompanying examples of repositories in this <a href="https://huggingface.co/collections/datasets-examples/image-dataset-6568e7cf28639db76eb92d65" rel="nofollow">Image datasets examples collection</a>.</p> <p data-svelte-h="svelte-u1gdfq">A dataset with a supported structure and <a href="./datasets-adding#file-formats">file formats</a> automatically has a Dataset Viewer on its page on the Hub.</p> <p data-svelte-h="svelte-1ymhost">Additional information about your images - such as captions or bounding boxes for object detection - is automatically loaded as long as you include this information in a metadata file (<code>metadata.csv</code>/<code>metadata.jsonl</code>/<code>metadata.parquet</code>).</p> <p data-svelte-h="svelte-8l0ovg">Alternatively, images can be in Parquet files or in TAR archives following the <a href="https://github.com/webdataset/webdataset" rel="nofollow">WebDataset</a> format.</p> <h2 class="relative group"><a id="only-images" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#only-images"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Only images</span></h2> <p data-svelte-h="svelte-37nscg">If your dataset only consists of one column with images, you can simply store your image files at the root:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->my_dataset_repository/ | |
| ├── <span class="hljs-number">1</span><span class="hljs-selector-class">.jpg</span> | |
| ├── <span class="hljs-number">2</span><span class="hljs-selector-class">.jpg</span> | |
| ├── <span class="hljs-number">3</span><span class="hljs-selector-class">.jpg</span> | |
| └── <span class="hljs-number">4</span>.jpg<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-19pjkki">or in a subdirectory:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->my_dataset_repository/ | |
| └── images | |
| ├── <span class="hljs-number">1</span><span class="hljs-selector-class">.jpg</span> | |
| ├── <span class="hljs-number">2</span><span class="hljs-selector-class">.jpg</span> | |
| ├── <span class="hljs-number">3</span><span class="hljs-selector-class">.jpg</span> | |
| └── <span class="hljs-number">4</span>.jpg<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-e5vckm">Multiple <a href="./datasets-adding#file-formats">formats</a> are supported at the same time, including PNG, JPEG, TIFF and WebP.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->my_dataset_repository/ | |
| └── images | |
| ├── <span class="hljs-number">1</span><span class="hljs-selector-class">.jpg</span> | |
| ├── <span class="hljs-number">2</span><span class="hljs-selector-class">.png</span> | |
| ├── <span class="hljs-number">3</span><span class="hljs-selector-class">.tiff</span> | |
| └── <span class="hljs-number">4</span>.webp<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-l8z8xp">If you have several splits, you can put your images into directories named accordingly:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->my_dataset_repository/ | |
| ├── train | |
| │ ├── <span class="hljs-number">1</span><span class="hljs-selector-class">.jpg</span> | |
| │ └── <span class="hljs-number">2</span><span class="hljs-selector-class">.jpg</span> | |
| └── test | |
| ├── <span class="hljs-number">3</span><span class="hljs-selector-class">.jpg</span> | |
| └── <span class="hljs-number">4</span>.jpg<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-kefeul">See <a href="./datasets-file-names-and-splits">File names and splits</a> for more information and other ways to organize data by splits.</p> <h2 class="relative group"><a id="additional-columns" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#additional-columns"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Additional columns</span></h2> <p data-svelte-h="svelte-1ehvjow">If there is additional information you’d like to include about your dataset, like text captions or bounding boxes, add it as a <code>metadata.csv</code> file in your repository. This lets you quickly create datasets for different computer vision tasks like <a href="https://huggingface.co/tasks/image-to-text" rel="nofollow">text captioning</a> or <a href="https://huggingface.co/tasks/object-detection" rel="nofollow">object detection</a>.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->my_dataset_repository/ | |
| └── train | |
| ├── <span class="hljs-number">1</span><span class="hljs-selector-class">.jpg</span> | |
| ├── <span class="hljs-number">2</span><span class="hljs-selector-class">.jpg</span> | |
| ├── <span class="hljs-number">3</span><span class="hljs-selector-class">.jpg</span> | |
| ├── <span class="hljs-number">4</span><span class="hljs-selector-class">.jpg</span> | |
| └── metadata.csv<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-16f2gib">Your <code>metadata.csv</code> file must have a <code>file_name</code> column which links image files with their metadata:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->file_name,<span class="hljs-keyword">text</span> | |
| <span class="hljs-number">1.</span>jpg,<span class="hljs-keyword">a</span> drawing <span class="hljs-keyword">of</span> <span class="hljs-keyword">a</span> green pokemon <span class="hljs-keyword">with</span> red eyes | |
| <span class="hljs-number">2.</span>jpg,<span class="hljs-keyword">a</span> green <span class="hljs-keyword">and</span> yellow toy <span class="hljs-keyword">with</span> <span class="hljs-keyword">a</span> red nose | |
| <span class="hljs-number">3.</span>jpg,<span class="hljs-keyword">a</span> red <span class="hljs-keyword">and</span> white ball <span class="hljs-keyword">with</span> <span class="hljs-keyword">an</span> angry look <span class="hljs-keyword">on</span> <span class="hljs-title">its</span> <span class="hljs-title">face</span> | |
| <span class="hljs-number">4.</span>jpg,<span class="hljs-keyword">a</span> cartoon ball <span class="hljs-keyword">with</span> <span class="hljs-keyword">a</span> smile <span class="hljs-keyword">on</span> <span class="hljs-title">its</span> <span class="hljs-title">face</span><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1md2l6g">You can also use a <a href="https://jsonlines.org/" rel="nofollow">JSONL</a> file <code>metadata.jsonl</code>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->{<span class="hljs-comment">"file_name"</span>: <span class="hljs-comment">"1.jpg"</span>,<span class="hljs-comment">"text"</span>: <span class="hljs-comment">"a drawing of a green pokemon with red eyes"</span>} | |
| {<span class="hljs-comment">"file_name"</span>: <span class="hljs-comment">"2.jpg"</span>,<span class="hljs-comment">"text"</span>: <span class="hljs-comment">"a green and yellow toy with a red nose"</span>} | |
| {<span class="hljs-comment">"file_name"</span>: <span class="hljs-comment">"3.jpg"</span>,<span class="hljs-comment">"text"</span>: <span class="hljs-comment">"a red and white ball with an angry look on its face"</span>} | |
| {<span class="hljs-comment">"file_name"</span>: <span class="hljs-comment">"4.jpg"</span>,<span class="hljs-comment">"text"</span>: <span class="hljs-comment">"a cartoon ball with a smile on its face"</span>}<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-17vzj19">And for bigger datasets or if you are interested in advanced data retrieval features, you can use a <a href="https://parquet.apache.org/" rel="nofollow">Parquet</a> file <code>metadata.parquet</code>.</p> <h2 class="relative group"><a id="relative-paths" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#relative-paths"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Relative paths</span></h2> <p data-svelte-h="svelte-1gwaj92">Metadata file must be located either in the same directory with the images it is linked to, or in any parent directory, like in this example:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->my_dataset_repository/ | |
| └── train | |
| ├── images | |
| │ ├── <span class="hljs-number">1</span><span class="hljs-selector-class">.jpg</span> | |
| │ ├── <span class="hljs-number">2</span><span class="hljs-selector-class">.jpg</span> | |
| │ ├── <span class="hljs-number">3</span><span class="hljs-selector-class">.jpg</span> | |
| │ └── <span class="hljs-number">4</span><span class="hljs-selector-class">.jpg</span> | |
| └── metadata.csv<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1wjvumb">In this case, the <code>file_name</code> column must be a full relative path to the images, not just the filename:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->file_name,<span class="hljs-keyword">text</span> | |
| images/<span class="hljs-number">1.</span>jpg,<span class="hljs-keyword">a</span> drawing <span class="hljs-keyword">of</span> <span class="hljs-keyword">a</span> green pokemon <span class="hljs-keyword">with</span> red eyes | |
| images/<span class="hljs-number">2.</span>jpg,<span class="hljs-keyword">a</span> green <span class="hljs-keyword">and</span> yellow toy <span class="hljs-keyword">with</span> <span class="hljs-keyword">a</span> red nose | |
| images/<span class="hljs-number">3.</span>jpg,<span class="hljs-keyword">a</span> red <span class="hljs-keyword">and</span> white ball <span class="hljs-keyword">with</span> <span class="hljs-keyword">an</span> angry look <span class="hljs-keyword">on</span> <span class="hljs-title">its</span> <span class="hljs-title">face</span> | |
| images/<span class="hljs-number">4.</span>jpg,<span class="hljs-keyword">a</span> cartoon ball <span class="hljs-keyword">with</span> <span class="hljs-keyword">a</span> smile <span class="hljs-keyword">on</span> <span class="hljs-title">it</span><span class="hljs-string">'s face</span><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1xb8ill">Metadata files cannot be put in subdirectories of a directory with the images.</p> <p data-svelte-h="svelte-15baaod">More generally, any column named <code>file_name</code> or <code>*_file_name</code> should contain the full relative path to the images.</p> <h2 class="relative group"><a id="image-classification" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#image-classification"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Image classification</span></h2> <p data-svelte-h="svelte-z798u1">For image classification datasets, you can also use a simple setup: use directories to name the image classes. Store your image files in a directory structure like:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->my_dataset_repository/ | |
| ├── green | |
| │ ├── <span class="hljs-number">1</span><span class="hljs-selector-class">.jpg</span> | |
| │ └── <span class="hljs-number">2</span><span class="hljs-selector-class">.jpg</span> | |
| └── red | |
| ├── <span class="hljs-number">3</span><span class="hljs-selector-class">.jpg</span> | |
| └── <span class="hljs-number">4</span>.jpg<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1j7sm10">The dataset created with this structure contains two columns: <code>image</code> and <code>label</code> (with values <code>green</code> and <code>red</code>).</p> <p data-svelte-h="svelte-jxxgnh">You can also provide multiple splits. To do so, your dataset directory should have the following structure (see <a href="./datasets-file-names-and-splits">File names and splits</a> for more information):</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->my_dataset_repository/ | |
| ├── test | |
| │ ├── <span class="hljs-built_in">green</span> | |
| │ │ └── <span class="hljs-number">2.</span>jpg | |
| │ └── <span class="hljs-built_in">red</span> | |
| │ └── <span class="hljs-number">4.</span>jpg | |
| └── train | |
| ├── <span class="hljs-built_in">green</span> | |
| │ └── <span class="hljs-number">1.</span>jpg | |
| └── <span class="hljs-built_in">red</span> | |
| └── <span class="hljs-number">3.</span>jpg<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-op9ivs">You can disable this automatic addition of the <code>label</code> column in the <a href="./datasets-manual-configuration">YAML configuration</a>. If your directory names have no special meaning, set <code>drop_labels: true</code> in the README header:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-attr">configs:</span> | |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">config_name:</span> <span class="hljs-string">default</span> <span class="hljs-comment"># Name of the dataset subset, if applicable.</span> | |
| <span class="hljs-attr">drop_labels:</span> <span class="hljs-literal">true</span><!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="large-scale-datasets" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#large-scale-datasets"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Large scale datasets</span></h2> <h3 class="relative group"><a id="webdataset-format" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#webdataset-format"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>WebDataset format</span></h3> <p data-svelte-h="svelte-1xyr3zu">The <a href="./datasets-webdataset">WebDataset</a> format is well suited for large scale image datasets (see <a href="https://huggingface.co/datasets/timm/imagenet-12k-wds" rel="nofollow">timm/imagenet-12k-wds</a> for example). | |
| It consists of TAR archives containing images and their metadata and is optimized for streaming. It is useful if you have a large number of images and to get streaming data loaders for large scale training.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->my_dataset_repository/ | |
| ├── train<span class="hljs-string">-0000</span>.tar | |
| ├── train<span class="hljs-string">-0001</span>.tar | |
| ├── ... | |
| └── train<span class="hljs-string">-1023</span>.tar<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1gf9xhb">To make a WebDataset TAR archive, create a directory containing the images and metadata files to be archived and create the TAR archive using e.g. the <code>tar</code> command. | |
| The usual size per archive is generally around 1GB. | |
| Make sure each image and metadata pair share the same file prefix, for example:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->train-<span class="hljs-number">0000</span>/ | |
| ├── <span class="hljs-number">000</span><span class="hljs-selector-class">.jpg</span> | |
| ├── <span class="hljs-number">000</span><span class="hljs-selector-class">.json</span> | |
| ├── <span class="hljs-number">001</span><span class="hljs-selector-class">.jpg</span> | |
| ├── <span class="hljs-number">001</span><span class="hljs-selector-class">.json</span> | |
| ├── ... | |
| ├── <span class="hljs-number">999</span><span class="hljs-selector-class">.jpg</span> | |
| └── <span class="hljs-number">999</span>.json<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-zo0rpw">Note that for user convenience and to enable the <a href="./data-studio">Dataset Viewer</a>, every dataset hosted in the Hub is automatically converted to Parquet format up to 5GB. | |
| Read more about it in the <a href="./data-studio#access-the-parquet-files">Parquet format</a> documentation.</p> <h3 class="relative group"><a id="parquet-format" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#parquet-format"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Parquet format</span></h3> <p data-svelte-h="svelte-k79ne">Instead of uploading the images and metadata as individual files, you can embed everything inside a <a href="https://parquet.apache.org/" rel="nofollow">Parquet</a> file. | |
| This is useful if you have a large number of images, if you want to embed multiple image columns, or if you want to store additional information about the images in the same file. | |
| Parquet is also useful for storing data such as raw bytes, which is not supported by JSON/CSV.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->my_dataset_repository/ | |
| └── train.parquet<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-y4diuk">Parquet files with image data can be created using <code>pandas</code> or the <code>datasets</code> library. To create Parquet files with image data in <code>pandas</code>, you can use <a href="https://github.com/lhoestq/pandas-image-methods" rel="nofollow">pandas-image-methods</a> and <code>df.to_parquet()</code>. In <code>datasets</code>, you can set the column type to <code>Image()</code> and use the <code>ds.to_parquet(...)</code> method or <code>ds.push_to_hub(...)</code>. You can find a guide on loading image datasets in <code>datasets</code> <a href="/docs/datasets/image_load">here</a>.</p> <p data-svelte-h="svelte-149tbjs">Alternatively you can manually set the image type of Parquet created using other tools. First, make sure your image columns are of type <em>struct</em>, with a binary field <code>"bytes"</code> for the image data and a string field <code>"path"</code> for the image file name or path. Then you should specify the feature types of the columns directly in YAML in the README header, for example:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-attr">dataset_info:</span> | |
| <span class="hljs-attr">features:</span> | |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">image</span> | |
| <span class="hljs-attr">dtype:</span> <span class="hljs-string">image</span> | |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">caption</span> | |
| <span class="hljs-attr">dtype:</span> <span class="hljs-string">string</span><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-59yffi">Note that Parquet is recommended for small images (<1MB per image) and small row groups (100 rows per row group, which is what <code>datasets</code> uses for images). For larger images it is recommended to use the WebDataset format, or to share the original image files (optionally with metadata files, and following the <a href="https://huggingface.co/docs/hub/en/storage-limits" rel="nofollow">repositories recommendations and limits</a> for storage and number of files).</p> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/hub-docs/blob/main/docs/hub/datasets-image.md" target="_blank"><svg class="mr-1" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M31,16l-7,7l-1.41-1.41L28.17,16l-5.58-5.59L24,9l7,7z"></path><path d="M1,16l7-7l1.41,1.41L3.83,16l5.58,5.59L8,23l-7-7z"></path><path d="M12.419,25.484L17.639,6.552l1.932,0.518L14.351,26.002z"></path></svg> <span data-svelte-h="svelte-zjs2n5"><span class="underline">Update</span> on GitHub</span></a> <p></p> | |
| <script> | |
| { | |
| __sveltekit_ummgov = { | |
| assets: "/docs/hub/pr_2437/en", | |
| base: "/docs/hub/pr_2437/en", | |
| env: {} | |
| }; | |
| const element = document.currentScript.parentElement; | |
| const data = [null,null]; | |
| Promise.all([ | |
| import("/docs/hub/pr_2437/en/_app/immutable/entry/start.5e3ed1fd.js"), | |
| import("/docs/hub/pr_2437/en/_app/immutable/entry/app.93e7704b.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 46], | |
| data, | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 51.1 kB
- Xet hash:
- 6051cfcd967c6f1ecd75b2d1b9c526a29fd2f191671b52b742e94adcccdb3807
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.