Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Load pdf data","local":"load-pdf-data","sections":[{"title":"Read pages","local":"read-pages","sections":[],"depth":2},{"title":"Local files","local":"local-files","sections":[],"depth":2},{"title":"PdfFolder","local":"pdffolder","sections":[],"depth":2},{"title":"Pdf decoding","local":"pdf-decoding","sections":[],"depth":2}],"depth":1}"> | |
| <link href="/docs/datasets/pr_7489/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/entry/start.e78fe3cb.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/chunks/scheduler.bdbef820.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/chunks/singletons.de3d48f2.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/chunks/index.8a885b74.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/chunks/paths.8ce597b3.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/entry/app.ec9204ff.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/chunks/index.c0aea24a.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/nodes/0.06409290.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/chunks/each.e59479a4.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/nodes/19.7cf73e8b.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/chunks/Tip.31005f7d.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/chunks/CodeBlock.e814ab8d.js"> | |
| <link rel="modulepreload" href="/docs/datasets/pr_7489/en/_app/immutable/chunks/index.6bcf9ddd.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{"title":"Load pdf data","local":"load-pdf-data","sections":[{"title":"Read pages","local":"read-pages","sections":[],"depth":2},{"title":"Local files","local":"local-files","sections":[],"depth":2},{"title":"PdfFolder","local":"pdffolder","sections":[],"depth":2},{"title":"Pdf decoding","local":"pdf-decoding","sections":[],"depth":2}],"depth":1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <h1 class="relative group"><a id="load-pdf-data" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#load-pdf-data"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Load pdf data</span></h1> <div class="course-tip course-tip-orange bg-gradient-to-br dark:bg-gradient-to-r before:border-orange-500 dark:before:border-orange-800 from-orange-50 dark:from-gray-900 to-white dark:to-gray-950 border border-orange-50 text-orange-700 dark:text-gray-400"><p data-svelte-h="svelte-sbwph">Pdf support is experimental and is subject to change.</p></div> <p data-svelte-h="svelte-zfn130">Pdf datasets have <a href="/docs/datasets/pr_7489/en/package_reference/main_classes#datasets.Pdf">Pdf</a> type columns, which contain <code>pdfplumber</code> objects.</p> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-p0zkvn">To work with pdf datasets, you need to have the <code>pdfplumber</code> package installed. Check out the <a href="https://github.com/pytorch/vision#installation" rel="nofollow">installation</a> guide to learn how to install it.</p></div> <p data-svelte-h="svelte-mkphxd">When you load an pdf dataset and call the pdf column, the pdfs are decoded as <code>pdfplumber</code> Pdfs:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> datasets <span class="hljs-keyword">import</span> load_dataset, Pdf | |
| <span class="hljs-meta">>>> </span>dataset = load_dataset(<span class="hljs-string">"path/to/pdf/folder"</span>, split=<span class="hljs-string">"train"</span>) | |
| <span class="hljs-meta">>>> </span>dataset[<span class="hljs-number">0</span>][<span class="hljs-string">"pdf"</span>] | |
| <pdfplumber.pdf.PDF at <span class="hljs-number">0x1075bc320</span>><!-- HTML_TAG_END --></pre></div> <div class="course-tip course-tip-orange bg-gradient-to-br dark:bg-gradient-to-r before:border-orange-500 dark:before:border-orange-800 from-orange-50 dark:from-gray-900 to-white dark:to-gray-950 border border-orange-50 text-orange-700 dark:text-gray-400"><p data-svelte-h="svelte-1v54wby">Index into an pdf dataset using the row index first and then the <code>pdf</code> column - <code>dataset[0]["pdf"]</code> - to avoid creating all the pdf objects in the dataset. Otherwise, this can be a slow and time-consuming process if you have a large dataset.</p></div> <p data-svelte-h="svelte-12zk6g2">For a guide on how to load any type of dataset, take a look at the <a class="underline decoration-sky-400 decoration-2 font-semibold" href="./loading">general loading guide</a>.</p> <h2 class="relative group"><a id="read-pages" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#read-pages"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Read pages</span></h2> <p data-svelte-h="svelte-7gbnp2">Access pages directly from a pdf using the <code>PDF</code> using <code>.pages</code>.</p> <p data-svelte-h="svelte-ltkjtk">Then you can use the <code>pdfplumber</code> functions to read texts, tables and images, e.g.:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span>pdf = dataset[<span class="hljs-number">0</span>][<span class="hljs-string">"pdf"</span>] | |
| <span class="hljs-meta">>>> </span>first_page = pdf.pages[<span class="hljs-number">0</span>] | |
| <span class="hljs-meta">>>> </span>first_page | |
| <Page:<span class="hljs-number">1</span>> | |
| <span class="hljs-meta">>>> </span>first_page.extract_text() | |
| Docling Technical Report | |
| Version1<span class="hljs-number">.0</span> | |
| ChristophAuer MaksymLysak AhmedNassar MicheleDolfi NikolaosLivathinos | |
| PanosVagenas CesarBerrospiRamis MatteoOmenetti FabianLindlbauer | |
| KasperDinkla LokeshMishra YusikKim ShubhamGupta RafaelTeixeiradeLima | |
| ValeryWeber LucasMorin IngmarMeijer ViktorKuropiatnyk PeterW.J.Staar | |
| AI4KGroup,IBMResearch | |
| Ru¨schlikon,Switzerland | |
| Abstract | |
| This technical report introduces Docling, an easy to use, self-contained, MIT- | |
| licensed <span class="hljs-built_in">open</span>-source package <span class="hljs-keyword">for</span> PDF document conversion. | |
| ... | |
| <span class="hljs-meta">>>> </span>first_page.images | |
| In [<span class="hljs-number">24</span>]: first_page.images | |
| Out[<span class="hljs-number">24</span>]: | |
| [{<span class="hljs-string">'x0'</span>: <span class="hljs-number">256.5</span>, | |
| <span class="hljs-string">'y0'</span>: <span class="hljs-number">621.0</span>, | |
| <span class="hljs-string">'x1'</span>: <span class="hljs-number">355.49519999999995</span>, | |
| <span class="hljs-string">'y1'</span>: <span class="hljs-number">719.9952</span>, | |
| <span class="hljs-string">'width'</span>: <span class="hljs-number">98.99519999999995</span>, | |
| <span class="hljs-string">'height'</span>: <span class="hljs-number">98.99519999999995</span>, | |
| <span class="hljs-string">'name'</span>: <span class="hljs-string">'Im1'</span>, | |
| <span class="hljs-string">'stream'</span>: <PDFStream(<span class="hljs-number">44</span>): raw=<span class="hljs-number">88980</span>, {<span class="hljs-string">'Type'</span>: /<span class="hljs-string">'XObject'</span>, <span class="hljs-string">'Subtype'</span>: /<span class="hljs-string">'Image'</span>, <span class="hljs-string">'BitsPerComponent'</span>: <span class="hljs-number">8</span>, <span class="hljs-string">'ColorSpace'</span>: /<span class="hljs-string">'DeviceRGB'</span>, <span class="hljs-string">'Filter'</span>: /<span class="hljs-string">'DCTDecode'</span>, <span class="hljs-string">'Height'</span>: <span class="hljs-number">1024</span>, <span class="hljs-string">'Length'</span>: <span class="hljs-number">88980</span>, <span class="hljs-string">'Width'</span>: <span class="hljs-number">1024</span>}>, | |
| <span class="hljs-string">'srcsize'</span>: (<span class="hljs-number">1024</span>, <span class="hljs-number">1024</span>), | |
| <span class="hljs-string">'imagemask'</span>: <span class="hljs-literal">None</span>, | |
| <span class="hljs-string">'bits'</span>: <span class="hljs-number">8</span>, | |
| <span class="hljs-string">'colorspace'</span>: [/<span class="hljs-string">'DeviceRGB'</span>], | |
| <span class="hljs-string">'mcid'</span>: <span class="hljs-literal">None</span>, | |
| <span class="hljs-string">'tag'</span>: <span class="hljs-literal">None</span>, | |
| <span class="hljs-string">'object_type'</span>: <span class="hljs-string">'image'</span>, | |
| <span class="hljs-string">'page_number'</span>: <span class="hljs-number">1</span>, | |
| <span class="hljs-string">'top'</span>: <span class="hljs-number">72.00480000000005</span>, | |
| <span class="hljs-string">'bottom'</span>: <span class="hljs-number">171.0</span>, | |
| <span class="hljs-string">'doctop'</span>: <span class="hljs-number">72.00480000000005</span>}] | |
| <span class="hljs-meta">>>> </span>first_page.extract_tables() | |
| []<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-18a6r8m">You can also load each page as a <code>PIL.Image</code>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">import</span> PIL.Image | |
| <span class="hljs-meta">>>> </span><span class="hljs-keyword">import</span> io | |
| <span class="hljs-meta">>>> </span>first_page.to_image() | |
| <pdfplumber.display.PageImage at <span class="hljs-number">0x107d68dd0</span>> | |
| <span class="hljs-meta">>>> </span>buffer = io.BytesIO() | |
| <span class="hljs-meta">>>> </span>first_page.to_image().save(buffer) | |
| <span class="hljs-meta">>>> </span>img = PIL.Image.<span class="hljs-built_in">open</span>(buffer) | |
| <span class="hljs-meta">>>> </span>img | |
| <PIL.PngImagePlugin.PngImageFile image mode=P size=612x792><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-wf7yny">Note that you can pass <code>resolution=</code> to <code>.to_image()</code> to render the image in higher resolution that the default (72 ppi).</p> <h2 class="relative group"><a id="local-files" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#local-files"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Local files</span></h2> <p data-svelte-h="svelte-1jymht4">You can load a dataset from the pdf path. Use the <a href="/docs/datasets/pr_7489/en/package_reference/main_classes#datasets.Dataset.cast_column">cast_column()</a> function to accept a column of pdf file paths, and decode it into a <code>pdfplumber</code> pdf with the <a href="/docs/datasets/pr_7489/en/package_reference/main_classes#datasets.Pdf">Pdf</a> feature:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> datasets <span class="hljs-keyword">import</span> Dataset, Pdf | |
| <span class="hljs-meta">>>> </span>dataset = Dataset.from_dict({<span class="hljs-string">"pdf"</span>: [<span class="hljs-string">"path/to/pdf_1"</span>, <span class="hljs-string">"path/to/pdf_2"</span>, ..., <span class="hljs-string">"path/to/pdf_n"</span>]}).cast_column(<span class="hljs-string">"pdf"</span>, Pdf()) | |
| <span class="hljs-meta">>>> </span>dataset[<span class="hljs-number">0</span>][<span class="hljs-string">"pdf"</span>] | |
| <pdfplumber.pdf.PDF at <span class="hljs-number">0x1657d0280</span>><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-ctm6j1">If you only want to load the underlying path to the pdf dataset without decoding the pdf object, set <code>decode=False</code> in the <a href="/docs/datasets/pr_7489/en/package_reference/main_classes#datasets.Pdf">Pdf</a> feature:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span>dataset = dataset.cast_column(<span class="hljs-string">"pdf"</span>, Pdf(decode=<span class="hljs-literal">False</span>)) | |
| <span class="hljs-meta">>>> </span>dataset[<span class="hljs-number">0</span>][<span class="hljs-string">"pdf"</span>] | |
| {<span class="hljs-string">'bytes'</span>: <span class="hljs-literal">None</span>, | |
| <span class="hljs-string">'path'</span>: <span class="hljs-string">'path/to/pdf/folder/pdf0.pdf'</span>}<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="pdffolder" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#pdffolder"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>PdfFolder</span></h2> <p data-svelte-h="svelte-172l8l0">You can also load a dataset with an <code>PdfFolder</code> dataset builder which does not require writing a custom dataloader. This makes <code>PdfFolder</code> ideal for quickly creating and loading pdf datasets with several thousand pdfs for different vision tasks. Your pdf dataset structure should look like this:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->folder<span class="hljs-regexp">/train/</span>resume/<span class="hljs-number">0001</span>.pdf | |
| folder<span class="hljs-regexp">/train/</span>resume/<span class="hljs-number">0002</span>.pdf | |
| folder<span class="hljs-regexp">/train/</span>resume/<span class="hljs-number">0003</span>.pdf | |
| folder<span class="hljs-regexp">/train/i</span>nvoice/<span class="hljs-number">0001</span>.pdf | |
| folder<span class="hljs-regexp">/train/i</span>nvoice/<span class="hljs-number">0002</span>.pdf | |
| folder<span class="hljs-regexp">/train/i</span>nvoice/<span class="hljs-number">0003</span>.pdf<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-caemd3">If the dataset follows the <code>PdfFolder</code> structure, then you can load it directly with <a href="/docs/datasets/pr_7489/en/package_reference/loading_methods#datasets.load_dataset">load_dataset()</a>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> datasets <span class="hljs-keyword">import</span> load_dataset | |
| <span class="hljs-meta">>>> </span>dataset = load_dataset(<span class="hljs-string">"username/dataset_name"</span>) | |
| <span class="hljs-meta">>>> </span><span class="hljs-comment"># OR locally:</span> | |
| <span class="hljs-meta">>>> </span>dataset = load_dataset(<span class="hljs-string">"/path/to/folder"</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-j3sf20">For local datasets, this is equivalent to passing <code>pdffolder</code> manually in <a href="/docs/datasets/pr_7489/en/package_reference/loading_methods#datasets.load_dataset">load_dataset()</a> and the directory in <code>data_dir</code>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span>dataset = load_dataset(<span class="hljs-string">"pdffolder"</span>, data_dir=<span class="hljs-string">"/path/to/folder"</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-fw8xy6">Then you can access the pdfs as <code>pdfplumber.pdf.PDF</code> objects:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->>>> dataset<span class="hljs-selector-attr">[<span class="hljs-string">"train"</span>]</span><span class="hljs-selector-attr">[0]</span> | |
| {<span class="hljs-string">"pdf"</span>: <pdfplumber<span class="hljs-selector-class">.pdf</span><span class="hljs-selector-class">.PDF</span> at <span class="hljs-number">0</span>x161715e50>, <span class="hljs-string">"label"</span>: <span class="hljs-number">0</span>} | |
| >>> dataset<span class="hljs-selector-attr">[<span class="hljs-string">"train"</span>]</span><span class="hljs-selector-attr">[-1]</span> | |
| {<span class="hljs-string">"pdf"</span>: <pdfplumber<span class="hljs-selector-class">.pdf</span><span class="hljs-selector-class">.PDF</span> at <span class="hljs-number">0</span>x16170bd90>, <span class="hljs-string">"label"</span>: <span class="hljs-number">1</span>}<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1qv6npe">To ignore the information in the metadata file, set <code>drop_metadata=True</code> in <a href="/docs/datasets/pr_7489/en/package_reference/loading_methods#datasets.load_dataset">load_dataset()</a>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> datasets <span class="hljs-keyword">import</span> load_dataset | |
| <span class="hljs-meta">>>> </span>dataset = load_dataset(<span class="hljs-string">"username/dataset_with_metadata"</span>, drop_metadata=<span class="hljs-literal">True</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1yr2vgw">If you don’t have a metadata file, <code>PdfFolder</code> automatically infers the label name from the directory name. | |
| If you want to drop automatically created labels, set <code>drop_labels=True</code>. | |
| In this case, your dataset will only contain an pdf column:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> datasets <span class="hljs-keyword">import</span> load_dataset | |
| <span class="hljs-meta">>>> </span>dataset = load_dataset(<span class="hljs-string">"username/dataset_without_metadata"</span>, drop_labels=<span class="hljs-literal">True</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1jur300">Finally the <code>filters</code> argument lets you load only a subset of the dataset, based on a condition on the label or the metadata. This is especially useful if the metadata is in Parquet format, since this format enables fast filtering. It is also recommended to use this argument with <code>streaming=True</code>, because by default the dataset is fully downloaded before filtering.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span>filters = [(<span class="hljs-string">"label"</span>, <span class="hljs-string">"="</span>, <span class="hljs-number">0</span>)] | |
| <span class="hljs-meta">>>> </span>dataset = load_dataset(<span class="hljs-string">"username/dataset_name"</span>, streaming=<span class="hljs-literal">True</span>, filters=filters)<!-- HTML_TAG_END --></pre></div> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-uqvt95">For more information about creating your own <code>PdfFolder</code> dataset, take a look at the <a href="./pdf_dataset">Create a pdf dataset</a> guide.</p></div> <h2 class="relative group"><a id="pdf-decoding" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#pdf-decoding"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Pdf decoding</span></h2> <p data-svelte-h="svelte-1445417">By default, pdfs are decoded sequentially as pdfplumber <code>PDFs</code> when you iterate on a dataset. | |
| It sequentially decodes the metadata of the pdfs, and doesn’t read the pdf pages until you access them.</p> <p data-svelte-h="svelte-fdii9b">However it is possible to speed up the dataset significantly using multithreaded decoding:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">import</span> os | |
| <span class="hljs-meta">>>> </span>num_threads = num_threads = <span class="hljs-built_in">min</span>(<span class="hljs-number">32</span>, (os.cpu_count() <span class="hljs-keyword">or</span> <span class="hljs-number">1</span>) + <span class="hljs-number">4</span>) | |
| <span class="hljs-meta">>>> </span>dataset = dataset.decode(num_threads=num_threads) | |
| <span class="hljs-meta">>>> </span><span class="hljs-keyword">for</span> example <span class="hljs-keyword">in</span> dataset: <span class="hljs-comment"># up to 20 times faster !</span> | |
| <span class="hljs-meta">... </span> ...<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1hi5tci">You can enable multithreading using <code>num_threads</code>. This is especially useful to speed up remote data streaming. | |
| However it can be slower than <code>num_threads=0</code> for local data on fast disks.</p> <p data-svelte-h="svelte-v56ibb">If you are not interested in the documents decoded as pdfplumber <code>PDFs</code> and would like to access the path/bytes instead, you can disable decoding:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span>dataset = dataset.decode(<span class="hljs-literal">False</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-wbxk4b">Note: <a href="/docs/datasets/pr_7489/en/package_reference/main_classes#datasets.IterableDataset.decode">IterableDataset.decode()</a> is only available for streaming datasets at the moment.</p> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/datasets/blob/main/docs/source/document_load.mdx" target="_blank"><span data-svelte-h="svelte-1kd6by1"><</span> <span data-svelte-h="svelte-x0xyl0">></span> <span data-svelte-h="svelte-1dajgef"><span class="underline ml-1.5">Update</span> on GitHub</span></a> <p></p> | |
| <script> | |
| { | |
| __sveltekit_maf72q = { | |
| assets: "/docs/datasets/pr_7489/en", | |
| base: "/docs/datasets/pr_7489/en", | |
| env: {} | |
| }; | |
| const element = document.currentScript.parentElement; | |
| const data = [null,null]; | |
| Promise.all([ | |
| import("/docs/datasets/pr_7489/en/_app/immutable/entry/start.e78fe3cb.js"), | |
| import("/docs/datasets/pr_7489/en/_app/immutable/entry/app.ec9204ff.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 19], | |
| data, | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 43.8 kB
- Xet hash:
- 48a9925e2d3cbe082a5dc31e2e7c9a567637139e8e078c5d128240907428f7c6
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.