Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Build an embedding pipeline with datasets","local":"build-an-embedding-pipeline-with-datasets","sections":[{"title":"Create your embedding Endpoint","local":"create-your-embedding-endpoint","sections":[],"depth":2},{"title":"Test your Endpoint","local":"test-your-endpoint","sections":[],"depth":2},{"title":"Get your Endpoint’s details","local":"get-your-endpoints-details","sections":[],"depth":2},{"title":"Building the embedding script","local":"building-the-embedding-script","sections":[{"title":"Step 1: Set up dependencies and imports","local":"step-1-set-up-dependencies-and-imports","sections":[],"depth":3},{"title":"Step 2: Configure the connection","local":"step-2-configure-the-connection","sections":[],"depth":3},{"title":"Step 3: Create the embedding function","local":"step-3-create-the-embedding-function","sections":[],"depth":3},{"title":"Step 4: Load and process your dataset","local":"step-4-load-and-process-your-dataset","sections":[],"depth":3},{"title":"Step 5: Save and Share your results","local":"step-5-save-and-share-your-results","sections":[],"depth":3}],"depth":2},{"title":"Next steps","local":"next-steps","sections":[],"depth":2}],"depth":1}"> | |
| <link href="/docs/inference-endpoints/pr_136/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_136/en/_app/immutable/entry/start.fb9ab4d6.js"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_136/en/_app/immutable/chunks/scheduler.f6b352c8.js"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_136/en/_app/immutable/chunks/singletons.ceca4163.js"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_136/en/_app/immutable/chunks/index.26cf6c5a.js"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_136/en/_app/immutable/chunks/paths.142cd5df.js"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_136/en/_app/immutable/entry/app.6247727a.js"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_136/en/_app/immutable/chunks/index.b90df637.js"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_136/en/_app/immutable/nodes/0.2fcde12d.js"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_136/en/_app/immutable/chunks/each.e59479a4.js"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_136/en/_app/immutable/nodes/23.9af61fce.js"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_136/en/_app/immutable/chunks/Tip.366d2e6e.js"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_136/en/_app/immutable/chunks/CodeBlock.e5718f9d.js"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_136/en/_app/immutable/chunks/getInferenceSnippets.1e3ae0bf.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{"title":"Build an embedding pipeline with datasets","local":"build-an-embedding-pipeline-with-datasets","sections":[{"title":"Create your embedding Endpoint","local":"create-your-embedding-endpoint","sections":[],"depth":2},{"title":"Test your Endpoint","local":"test-your-endpoint","sections":[],"depth":2},{"title":"Get your Endpoint’s details","local":"get-your-endpoints-details","sections":[],"depth":2},{"title":"Building the embedding script","local":"building-the-embedding-script","sections":[{"title":"Step 1: Set up dependencies and imports","local":"step-1-set-up-dependencies-and-imports","sections":[],"depth":3},{"title":"Step 2: Configure the connection","local":"step-2-configure-the-connection","sections":[],"depth":3},{"title":"Step 3: Create the embedding function","local":"step-3-create-the-embedding-function","sections":[],"depth":3},{"title":"Step 4: Load and process your dataset","local":"step-4-load-and-process-your-dataset","sections":[],"depth":3},{"title":"Step 5: Save and Share your results","local":"step-5-save-and-share-your-results","sections":[],"depth":3}],"depth":2},{"title":"Next steps","local":"next-steps","sections":[],"depth":2}],"depth":1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <h1 class="relative group"><a id="build-an-embedding-pipeline-with-datasets" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#build-an-embedding-pipeline-with-datasets"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Build an embedding pipeline with datasets</span></h1> <p data-svelte-h="svelte-y0ssck">This tutorial will guide you through deploying an embedding endpoint and building a Python script to efficiently process datasets with embeddings. We’ll use the powerful <a href="https://huggingface.co/Qwen/Qwen3-Embedding-4B" rel="nofollow">Qwen/Qwen3-Embedding-4B</a> model to create high-quality embeddings for your data.</p> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-1s3ow1a">This tutorial focuses on creating a production-ready script that can process any dataset and add embeddings using the <strong>Text Embeddings Inference (TEI)</strong> engine for optimized performance.</p></div> <h2 class="relative group"><a id="create-your-embedding-endpoint" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#create-your-embedding-endpoint"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Create your embedding Endpoint</span></h2> <p data-svelte-h="svelte-qqd7zu">First, we need to create an Inference Endpoint optimized for embeddings.</p> <p data-svelte-h="svelte-147h3qd">Start by navigating to the Inference Endpoints UI, and once you have logged in you should see a button for creating a new Inference | |
| Endpoint. Click the “New” button.</p> <p data-svelte-h="svelte-dnyg4"><img src="https://raw.githubusercontent.com/huggingface/hf-endpoints-documentation/main/assets/quick_start/1-new-button.png" alt="new-button"></p> <p data-svelte-h="svelte-161g95l">From there you’ll be directed to the catalog. The Model Catalog consists of popular models which have tuned configurations to work as one-click | |
| deploys. You can search for embedding models or create a custom endpoint.</p> <p data-svelte-h="svelte-fxecmn"><img src="https://raw.githubusercontent.com/huggingface/hf-endpoints-documentation/main/assets/quick_start/2-catalog.png" alt="catalog"></p> <p data-svelte-h="svelte-hre7rz">For this tutorial, we’ll use the Qwen3-Embedding-4B model available in the Inference Endpoints Model Catalog. Note if it’s ever not in the catalog, you can deploy a model as a custom Endpoint from the Hugging Face Hub by entering the model repository ID <code>Qwen/Qwen3-Embedding-4B</code>.</p> <p data-svelte-h="svelte-47pfzw">For embedding models, we recommend:</p> <ul data-svelte-h="svelte-me3ygp"><li><strong>GPU</strong>: NVIDIA, T4, L4 or A10G for good performance.</li> <li><strong>Instance Size</strong>: x1 (sufficient for most embedding workloads)</li> <li><strong>Auto-scaling</strong>: Enable scale-to-zero to save costs by switching the endpoint to a paused state when it’s not in use.</li> <li><strong>Timeout</strong>: Set a timeout of 10 minutes to avoid long-running requests. You should define a timeout based on how you expect your endpoint to be used.</li></ul> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-1a40r0f">If you’re looking for a model with less compute requirements, you can use the <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2" rel="nofollow">sentence-transformers/all-MiniLM-L6-v2</a> model.</p></div> <p data-svelte-h="svelte-qj37e3">The Qwen3-Embedding-4B model will automatically use the <strong>Text Embeddings Inference (TEI)</strong> engine, which provides optimized inference and automatic batching.</p> <p data-svelte-h="svelte-g3g7p9">Click “Create Endpoint” to deploy your embedding service.</p> <p data-svelte-h="svelte-14nerpn"><img src="https://raw.githubusercontent.com/huggingface/hf-endpoints-documentation/main/assets/quick_start/4-config.png" alt="config"></p> <p data-svelte-h="svelte-1ih12xw">This may take about 5 minutes to initialize.</p> <h2 class="relative group"><a id="test-your-endpoint" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#test-your-endpoint"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Test your Endpoint</span></h2> <p data-svelte-h="svelte-8cmlaj">Once your Inference Endpoint is running, you can test it directly in the playground. It accepts text input and returns high-dimensional vectors.</p> <p data-svelte-h="svelte-meccj2"><img src="https://raw.githubusercontent.com/huggingface/hf-endpoints-documentation/embedding-tutorial/assets/tutorials/embedding/playground.png" alt="playground"></p> <p data-svelte-h="svelte-eb5krh">Try entering some sample text like “Machine learning is transforming how we process data” and see the embedding output.</p> <h2 class="relative group"><a id="get-your-endpoints-details" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#get-your-endpoints-details"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Get your Endpoint’s details</span></h2> <p data-svelte-h="svelte-grmwad">To use your endpoint programmatically, you’ll need these details from the Endpoint’s <a href="https://endpoints.huggingface.co/" rel="nofollow">Overview</a>:</p> <ul data-svelte-h="svelte-1mxq0ya"><li><strong>Base URL</strong>: <code>https://<endpoint-name>.endpoints.huggingface.cloud/v1/</code></li> <li><strong>Model name</strong>: The name of your endpoint</li> <li><strong>Token</strong>: Your HF token from <a href="https://huggingface.co/settings/tokens" rel="nofollow">settings</a></li></ul> <p data-svelte-h="svelte-hjx1sa"><img src="https://raw.githubusercontent.com/huggingface/hf-endpoints-documentation/main/assets/tutorials/embedding/endpoint-page.png" alt="endpoint-details"></p> <h2 class="relative group"><a id="building-the-embedding-script" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#building-the-embedding-script"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Building the embedding script</span></h2> <p data-svelte-h="svelte-1jb2qdd">Now let’s build a script step by step to process datasets with embeddings. We’ll break it down into logical blocks.</p> <h3 class="relative group"><a id="step-1-set-up-dependencies-and-imports" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#step-1-set-up-dependencies-and-imports"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Step 1: Set up dependencies and imports</span></h3> <p data-svelte-h="svelte-1uq3z6b">We’ll use the OpenAI client to connect to the endpoint and the datasets library to load and process the dataset. So let’s install the required packages:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->pip install datasets openai<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-16rd7fj">Then, set up your imports in a new Python file:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> os | |
| <span class="hljs-keyword">from</span> datasets <span class="hljs-keyword">import</span> load_dataset | |
| <span class="hljs-keyword">from</span> openai <span class="hljs-keyword">import</span> OpenAI<!-- HTML_TAG_END --></pre></div> <h3 class="relative group"><a id="step-2-configure-the-connection" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#step-2-configure-the-connection"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Step 2: Configure the connection</span></h3> <p data-svelte-h="svelte-1rvx9zz">Set up the configuration to connect to your Inference Endpoint based on the details you collected in the previous step.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-comment"># Configuration</span> | |
| ENDPOINT_URL = <span class="hljs-string">"https://your-endpoint-name.endpoints.huggingface.cloud/v1/"</span> <span class="hljs-comment"># Endpoint URL + version</span> | |
| HF_TOKEN = os.getenv(<span class="hljs-string">"HF_TOKEN"</span>) <span class="hljs-comment"># Your Hugging Face Hub token from hf.co/settings/tokens</span> | |
| <span class="hljs-comment"># Initialize OpenAI client for your endpoint</span> | |
| client = OpenAI( | |
| base_url=ENDPOINT_URL, | |
| api_key=HF_TOKEN, | |
| )<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-6mzhuo">Your OpenAI client is now configured to connect to your Inference Endpoint. For further reading you can check out the client documentation on text embeddings <a href="https://platform.openai.com/docs/api-reference/embeddings" target="_blank" rel="noopener noreferrer">here</a>.</p> <h3 class="relative group"><a id="step-3-create-the-embedding-function" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#step-3-create-the-embedding-function"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Step 3: Create the embedding function</span></h3> <p data-svelte-h="svelte-103gfbk">Next, we’ll create a function to process batches of text and return embeddings.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">def</span> <span class="hljs-title function_">get_embeddings</span>(<span class="hljs-params">examples</span>): | |
| <span class="hljs-string">"""Get embeddings for a batch of texts."""</span> | |
| response = client.embeddings.create( | |
| model=<span class="hljs-string">"your-endpoint-name"</span>, <span class="hljs-comment"># Replace with your actual endpoint name</span> | |
| <span class="hljs-built_in">input</span>=examples[<span class="hljs-string">"context"</span>], <span class="hljs-comment"># In the squad dataset, the text is in the "context" column</span> | |
| ) | |
| <span class="hljs-comment"># Extract embeddings from response objects</span> | |
| embeddings = [sample.embedding <span class="hljs-keyword">for</span> sample <span class="hljs-keyword">in</span> response.data] | |
| <span class="hljs-keyword">return</span> {<span class="hljs-string">"embeddings"</span>: embeddings} <span class="hljs-comment"># datasets expects a dictionary with a key "embeddings" and a value of a list of embeddings</span><!-- HTML_TAG_END --></pre></div> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-ybk8oj">The <code>datasets</code> library will pass our function a batch of examples from the dataset, as a dictionary of batch values. The key will be the name of the column we want to embed, and the value will be a list of values from that column.</p></div> <h3 class="relative group"><a id="step-4-load-and-process-your-dataset" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#step-4-load-and-process-your-dataset"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Step 4: Load and process your dataset</span></h3> <p data-svelte-h="svelte-75q4le">Load your dataset and apply the embedding function:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-comment"># Load a sample dataset (you can replace this with your own)</span> | |
| dataset = load_dataset(<span class="hljs-string">"squad"</span>, split=<span class="hljs-string">"train[:100]"</span>) <span class="hljs-comment"># Using first 100 examples for demo</span> | |
| <span class="hljs-comment"># Process the dataset with embeddings</span> | |
| dataset_with_embeddings = dataset.<span class="hljs-built_in">map</span>( | |
| get_embeddings, | |
| batched=<span class="hljs-literal">True</span>, | |
| batch_size=<span class="hljs-number">10</span>, <span class="hljs-comment"># Process in small batches to avoid timeouts</span> | |
| desc=<span class="hljs-string">"Adding embeddings"</span>, | |
| )<!-- HTML_TAG_END --></pre></div> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-1sfo5v4">The <code>datasets</code> library’s <code>map</code> function is optimized for performance and will automatically batch the rows for us. Inference Endpoints can also scale to meet the demand of the batch size, so to get the best performance, you should calibrate the batch size with your Inference Endpoints’s configuration.</p> <p data-svelte-h="svelte-1u02m5o">For example, select the highest possible batch size for you model and synchronize the batch size with your Inference Endpoint’s configuration in <code>max_concurrent_requests</code>.</p></div> <h3 class="relative group"><a id="step-5-save-and-share-your-results" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#step-5-save-and-share-your-results"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Step 5: Save and Share your results</span></h3> <p data-svelte-h="svelte-1aql2k2">Finally, let’s save our embedded dataset locally or push it to the Hugging Face Hub:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-comment"># Save the processed dataset locally</span> | |
| dataset_with_embeddings.save_to_disk(<span class="hljs-string">"./embedded_dataset"</span>) | |
| <span class="hljs-comment"># Or push directly to Hugging Face Hub</span> | |
| dataset_with_embeddings.push_to_hub(<span class="hljs-string">"your-username/squad-embeddings"</span>)<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="next-steps" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#next-steps"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Next steps</span></h2> <p data-svelte-h="svelte-vzfm8t">Nice work! You’ve now built an embedding pipeline that can process any dataset. Here’s the complete script:</p> <details><summary data-svelte-h="svelte-1ri4alz">Click to view the complete script</summary> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> os | |
| <span class="hljs-keyword">from</span> datasets <span class="hljs-keyword">import</span> load_dataset | |
| <span class="hljs-keyword">from</span> dotenv <span class="hljs-keyword">import</span> load_dotenv | |
| <span class="hljs-keyword">from</span> openai <span class="hljs-keyword">import</span> OpenAI | |
| load_dotenv() | |
| <span class="hljs-comment"># Configuration</span> | |
| ENDPOINT_URL = <span class="hljs-string">"https://your-endpoint-name.endpoints.huggingface.cloud/v1/"</span> | |
| HF_TOKEN = os.getenv(<span class="hljs-string">"HF_TOKEN"</span>) | |
| <span class="hljs-comment"># Initialize OpenAI client for your endpoint</span> | |
| client = OpenAI( | |
| base_url=ENDPOINT_URL, | |
| api_key=HF_TOKEN, | |
| ) | |
| <span class="hljs-keyword">def</span> <span class="hljs-title function_">get_embeddings</span>(<span class="hljs-params">examples</span>): | |
| <span class="hljs-string">"""Get embeddings for a batch of texts."""</span> | |
| response = client.embeddings.create( | |
| model=<span class="hljs-string">"your-endpoint-name"</span>, <span class="hljs-comment"># Replace with your actual endpoint name</span> | |
| <span class="hljs-built_in">input</span>=examples[<span class="hljs-string">"context"</span>], | |
| ) | |
| <span class="hljs-comment"># Extract embeddings from response</span> | |
| embeddings = [sample.embedding <span class="hljs-keyword">for</span> sample <span class="hljs-keyword">in</span> response.data] | |
| <span class="hljs-keyword">return</span> {<span class="hljs-string">"embeddings"</span>: embeddings} | |
| <span class="hljs-comment"># Load a sample dataset (you can replace this with your own)</span> | |
| <span class="hljs-built_in">print</span>(<span class="hljs-string">"Loading dataset..."</span>) | |
| dataset = load_dataset(<span class="hljs-string">"squad"</span>, split=<span class="hljs-string">"train[:1000]"</span>) <span class="hljs-comment"># Using first 1000 examples for demo</span> | |
| <span class="hljs-comment"># Process the dataset with embeddings</span> | |
| <span class="hljs-built_in">print</span>(<span class="hljs-string">"Processing dataset with embeddings..."</span>) | |
| dataset_with_embeddings = dataset.<span class="hljs-built_in">map</span>( | |
| get_embeddings, | |
| batched=<span class="hljs-literal">True</span>, | |
| batch_size=<span class="hljs-number">10</span>, <span class="hljs-comment"># Process in small batches to avoid timeouts</span> | |
| desc=<span class="hljs-string">"Adding embeddings"</span>, | |
| ) | |
| <span class="hljs-comment"># Save the processed dataset locally</span> | |
| <span class="hljs-built_in">print</span>(<span class="hljs-string">"Saving processed dataset..."</span>) | |
| dataset_with_embeddings.save_to_disk(<span class="hljs-string">"./embedded_dataset"</span>) | |
| <span class="hljs-comment"># Or push directly to Hugging Face Hub</span> | |
| <span class="hljs-built_in">print</span>(<span class="hljs-string">"Pushing to Hugging Face Hub..."</span>) | |
| dataset_with_embeddings.push_to_hub(<span class="hljs-string">"your-username/squad-embeddings"</span>) | |
| <span class="hljs-built_in">print</span>(<span class="hljs-string">"Dataset processing complete!"</span>)<!-- HTML_TAG_END --></pre></div></details> <p data-svelte-h="svelte-1kuxjv9">Here are some ways to extend your script:</p> <ul data-svelte-h="svelte-1sd2to"><li><strong>Process multiple datasets</strong>: Modify the script to handle different dataset sources</li> <li><strong>Add error handling</strong>: Implement retry logic for failed API calls</li> <li><strong>Optimize batch sizes</strong>: Experiment with different batch sizes for better performance</li> <li><strong>Add validation</strong>: Check embedding quality and dimensions</li> <li><strong>Custom preprocessing</strong>: Add text cleaning or normalization steps</li> <li><strong>Build a Semantic Search Application</strong>: Use the embeddings to build a semantic search application.</li></ul> <p data-svelte-h="svelte-vw83lh">Your embedded datasets are now ready for downstream tasks like semantic search, recommendation systems, or RAG applications!</p> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/hf-endpoints-documentation/blob/main/docs/source/tutorials/embedding.mdx" target="_blank"><span data-svelte-h="svelte-1kd6by1"><</span> <span data-svelte-h="svelte-x0xyl0">></span> <span data-svelte-h="svelte-1dajgef"><span class="underline ml-1.5">Update</span> on GitHub</span></a> <p></p> | |
| <script> | |
| { | |
| __sveltekit_1q0n26o = { | |
| assets: "/docs/inference-endpoints/pr_136/en", | |
| base: "/docs/inference-endpoints/pr_136/en", | |
| env: {} | |
| }; | |
| const element = document.currentScript.parentElement; | |
| const data = [null,null]; | |
| Promise.all([ | |
| import("/docs/inference-endpoints/pr_136/en/_app/immutable/entry/start.fb9ab4d6.js"), | |
| import("/docs/inference-endpoints/pr_136/en/_app/immutable/entry/app.6247727a.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 23], | |
| data, | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 42.4 kB
- Xet hash:
- c71614001bc225a791e0cddcd4cc9c0170b0a6f39bf4166bcc695f0ecf15404f
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.