transformers/main/en/philosophy.html

<meta charset="utf-8" /><meta name="hf:doc:metadata" content="{&quot;title&quot;:&quot;Philosophy&quot;,&quot;local&quot;:&quot;philosophy&quot;,&quot;sections&quot;:[{&quot;title&quot;:&quot;Main concepts&quot;,&quot;local&quot;:&quot;main-concepts&quot;,&quot;sections&quot;:[],&quot;depth&quot;:2}],&quot;depth&quot;:1}">
		<link href="/docs/transformers/main/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload">
		<link rel="modulepreload" href="/docs/transformers/main/en/_app/immutable/entry/start.2135b7e6.js">
		<link rel="modulepreload" href="/docs/transformers/main/en/_app/immutable/chunks/scheduler.25b97de1.js">
		<link rel="modulepreload" href="/docs/transformers/main/en/_app/immutable/chunks/singletons.0f2b7d5f.js">
		<link rel="modulepreload" href="/docs/transformers/main/en/_app/immutable/chunks/index.e188933d.js">
		<link rel="modulepreload" href="/docs/transformers/main/en/_app/immutable/chunks/paths.3d04d2c6.js">
		<link rel="modulepreload" href="/docs/transformers/main/en/_app/immutable/entry/app.24372c84.js">
		<link rel="modulepreload" href="/docs/transformers/main/en/_app/immutable/chunks/index.d9030fc9.js">
		<link rel="modulepreload" href="/docs/transformers/main/en/_app/immutable/nodes/0.026d2fdd.js">
		<link rel="modulepreload" href="/docs/transformers/main/en/_app/immutable/chunks/each.e59479a4.js">
		<link rel="modulepreload" href="/docs/transformers/main/en/_app/immutable/nodes/377.7b99b1ac.js">
		<link rel="modulepreload" href="/docs/transformers/main/en/_app/immutable/chunks/EditOnGithub.91d95064.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{&quot;title&quot;:&quot;Philosophy&quot;,&quot;local&quot;:&quot;philosophy&quot;,&quot;sections&quot;:[{&quot;title&quot;:&quot;Main concepts&quot;,&quot;local&quot;:&quot;main-concepts&quot;,&quot;sections&quot;:[],&quot;depth&quot;:2}],&quot;depth&quot;:1}"><!-- HEAD_svelte-u9bgzb_END -->      <p></p>   <h1 class="relative group"><a id="philosophy" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#philosophy"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Philosophy</span></h1> <p data-svelte-h="svelte-1p1j4j2">🤗 Transformers is an opinionated library built for:</p> <ul data-svelte-h="svelte-15p34re"><li>machine learning researchers and educators seeking to use, study or extend large-scale Transformers models.</li> <li>hands-on practitioners who want to fine-tune those models or serve them in production, or both.</li> <li>engineers who just want to download a pretrained model and use it to solve a given machine learning task.</li></ul> <p data-svelte-h="svelte-l3cofq">The library was designed with two strong goals in mind:</p> <ol data-svelte-h="svelte-a039tw"><li>Be as easy and fast to use as possible:</li></ol> <ul data-svelte-h="svelte-ktebwu"><li>We strongly limited the number of user-facing abstractions to learn, in fact, there are almost no abstractions,
just three standard classes required to use each model: <a href="main_classes/configuration">configuration</a>,
<a href="main_classes/model">models</a>, and a preprocessing class (<a href="main_classes/tokenizer">tokenizer</a> for NLP, <a href="main_classes/image_processor">image processor</a> for vision, <a href="main_classes/feature_extractor">feature extractor</a> for audio, and <a href="main_classes/processors">processor</a> for multimodal inputs).</li> <li>All of these classes can be initialized in a simple and unified way from pretrained instances by using a common
<code>from_pretrained()</code> method which downloads (if needed), caches and
loads the related class instance and associated data (configurations’ hyperparameters, tokenizers’ vocabulary,
and models’ weights) from a pretrained checkpoint provided on <a href="https://huggingface.co/models" rel="nofollow">Hugging Face Hub</a> or your own saved checkpoint.</li> <li>On top of those three base classes, the library provides two APIs: <a href="/docs/transformers/main/en/main_classes/pipelines#transformers.pipeline">pipeline()</a> for quickly
using a model for inference on a given task and <a href="/docs/transformers/main/en/main_classes/trainer#transformers.Trainer">Trainer</a> to quickly train or fine-tune a PyTorch model (all TensorFlow models are compatible with <code>Keras.fit</code>).</li> <li>As a consequence, this library is NOT a modular toolbox of building blocks for neural nets. If you want to
extend or build upon the library, just use regular Python, PyTorch, TensorFlow, Keras modules and inherit from the base
classes of the library to reuse functionalities like model loading and saving. If you’d like to learn more about our coding philosophy for models, check out our <a href="https://huggingface.co/blog/transformers-design-philosophy" rel="nofollow">Repeat Yourself</a> blog post.</li></ul> <ol start="2" data-svelte-h="svelte-1j3ty0h"><li>Provide state-of-the-art models with performances as close as possible to the original models:</li></ol> <ul data-svelte-h="svelte-eaof5h"><li>We provide at least one example for each architecture which reproduces a result provided by the official authors
of said architecture.</li> <li>The code is usually as close to the original code base as possible which means some PyTorch code may be not as
<em>pytorchic</em> as it could be as a result of being converted TensorFlow code and vice versa.</li></ul> <p data-svelte-h="svelte-16fl7yl">A few other goals:</p> <ul data-svelte-h="svelte-r5ixu4"><li><p>Expose the models’ internals as consistently as possible:</p> <ul><li>We give access, using a single API, to the full hidden-states and attention weights.</li> <li>The preprocessing classes and base model APIs are standardized to easily switch between models.</li></ul></li> <li><p>Incorporate a subjective selection of promising tools for fine-tuning and investigating these models:</p> <ul><li>A simple and consistent way to add new tokens to the vocabulary and embeddings for fine-tuning.</li> <li>Simple ways to mask and prune Transformer heads.</li></ul></li> <li><p>Easily switch between PyTorch, TensorFlow 2.0 and Flax, allowing training with one framework and inference with another.</p></li></ul>  <h2 class="relative group"><a id="main-concepts" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#main-concepts"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Main concepts</span></h2> <p data-svelte-h="svelte-60ebt4">The library is built around three types of classes for each model:</p> <ul data-svelte-h="svelte-316anx"><li><strong>Model classes</strong> can be PyTorch models (<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Module" rel="nofollow">torch.nn.Module</a>), Keras models (<a href="https://www.tensorflow.org/api_docs/python/tf/keras/Model" rel="nofollow">tf.keras.Model</a>) or JAX/Flax models (<a href="https://flax.readthedocs.io/en/latest/api_reference/flax.linen/module.html" rel="nofollow">flax.linen.Module</a>) that work with the pretrained weights provided in the library.</li> <li><strong>Configuration classes</strong> store the hyperparameters required to build a model (such as the number of layers and hidden size). You don’t always need to instantiate these yourself. In particular, if you are using a pretrained model without any modification, creating the model will automatically take care of instantiating the configuration (which is part of the model).</li> <li><strong>Preprocessing classes</strong> convert the raw data into a format accepted by the model. A <a href="main_classes/tokenizer">tokenizer</a> stores the vocabulary for each model and provide methods for encoding and decoding strings in a list of token embedding indices to be fed to a model. <a href="main_classes/image_processor">Image processors</a> preprocess vision inputs, <a href="main_classes/feature_extractor">feature extractors</a> preprocess audio inputs, and a <a href="main_classes/processors">processor</a> handles multimodal inputs.</li></ul> <p data-svelte-h="svelte-ff8m9n">All these classes can be instantiated from pretrained instances, saved locally, and shared on the Hub with three methods:</p> <ul data-svelte-h="svelte-1p0er3n"><li><code>from_pretrained()</code> lets you instantiate a model, configuration, and preprocessing class from a pretrained version either
provided by the library itself (the supported models can be found on the <a href="https://huggingface.co/models" rel="nofollow">Model Hub</a>) or
stored locally (or on a server) by the user.</li> <li><code>save_pretrained()</code> lets you save a model, configuration, and preprocessing class locally so that it can be reloaded using
<code>from_pretrained()</code>.</li> <li><code>push_to_hub()</code> lets you share a model, configuration, and a preprocessing class to the Hub, so it is easily accessible to everyone.</li></ul> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/transformers/blob/main/docs/source/en/philosophy.md" target="_blank"><span data-svelte-h="svelte-1kd6by1">&lt;</span> <span data-svelte-h="svelte-x0xyl0">&gt;</span> <span data-svelte-h="svelte-1dajgef"><span class="underline ml-1.5">Update</span> on GitHub</span></a>  <p></p> 
			
			<script>
				{
					__sveltekit_1xexzbk = {
						assets: "/docs/transformers/main/en",
						base: "/docs/transformers/main/en",
						env: {}
					};

					const element = document.currentScript.parentElement;

					const data = [null,null];

					Promise.all([
						import("/docs/transformers/main/en/_app/immutable/entry/start.2135b7e6.js"),
						import("/docs/transformers/main/en/_app/immutable/entry/app.24372c84.js")
					]).then(([kit, app]) => {
						kit.start(app, element, {
							node_ids: [0, 377],
							data,
							form: null,
							error: null
						});
					});
				}
			</script>