|
|
| <!doctype html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/> |
|
|
| <title>Training Transformers Together</title> |
| <meta name="description" content="A NeurIPS'21 demonstration that explains how to train large models together with multiple collaborators."> |
| <link rel="mask-icon" href="https://learning-at-home.github.io/logo_small.png"> |
| <link rel="alternate icon" class="js-site-favicon" type="image/png" href="https://learning-at-home.github.io/logo.png"> |
| <link rel="icon" class="js-site-favicon" type="image/png" href="https://learning-at-home.github.io/logo.png"> |
| <meta property="og:url" content="https://training-transformers-together.github.io"> |
| <meta property="og:site_name" content="Training Transformers Together"> |
| <meta property="og:title" content="Train vast neural networks together"> |
| <meta property="og:description" content="A NeurIPS'21 demonstration that explains how to train large models together with multiple collaborators."> |
| <meta property="og:image" content="https://learning-at-home.github.io/logo_small.png"> |
| <meta property="og:image:type" content="image/png"> |
| <meta property="og:image:width" content="96"> |
| <meta property="og:image:height" content="96"> |
| <meta property="twitter:site" content="https://training-transformers-together.github.io"> |
| <meta property="twitter:creator" content="Yandex, Hugging Face, University of Washington, Hivemind team & contributors"> |
| <meta property="twitter:card" content="summary_large_image"> |
| <meta property="twitter:title" content="Training Transformers Together"> |
| <meta property="twitter:description" content="A NeurIPS'21 demonstration that explains how to train large models together with multiple collaborators."> |
| <meta property="twitter:image:src" content="https://learning-at-home.github.io/logo_horizontal.png"> |
| <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> |
|
|
| |
| <link href="https://bootswatch.com/5/flatly/bootstrap.css" rel="stylesheet"> |
|
|
| |
| <link href="./style.css" rel="stylesheet"> |
| </head> |
|
|
| <body> |
| <div id="header_main" style="display: block;" class="mb-0 pb-0"> |
| <canvas></canvas> |
| <div id="overlay"> |
| <div id="header_window"> |
| <div id="header"> |
| <img src="https://learning-at-home.github.io/logo.png" id="bug-logo" |
| style="width: 40%; max-height: 320px; max-width: 320px; z-index:1000; position: relative;"> |
| <br> |
| <h1 class="faded title title_elem mb-1 pb-1" style="margin-top:-25px; margin-bottom:-10px"> |
| <p style="margin-top: 0px; font-weight:bolder; margin-bottom:0px;"> |
| <span id="title_text">Training Transformers Together</span> |
| </p> |
| <p style="font-size: 18px; margin-top:0px; margin-bottom:5px;"> |
| large-scale deep learning for everyone, by everyone</p> |
| <p style="font-size: 18px; font-weight:lighter; margin-top:0px; margin-bottom:0px;"> |
| A NeurIPS 2021 Demonstration</p> |
| </h1> |
| </div> |
| </div> |
| </div> |
| </div> |
| <script src="./header-animate.js"></script> |
|
|
| <div class="container d-flex justify-content-center mb-2 pb-2" style="max-width: 500px"> |
| <div class="row text-center align-items-center justify-content-center"> |
| <div class="col-3"> |
| <a href="https://research.yandex.com/"> |
| <img src="logos/yandex.png" class="img-fluid center-block" style="max-width: 66%" alt="Yandex Research"> |
| </a> |
| </div> |
| <div class="col-3 px-2"> |
| <a href="https://huggingface.co/"> |
| <img src="logos/huggingface.png" class="img-fluid center-block" style="max-width: 66%" alt="Hugging Face"> |
| </a> |
| </div> |
| <div class="col-3 px-3"> |
| <a href="https://www.hse.ru/en/"> |
| <img src="logos/hse.png" class="img-fluid center-block" style="max-width: 66%" alt="HSE University"> |
| </a> |
| </div> |
| <div class="col-3 px-2"> |
| <a href="http://www.washington.edu/"> |
| <img src="logos/uwash.png" class="img-fluid center-block" alt="University of Washington"> |
| </a> |
| </div> |
| </div> |
| </div> |
|
|
| <div class="container" style="display: block;"> |
| <p> |
| There was a time when you could comfortably train state-of-the-art vision and language models at home on your workstation. |
| The first convolutional neural net to beat ImageNet |
| (<a target="_blank" href="https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf">AlexNet</a>) |
| was trained for 5-6 days on two gamer-grade GPUs. In contrast, today's Top-1 ImageNet model |
| (<a target="_blank" href="https://arxiv.org/abs/2106.04803">CoAtNet</a>) |
| takes 20,000 TPU-v3 days. And things are even worse in the NLP world: training |
| <a target="_blank" href="https://arxiv.org/abs/2005.14165">GPT‑3</a> |
| on a top-tier server with 8x A100 would take decades. |
| </p> |
| <p> |
| So, can individual researchers and small labs still train state-of-the-art models? Yes we can! |
| All it takes is for a bunch of us to come together. In fact, we're doing it right now and <b>you are invited to join!</b> |
| </p> |
| <iframe id="iframe_main" src="https://hf.space/streamlitiframe/training-transformers-together/dashboard-embedded/+" |
| data-src="https://hf.space/streamlitiframe/training-transformers-together/dashboard-embedded/+" |
| data-sdk="streamlit" |
| title="Streamlit app" class="container p-0 flex-grow space-iframe" |
| allow="accelerometer; ambient-light-sensor; autoplay; battery; camera; document-domain; encrypted-media; fullscreen; geolocation; gyroscope; layout-animations; legacy-image-formats; magnetometer; microphone; midi; oversized-images; payment; picture-in-picture; publickey-credentials-get; sync-xhr; usb; vr ; wake-lock; xr-spatial-tracking" |
| sandbox="allow-forms allow-modals allow-popups allow-popups-to-escape-sandbox allow-same-origin allow-scripts allow-downloads" |
| style="top:-200px; left:0; bottom:0; right:0; width:100%; height:200px; border:none; margin:0; padding:0; z-index:999999;" scrolling=no> |
| <p>This was meant to be an IFrame, but your browser did not display it.</p> |
| <p>Please go to <a href="https://huggingface.co/spaces/training-transformers-together/demo">https://huggingface.co/spaces/training-transformers-together/demo</a>.</p> |
| </iframe> |
| <p> |
| In this demo, we train a model similar to <a target="_blank" href="https://openai.com/blog/dall-e/">OpenAI DALL-E</a> — |
| a Transformer model that generates images from text descriptions. |
| It is trained on <a target="_blank" href="https://laion.ai/laion-400-open-dataset/">LAION-400M</a>, |
| the world's largest openly available image-text-pair dataset with 400 million samples. Our model is based on |
| the <a target="_blank" href="https://github.com/lucidrains/DALLE-pytorch">dalle‑pytorch</a> implementation |
| by <a target="_blank" href="https://github.com/lucidrains">Phil Wang</a> with a few tweaks to make it communication-efficient. |
| </p> |
| <div class="accordion" id="accordionExample"> |
| <div class="accordion-item"> |
| <h2 class="accordion-header" id="headingOne"> |
| <button class="accordion-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#collapseOne" aria-expanded="false" aria-controls="collapseOne"> |
| How to train efficiently over the Internet? |
| </button> |
| </h2> |
| <div id="collapseOne" class="accordion-collapse collapse" aria-labelledby="headingOne" data-bs-parent="#accordionExample"> |
| <div class="accordion-body"> |
| <p> |
| Modern distributed training algorithms are designed for HPC clusters with a 10-100 gigabit per second bandwidth. |
| In turn, a typical Internet connection runs at 10-100 megabits per second: that’s three orders of magnitude slower. |
| To make distributed training efficient, you need to win back these three orders of magnitude. |
| This may seem daunting at first, but in reality, DL researchers have already made all the necessary pieces for solving this puzzle: |
| </p> |
| <table class="table table-hover"> |
| <thead> |
| <tr> |
| <th scope="col">Speed‑up</th> |
| <th scope="col">How to achieve</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr><td class="centered"><strong>4-16x</strong></td><td> |
| <strong>Large-batch training:</strong> <a target="_blank" rel="noopener noreferrer" href="https://arxiv.org/abs/1904.00962">You et al. (2019)</a> proposed a way for training neural networks efficiently with larger batches, and hence, fewer communication rounds. |
| </td></tr> |
| <tr><td class="centered"><strong>4-32x</strong></td><td> |
| <strong>Gradient compression:</strong> from simple <a target="_blank" rel="noopener noreferrer" href="https://arxiv.org/abs/1511.04561">8-bit quantization</a> |
| to advanced techniques such as <a target="_blank" rel="noopener noreferrer" href="https://arxiv.org/abs/1712.01887">Deep Gradient Compression</a>, |
| <a target="_blank" rel="noopener noreferrer" href="https://arxiv.org/abs/1905.13727">PowerSGD</a>, <a target="_blank" rel="noopener noreferrer" href="https://arxiv.org/abs/2102.02888">1-bit Adam</a>, |
| and many others. As a rule of thumb, these techniques can safely reduce communication by 16-32x. More extreme compression is often |
| possible, but it may affect stability or final quality. |
| </td></tr> |
| <tr><td class="centered"><strong>4-24x</strong></td><td> |
| <strong>Parameter sharing:</strong> reusing parameters between model layers results in a model with fewer parameters, |
| and hence, fewer gradients to communicate. <a target="_blank" rel="noopener noreferrer" href="https://arxiv.org/abs/1909.11942">Lan et al. (2019)</a> and |
| <a target="_blank" rel="noopener noreferrer" href="https://arxiv.org/pdf/2107.11817.pdf">Xue et al. (2021)</a> propose efficient parameter sharing architectures |
| for NLP and computer vision. |
| </td></tr> |
| <tr><td class="centered"><strong>1.5-2x</strong></td><td> |
| <strong>Overlapping computation with communication:</strong> running network communication in background while |
| computing the next portion of gradients. This is a <a target="_blank" rel="noopener noreferrer" href="https://ur.booksc.eu/book/1624068/2d0506">long-standing trick from HPC</a> |
| that was recently adapted for DL training. <a target="_blank" rel="noopener noreferrer" href="https://arxiv.org/abs/2101.06840">Ren et al. (2021)</a> show that |
| updating parameters in background while computing the next batch of gradients does not harm convergence. |
| </td></tr> |
| </tbody> |
| </table> |
| <p> |
| These techniques are already more than enough to cover 1000x slower communication. |
| This means that in practice you can pick and choose which of them you want in your training run. |
| For this demo, we use 8x larger batches, 4x compression, 12x parameter sharing and partial overlapping. |
| If you don’t want parameter sharing, you can trade it for more advanced gradient compression or larger batches. |
| </p> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="accordion" id="accordionAnother" style="margin-top: 10px;"> |
| <div class="accordion-item"> |
| <h2 class="accordion-header" id="headingTwo"> |
| <button class="accordion-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#collapseTwo" aria-expanded="false" aria-controls="collapseOne"> |
| How to train with different device types? |
| </button> |
| </h2> |
| <div id="collapseTwo" class="accordion-collapse collapse" aria-labelledby="headingTwo" data-bs-parent="#accordionAnother"> |
| <div class="accordion-body"> |
| <p> |
| Most distributed DL frameworks assume that the computation is performed by a fleet of identical devices, |
| typically GPU servers or TPU cores. Under this assumption, each device can be assigned an equal part of |
| computation, such as processing a fixed batch size of training samples. |
| However, this quickly breaks down if workers use different device types. If one participant uses a GPU (e.g. P100) |
| and another runs on TPU-v2-8, it is difficult to find a regime where both devices will be fully utilized. |
| </p> |
| <p> |
| To make the best use of all available devices, we let each device accumulate gradients at its own pace |
| with individually tuned batch size and some other features (e.g. gradient checkpointing or using XLA). |
| Once workers collectively aggregate some predefined global batch size, they average their gradients |
| with weights proportional to each worker's individual contribution (i.e. number of samples processed). |
| </p> |
| <a class="block overflow-hidden"> |
| <div class="w-full h-40 mb-2 bg-gray-900 group-hover:bg-gray-850 rounded-lg flex items-start justify-start overflow-hidden"> |
| <iframe src="https://www.youtube.com/embed/zdVsg5zsGdc" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0" |
| style="width: 100%; height: 240px"></iframe> |
| </div> |
| </a> |
| <p> |
| This technique allows the "swarm" to automatically adjust its behavior as peers join, leave or fail. |
| For instance, if several high-performance peers join the experiment, other peers will need to process a smaller |
| number of samples per optimizer step, and hence, the collaboration will train faster with the same hyperparameters. |
| In turn, if one of the workers fails and loses its progress (e.g. due to a fp16 overflow), others will make |
| up for that by processing slightly more. For more details on how this works, please refer to |
| <a target="_blank" rel="noopener noreferrer" href="https://papers.nips.cc/paper/2021/hash/41a60377ba920919939d83326ebee5a1-Abstract.html"> |
| "Deep Learning In Open Collaborations"</a> paper or the corresponding <a target="_blank" rel="noopener noreferrer" href="https://huggingface.co/blog/collaborative-training">blog post</a>. |
| </p> |
| </div> |
| </div> |
| </div> |
| </div> |
|
|
|
|
| <h3 class="my-4">How do I join?</h3> |
|
|
| <p>This section will be updated <strong>on December 7</strong>.</p> |
|
|
| <h3 class="my-4">Practical aspects</h3> |
|
|
| <div class="border-bottom pb-3"> |
| |
| <ul class="nav nav-tabs m-3"> |
| <li class="nav-item"> |
| <a class="nav-link active" data-bs-toggle="tab" href="#memory-efficiency">Memory-Efficient Training</a> |
| </li> |
| <li class="nav-item"> |
| <a class="nav-link" data-bs-toggle="tab" href="#security">Security</a> |
| </li> |
| <li class="nav-item"> |
| <a class="nav-link" data-bs-toggle="tab" href="#make-your-own">Make Your Own</a> |
| </li> |
| </ul> |
|
|
| |
| <div class="tab-content"> |
| <div class="tab-pane fade active show" id="memory-efficiency"> |
| <p> |
| Our aim is to train a large model in a decentralized fashion on consumer hardware or low-end cloud instances. |
| This means we need to make the model, dataset, and other memory buffers fit onto a few GB of disk, 12-16 GB of CPU RAM, |
| and 8-12 GB of GPU memory. Unfortunately, this rules out many popular techniques such as |
| <a target="_blank" rel="noopener noreferrer" href="https://arxiv.org/abs/2101.06840">ZeRO-Offload</a>: |
| there is simply not enough RAM for that. Instead, we must make better use of what limited memory we have. |
| To do this, we use two techniques: 8-bit Optimizers for GPU memory and dataset streaming for RAM & HDD. |
| </p> |
| <p> |
| <b>8-bit optimizers:</b> |
| Using optimizers such as LAMB or Adam requires four times as much GPU memory as simply storing model parameters (8 bytes vs 2 bytes) |
| because of additional gradient statistics. |
| As such, for training large models with many parameters, the optimizer state takes the largest amount of memory. |
| With 8-bit optimizers, this amount is reduced by 75% (2 bytes), making it much easier to fit large models onto consumer GPUs. |
| </p><p> |
| Naturally, we can combine this technique with offloading and store 8-bit optimizer states in the CPU memory rather |
| than in the GPU memory (0 bytes GPU, 2 bytes CPU). To perform an optimizer update, we transfer the GPU gradients |
| to the CPU, update the model parameters, and then copy the new weights to the GPU. |
| We can do this for each weight one-by-one so that the additional CPU memory required for the |
| optimizer update is minimal. |
| This combination of offloading and 8-bit optimizers means that we conserve GPU memory (0 bytes per parameter) |
| and also use only a limited amount of CPU memory (2 bytes per parameter). |
|
|
| </p> |
| <p> |
| <b>Dataset streaming:</b> |
| Usually data is stored on disk and needs to be fully or partially loaded into RAM for training. |
| Large datasets used for pretraining measure in <a target="_blank" rel="noopener noreferrer" href="https://arxiv.org/abs/2101.00027">hundreds of gigabytes</a> |
| or even <a target="_blank" rel="noopener noreferrer" href="https://laion.ai/laion-400-open-dataset/">terabytes</a>. |
| This can pose a significant problem, as most desktop and cheap cloud instances simply do not have that much free space. |
| Furthermore, downloading the data over the Internet would take up hours before one can even begin training. |
| </p> |
| <center> |
| <img src="./logos/stream.gif" id="stream" |
| style="width: 80%; max-height: 200px; max-width: 640px; z-index:1000; top:-10px; position: relative;"> |
| </center> |
| <p> |
| To circumvent these problems, it is possible to stream the data in the same way as you stream online videos. |
| Participants download a small random portion of the training dataset and immediately begin training on it, |
| while additional data is loaded in the background. As such, we can train a model with virtually no storage |
| overhead from the dataset, and switching to a new dataset is as simple as changing an argument of the dataset class. |
| </p> |
| <h5><b>Here's our tutorial covering these methods:</b> |
| <a target="_blank" rel="noopener noreferrer" href="https://colab.research.google.com/gist/justheuristic/75f6a2a731f05a213a55cd2c8a458aaf/fine-tune-a-language-model-with-dataset-streaming-and-8-bit-optimizers.ipynb"> |
| <span> |
| <img src="https://colab.research.google.com/assets/colab-badge.svg" width="150px"> |
| </span> |
| </a></h5> |
|
|
| </div> |
| <div class="tab-pane fade" id="security"> |
| <p>In this section, we discuss common concerns related to security of collaborative training:</p> |
|
|
| <p> |
| <b>Q: If I join a collaborative experiment, do I allow other people to execute code on my computer?</b> |
| </p> |
|
|
| <p> |
| <b>A:</b> During the training, participants only exchange data (gradients, statistics, model weights) and never send code to each other. |
| No other peer can execute arbitrary code on your computer. |
| </p> |
|
|
| <p> |
| To join the experiment, you typically need to run the code (implementing the model, data streaming, training loop, etc.) |
| from a repository or a Colab notebook provided by the authors of the experiment. |
| This is no different from running any other open source project/Colab notebook. |
| </p> |
|
|
| <p> |
| <b>Q: Can a malicious participant influence the training outcome?</b> |
| </p> |
|
|
| <p> |
| <b>A:</b> It is indeed possible unless we use some defense mechanisms. |
| For instance, a malicious participant can damage model weights by sending large numbers instead of correct gradients. |
| The same can happen due to broken hardware or misconfiguration. |
| </p> |
|
|
| <ul> |
| <li> |
| <p> |
| One possible defense is using <b>authentication</b> combined with <b>model checkpointing</b>. |
| In this case, participants should log in (e.g. with their Hugging Face account) to interact with the rest of the collaboration. |
| In turn, moderators can screen potential participants and add them to an allowlist. |
| If something goes wrong (e.g. a participant sends invalid gradients and the model diverges), |
| the moderators remove them from the list and revert the model to the latest checkpoint unaffected by the attack. |
| </p> |
|
|
| |
|
|
| <p> |
| Nice bonus: using this data, the moderators can acknowledge the personal contribution of each participant. |
| </p> |
| </li> |
| <li> |
| <p> |
| Another defense is replacing the naive averaging of the peers' gradients with an <b>aggregation technique that is robust to outliers</b>. |
| <a target="_blank" rel="noopener noreferrer" href="https://arxiv.org/abs/2012.10333">Karimireddy et al. (2020)</a> |
| suggested such a technique (named CenteredClip) and proved that it does not significantly affect the model's convergence. |
| </p> |
|
|
| |
|
|
| <p> |
| In our case, CenteredClip is useful but not enough to protect from malicious participants, |
| since it implies that the CenteredClip procedure itself is performed by a trusted server. |
| By contrast, in our decentralized system, all participants can aggregate a part of the gradients, |
| and we cannot assume any of them to be trusted. |
| </p> |
|
|
| <p> |
| Recently, <a target="_blank" rel="noopener noreferrer" href="https://arxiv.org/abs/2106.11257">Gorbunov et al. (2021)</a> |
| proposed a robust aggregation protocol for decentralized systems that does not require this assumption. |
| This protocol uses CenteredClip as a subroutine but is able to detect and ban participants who performed it incorrectly. |
| </p> |
| </li> |
| </ul> |
| </div> |
| <div class="tab-pane fade" id="make-your-own"> |
| <p>In this section, we provide a recipe for you to run a collaborative training experiment yourself.</p> |
| <p> |
| <b>Got confused?</b> Feel free to ask any questions in our <a target="_blank" rel="noopener noreferrer" href="https://discord.gg/uGugx9zYvN">Discord</a>! |
| </p> |
| <ol> |
| <li class="mb-2"> |
| Set up dataset streaming: |
| <ul> |
| <li> |
| <a target="_blank" rel="noopener noreferrer" href="https://huggingface.co/docs/datasets/share_dataset.html">Upload</a> your dataset to the Hugging Face Hub |
| in a streaming-friendly format (<a target="_blank" rel="noopener noreferrer" href="https://huggingface.co/datasets/laion/laion_100m_vqgan_f8">example</a>). |
| </li> |
| <li>Set up dataset streaming (see the "Memory-Efficient Training" section).</li> |
| </ul> |
| </li> |
| <li class="mb-2"> |
| Write the code of training peers (<a target="_blank" rel="noopener noreferrer" href="https://github.com/learning-at-home/dalle-hivemind/blob/main/run_trainer.py">example</a>): |
| <ul> |
| <li>Implement your model, set up dataset streaming, and write the training loop.</li> |
| <li> |
| Get familiar with the <a href="https://github.com/learning-at-home/hivemind">hivemind</a> library |
| (<a target="_blank" rel="noopener noreferrer" href="https://learning-at-home.readthedocs.io/en/latest/user/quickstart.html">quickstart</a>). |
| </li> |
| <li> |
| In the training loop, wrap up your PyTorch optimizer with |
| <a target="_blank" rel="noopener noreferrer" href="https://learning-at-home.readthedocs.io/en/latest/modules/optim.html#hivemind.optim.experimental.optimizer.Optimizer">hivemind.Optimizer</a> |
| (<a target="_blank" rel="noopener noreferrer" href="https://github.com/learning-at-home/dalle-hivemind/blob/main/task.py#L121">example</a>). |
| </li> |
| </ul> |
| </li> |
| <li class="mb-2"> |
| <b>(optional)</b> Write the code of auxiliary peers (<a target="_blank" rel="noopener noreferrer" href="https://github.com/learning-at-home/dalle-hivemind/blob/main/run_aux_peer.py">example</a>): |
| <ul> |
| <li> |
| Auxiliary peers are a special kind of peers responsible for |
| logging experiment progress (e.g., to <a target="_blank" rel="noopener noreferrer" href="https://wandb.ai/">Weights & Biases</a>) |
| and uploading model checkpoints (e.g., to <a target="_blank" rel="noopener noreferrer" href="https://huggingface.co/docs/transformers/model_sharing">Hugging Face Hub</a>). |
| </li> |
| <li> |
| Such peers don't need to calculate gradients and may be launched on cheap machines without GPUs. |
| </li> |
| <li> |
| They can serve as a convenient entry point to |
| <a href="https://learning-at-home.readthedocs.io/en/latest/modules/dht.html">hivemind.DHT</a> |
| (i.e., their address can be specified as <code>initial_peers</code>). |
| </li> |
| <li> |
| It is useful to fix their address by providing <code>host_maddrs</code> and <code>identity_path</code> |
| arguments to <code>hivemind.DHT</code> |
| (these are forwarded to the underlying <a target="_blank" rel="noopener noreferrer" href="https://libp2p.io/">libp2p</a> daemon). |
| </li> |
| </ul> |
| </li> |
| <li class="mb-2"> |
| <b>(optional)</b> Make it easier for other people to join: |
| |
| <ul> |
| <li> |
| Create notebooks for free GPU providers (Google Colab, Kaggle, AWS SageMaker, etc.). |
| People may run them online and/or download and run them on their own hardware. |
| </li> |
| <li> |
| <a target="_blank" rel="noopener noreferrer" href="https://huggingface.co/organizations/new">Create</a> a Hugging Face organization |
| with all resources related to the training |
| (dataset, model, inference demo, how-to-join walkthrough, links to a dashboard with loss and other metrics, etc.). |
| Look at <a target="_blank" rel="noopener noreferrer" href="https://huggingface.co/training-transformers-together">ours</a> for an example. |
| </li> |
| <li> |
| Set up an authentication system (see the "Security" section). |
| For example, you can ask people to join your organization with their Hugging Face accounts |
| (the website allows either sharing a link for joining or manually approving new participants). |
| This allows you to screen the peers, |
| acknowledge their contributions (e.g., make a leaderboard), and |
| ban accounts who behave maliciously. You can use our <a href="https://collaborative-training-auth.huggingface.co/docs">authentication system</a> or deploy your own |
| (our <a href="https://github.com/huggingface/collaborative-training-auth/tree/demo-neurips">server implementation</a> might be a good start). |
| </li> |
| <li> |
| Set up an inference demo for your model (e.g., using <a target="_blank" rel="noopener noreferrer" href="https://huggingface.co/spaces">Spaces</a>) or |
| a script that periodically uploads the inference results to show the training progress. |
| </li> |
| </ul> |
| </li> |
| </ol> |
| </div> |
| </div> |
|
|
| </div> |
|
|
| <h3 class="my-3">Organizers</h3> |
|
|
| This demonstration was created by |
| <a href="https://twitter.com/sasha_borzunov">Alexander Borzunov*</a>, |
| <a href="https://twitter.com/m_ryabinin">Max Ryabinin*</a>, |
| <a href="https://twitter.com/Tim_Dettmers">Tim Dettmers*</a>, |
| <a href="https://twitter.com/qlhoest">Quentin Lhoest*</a>, |
| <a href="https://twitter.com/LucileSaulnier">Lucile Saulnier*</a>, |
| <a href="https://twitter.com/michael_diskin">Michael Diskin</a>, |
| <a href="https://twitter.com/YJernite">Yacine Jernite</a>, and |
| <a href="https://twitter.com/Thom_Wolf">Thomas Wolf</a>. |
|
|
| <h3 class="my-3">Learn more</h3> |
|
|
| <ul class="mb-5"> |
| <li>A NeurIPS 2021 <a href="https://arxiv.org/abs/2106.10207">paper</a> on collaborative deep learning.</li> |
| <li><a href="https://github.com/learning-at-home/hivemind">hivemind</a> is a PyTorch library for decentralized deep learning.</li> |
| <li><a href="https://github.com/huggingface/datasets">🤗 Datasets</a> allows uploading and streaming training data from the Hub.</li> |
| <li><a href="https://github.com/facebookresearch/bitsandbytes">bitsandbytes</a> contains implementations of 8-bit optimizers.</li> |
| <li>A <a href="https://arxiv.org/abs/2110.02861">paper</a> on blockwise quantization for communication-efficient training.</li> |
| |
| |
| </ul> |
|
|
| |
| |
| |
| <script src="https://getbootstrap.com/docs/5.0/dist/js/bootstrap.min.js"></script> |
| </body> |
| </html> |
|
|
