LoRA Hotswapping in Production: Serving Millions of Personalized Avatars

Everyone talks about training LoRAs. There are hundreds of guides about ranks, learning rates and datasets. Almost nobody talks about the next step: serving them in production, where every request needs a different LoRA and users expect results in under a minute.

At FERASET we build an AI avatar app used by millions of people. Every user trains a personal identity LoRA from their selfies. When they generate an avatar pack, our servers load their adapter plus a style adapter, generate 6 images with a diffusion transformer in one batch, and return the results. A different user means different weights, on every request, on every warm worker.

A job that takes about 118 seconds the naive way takes about 41 seconds in our setup. Two LoRAs are swapped in place, there is zero recompilation, and it all runs on a single H100 SXM. In this post we share the production recipe, a measured breakdown of where the speedup actually comes from, and five lessons we learned the hard way.

Our stack: diffusers 0.35.1, peft 0.17.0, torch 2.8.0, a diffusion transformer in bf16, H100 SXM GPUs on fal serverless. All numbers below were measured on this stack at our production settings: 1024x1536, 30 steps, batches of 6. Nothing here depends on the specific base model. The serving techniques are what transfer.

Why naive LoRA serving is slow

The obvious way to serve per-user LoRAs is what every notebook does:

pipeline.load_lora_weights(user_lora_path)
images = pipeline(...)
pipeline.unload_lora_weights()

This works fine once. In production it breaks down for three reasons that stack on top of each other:

1. Structural churn. load_lora_weights() normally changes the model itself: new PEFT modules, new tensors. Doing this on every request costs seconds and fragments GPU memory.
2. It does not survive torch.compile. Every structural change invalidates the compiled graph, and recompilation costs from tens of seconds up to minutes. So you either give up compilation (35% of our speedup, see the table below) or you give up per-user adapters. With the naive approach you cannot have both.
3. Sequential generation multiplies everything. Our original pipeline generated pack images one by one. Six images at around 19 seconds each, plus reload overhead, is how you end up above two minutes per pack.

Here is what this costs, measured on the same H100 with the same model and prompts:

The recipe: hotswapping and compilation together

The solution is LoRA hotswapping, which came to diffusers thanks to the PEFT team. The idea: build the adapter structure once, compile once, and after that only copy new weights into the existing tensors. Same shapes, same graph, no recompilation, no memory growth.

Our setup, once at cold start:

pipeline.model.set_attn_processor(FlashAttnProcessor3_0())  # FA3, see below

# 1. Reserve adapter slots padded to rank 128, so any user LoRA fits
pipeline.enable_lora_hotswap(target_rank=128)

# 2. Load two placeholder adapters to create the slots BEFORE compiling
pipeline.load_lora_weights(dummy_style_lora, adapter_name="style_lora")
pipeline.load_lora_weights(dummy_avatar_lora, adapter_name="avatar_lora")
pipeline.set_adapters(["avatar_lora", "style_lora"], [1.0, 0.8])

# 3. Compile the repeated transformer blocks (regional compilation)
pipeline.model.compile_repeated_blocks(fullgraph=True, dynamic=True)
warmup(pipeline, batch=6)

Then, on every request:

# Different user, different weights, same graph
pipeline.load_lora_weights(user_lora, hotswap=True, adapter_name="avatar_lora")
pipeline.load_lora_weights(style_lora, hotswap=True, adapter_name="style_lora")

Some production details that matter:

Two adapters, swapped at the same time. Most hotswap examples show a single adapter. We run two at once, the user's identity LoRA and a style LoRA, and both are hotswapped per request. This works well, with one constraint to know about: adapters swapped in later can only target the same layers (or a subset) as the first adapter you loaded. Your placeholder adapters define the ceiling.

target_rank=128 is the shape contract. User LoRAs come out of training with whatever rank the trainer used. Hotswapping needs fixed tensor shapes, so diffusers zero-pads every incoming adapter up to target_rank. Set it to the maximum rank you will ever serve, and train below it.

Regional compilation keeps warmup short. compile_repeated_blocks compiles one transformer block and reuses the result for all repeated blocks. We measured a 52 second full warmup from scratch, compared to several minutes for whole-graph compilation. On top of that we save the inductor cache artifacts to the worker's network volume (torch.compiler.save_cache_artifacts / load_cache_artifacts), so a fresh worker skips most of even that. The model weights live on that network volume too, not inside the container image, which keeps the container small and worker pull times short.

Batch the pack. All six images of a pack run as one batch-6 forward pass. This alone is a 1.47x improvement over the naive pipeline (115.7s to 78.7s), before any of the more advanced tricks.

Chunk bigger requests to the compiled batch size. The compiled graph and the VRAM budget are shaped by the warmup batch. A request for 60 images does not run as one batch of 60: it would not fit in memory, and it would push the compiler into shapes it has never seen. We split any larger request into chunks of the warmup batch size, so every forward pass hits the same compiled graph and the same memory ceiling.

FP8 FlashAttention-3. Attention runs through FA3 on the H100s, with Q/K/V cast to float8_e4m3fn. The kernel is wrapped in a torch.library.custom_op with a registered fake implementation, so it traces cleanly under fullgraph=True. How we get FA3 into the environment is a story of its own, see lesson 3.

Step caching. We run cache-dit (DBCache + TaylorSeer, residual threshold 0.12) to skip transformer computation on steps where the output is predictable.

Where the speedup actually comes from

It is easy to stack five optimizations, see a big number, and give the credit to your favorite one. So we turned each one off separately, at production settings, three runs per configuration, with adapter scales held constant:

The honest ranking: compilation is by far the biggest lever, FA3 is second, and the step cache, the most exotic trick in the stack, is worth a real but modest 11%.

And hotswapping itself? It contributes almost nothing to raw speed. Its job is different: it is what allows the other optimizations to survive per-user weights. An in-place swap of both adapters costs 0.5 seconds and leaves the compiled graph untouched. A naive reload costs 2.5 seconds and, under compilation, would trigger a recompile that eats the entire time budget.

End to end for one user: about 2 seconds to pull their adapter from storage (Cloudflare R2, multipart download), 0.5 seconds to hotswap both adapters, 40.6 seconds to generate six images. The recompile counter across all of it stays flat.

Five lessons the tutorials do not cover

1. Changing adapter weights is free. Changing adapter scales is not.

This one surprised us. We instrumented torch._dynamo counters and ran a probe sequence on a warm worker. Swapping in a different user's identity LoRA with the same scales: graph count stays at 3. Swapping the style LoRA together with its scale value: 3 becomes 5. Changing scales again: 7.

Calling set_adapters(["avatar_lora", "style_lora"], [new_scale, new_scale]) with values the compiler has not seen before changes a graph guard, which triggers recompilation, even though the weight swap itself is free. Each of our styles has its own tuned scale pair, so in the worst case a worker pays a hidden recompile the first time it sees each scale combination. Dynamo caches the graph once seen, and the inductor cache persists across workers, so the cost is bounded, but it is real.

If you can, keep scales constant and bake the per-style strength into the adapter weights at swap time. At minimum, know that a "no recompilation" claim needs this footnote, and that torch._dynamo.utils.counters is how you check yours.

2. Optimization recipes rot under version pins. Audit yours.

While preparing this post, we found fuse_qkv_projections() in our own setup code. It is a standard trick from every optimization guide for this model family, and we had copied it too. Under our pinned diffusers 0.35.1, with a custom attention processor installed, it turned out to be a complete no-op that silently held about 4 GB of VRAM: it copies the Q/K/V weights into fused tensors that our processor never reads. In an older diffusers version the same call also installed a fused attention processor, so it used to do something. The library changed underneath the recipe, and the call quietly became dead weight.

Worse, it left a trap behind. The fusion flag stays set, and the fused copies are frozen base weights from before any LoRA is loaded. If anyone had ever switched back to the stock attention processor, attention would have silently stopped seeing the hotswapped user adapters. No error anywhere, just avatars that stop looking like the user.

We deleted the call and measured 4.1 GB of VRAM back, with bit-identical outputs. The general lesson: every optimization you copied from a guide deserves a re-check each time you bump a pinned dependency.

3. Prebuilt kernel wheels break silently. The kernels library fixes this.

Getting FlashAttention-3 into a serverless container used to mean building the wheel yourself against your exact torch and CUDA versions, then hosting it somewhere. Ours lived in a HF model repo, built for torch 2.8.0 + cu12. This works until the environment drifts. While preparing this post, one unpinned dependency resolution gave us torch 2.13/cu130, and the wheel died with ImportError: libcudart.so.12: cannot open shared object file. That is the typical failure mode: cryptic, environment dependent, and always at the worst time.

Today the same thing is one line with the kernels library:

from kernels import get_kernel
fa3 = get_kernel("kernels-community/flash-attn3", version=1)

get_kernel picks a prebuilt build for your exact torch and CUDA at load time (for us: torch28-cxx11-cu128). We verified the swap before trusting it: at our production shapes, the Hub kernel's FP8 outputs are bitwise identical to our hand-built wheel (max abs diff 0.0), and latency is the same within noise (3.47 vs 3.51 ms per attention call). We migrated with the wheel kept as a fallback path. The compiled custom_op wrapper did not change at all.

4. Not every error deserves a pipeline rebuild

Our original error handling had one catch-all: on any exception, tear the whole pipeline down and rebuild it. Unload adapters, reset dynamo, move the model off the GPU, reload, recompile. About two minutes of H100 time.

Then we watched a malformed task, with a null LoRA path caused by an upstream race, take a perfectly healthy warm worker down for those two minutes. It surfaced as a confusing Hub validation error, because diffusers treats a nonexistent local path as a repo id. The pipeline was never corrupted. The request was simply bad.

The fix is simple but valuable: validate the task before touching the pipeline, check that the adapter file actually exists after download, and split the exception path in two. Data errors fail the request in milliseconds and keep the worker warm. Only genuine engine errors (CUDA faults, corrupted adapter state) pay for a rebuild. In a hotswapping system, the worker's warm state is exactly what all this machinery is protecting. Do not throw it away because of someone else's bad input.

5. Cold adapters are your tail latency

In steady state a pack takes about 41 seconds. Then a request selects a style LoRA that is not cached on that worker and pulls it cold from the Hugging Face Hub: 107 seconds. Same code path, 2.6x the latency, purely from a cold adapter download plus format conversion (community style LoRAs come in kohya, xlabs and other layouts, and need converting to the diffusers format).

Per-user identity LoRAs are cold by nature, since every user is new to every worker, but they are small and come from our own R2 bucket in about 2 seconds. The style catalog on the other hand is finite and known in advance. It belongs on the worker's volume before the first request, converted once. Pre-cache what you can enumerate, and budget your tail latency for what you cannot.

What we have not solved yet

• No per-user adapter cache. Every request re-downloads the user's LoRA from R2 (about 2 seconds). An LRU cache on the worker volume would remove this for repeat users. It has not been worth the complexity yet.
• Text encoder LoRAs cannot be hotswapped. This is a current diffusers limitation, so our adapters target the transformer only.
• The scale footnote from lesson 1. Two fixes are on our list: folding style strength into the adapter weights at swap time, or saving a compile cache per scale combination and loading the matching one on demand, so a first-seen combination costs a cache load instead of a full recompile.
• First-load ceiling. target_rank and the layer coverage of the placeholder adapters are commitments made before compilation. Serving an adapter that exceeds either one means a redeploy.

Takeaways

If you remember five things from this post:

1. Hotswapping is not a speed optimization. It is what lets your speed optimizations coexist with per-user weights. 0.5 second in-place swaps, zero recompiles, on a compiled model.
2. Measure your stack piece by piece. Ours ranked compile (35%) above FA3 (17%) above step cache (11%), which is not the order the hype suggests.
3. Batch the pack before anything clever: 1.47x for free.
4. Watch your recompile counters (torch._dynamo.utils.counters). Weight swaps are free, scale changes are not.
5. Re-audit inherited recipes on every dependency bump, and design failure paths that protect the warm state you worked so hard to build.

The end result: a 118 second job runs in 41 seconds, every user gets their own weights, and the infrastructure scales with demand.

---

This post builds on the diffusers/PEFT hotswapping work by Benjamin Bossan and Sayak Paul. Their post is the perfect companion tutorial to this production story. Thanks to the fal team, whose serverless H100s run everything above, and to kernels-community for making custom CUDA kernels easy to deploy.

If you enjoy working on problems like the ones in this article, we are hiring and I am always happy to talk about them. Reach out to me on LinkedIn.