sglang/docs/advanced_features/lora.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LoRA Serving"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SGLang enables the use of [LoRA adapters](https://arxiv.org/abs/2106.09685) with a base model. By incorporating techniques from [S-LoRA](https://arxiv.org/pdf/2311.03285) and [Punica](https://arxiv.org/pdf/2310.18547), SGLang can efficiently support multiple LoRA adapters for different sequences within a single batch of inputs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Arguments for LoRA Serving"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following server arguments are relevant for multi-LoRA serving:\n",
    "\n",
    "* `enable_lora`: Enable LoRA support for the model. This argument is automatically set to True if `--lora-paths` is provided for backward compatibility.\n",
    "\n",
    "* `enable_lora_overlap_loading`: Enable asynchronous LoRA weight loading in order to overlap H2D transfers with GPU compute. This should be enabled if you find that your LoRA workloads are bottlenecked by adapter weight loading, for example when frequently loading large LoRA adapters.\n",
    "\n",
    "* `lora_paths`: The list of LoRA adapters to load. Each adapter must be specified in one of the following formats: <PATH> | <NAME>=<PATH> | JSON with schema {\"lora_name\":str,\"lora_path\":str,\"pinned\":bool}.\n",
    "\n",
    "* `max_loras_per_batch`: Maximum number of adaptors used by each batch. This argument can affect the amount of GPU memory reserved for multi-LoRA serving, so it should be set to a smaller value when memory is scarce. Defaults to be 8.\n",
    "\n",
    "* `max_loaded_loras`: If specified, it limits the maximum number of LoRA adapters loaded in CPU memory at a time. The value must be greater than or equal to `max-loras-per-batch`.\n",
    "\n",
    "* `lora_eviction_policy`: LoRA adapter eviction policy when GPU memory pool is full. `lru`: Least Recently Used (default, better cache efficiency). `fifo`: First-In-First-Out.\n",
    "\n",
    "* `lora_backend`: The backend of running GEMM kernels for Lora modules. Currently we support Triton LoRA backend (`triton`) and Chunked SGMV backend (`csgmv`). In the future, faster backend built upon Cutlass or Cuda kernels will be added.\n",
    "\n",
    "* `max_lora_rank`: The maximum LoRA rank that should be supported. If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. This argument is needed when you expect to dynamically load adapters of larger LoRA rank after server startup.\n",
    "\n",
    "* `lora_target_modules`: The union set of all target modules where LoRA should be applied (e.g., `q_proj`, `k_proj`, `gate_proj`). If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. This argument is needed when you expect to dynamically load adapters of different target modules after server startup. You can also set it to `all` to enable LoRA for all supported modules. However, enabling LoRA on additional modules introduces a minor performance overhead. If your application is performance-sensitive, we recommend only specifying the modules for which you plan to load adapters.\n",
    "\n",
    "* `--max-lora-chunk-size`: Maximum chunk size for the ChunkedSGMV LoRA backend. Only used when --lora-backend is 'csgmv'. Choosing a larger value might improve performance. Please tune this value based on your hardware and workload as needed. Defaults to 16.\n",
    "\n",
    "* `tp_size`: LoRA serving along with Tensor Parallelism is supported by SGLang. `tp_size` controls the number of GPUs for tensor parallelism. More details on the tensor sharding strategy can be found in [S-Lora](https://arxiv.org/pdf/2311.03285) paper.\n",
    "\n",
    "From client side, the user needs to provide a list of strings as input batch, and a list of adaptor names that each input sequence corresponds to."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Usage\n",
    "\n",
    "### Serving Single Adaptor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** SGLang supports LoRA adapters through two APIs:\n",
    "\n",
    "1. **OpenAI-Compatible API** (`/v1/chat/completions`, `/v1/completions`): Use the `model:adapter-name` syntax. See [OpenAI API with LoRA](../basic_usage/openai_api_completions.ipynb#Using-LoRA-Adapters) for examples.\n",
    "\n",
    "2. **Native API** (`/generate`): Pass `lora_path` in the request body (shown below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import requests\n",
    "\n",
    "from sglang.test.doc_patch import launch_server_cmd\n",
    "from sglang.utils import wait_for_server, terminate_process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "server_process, port = launch_server_cmd(\n",
    "    # Here we set max-loras-per-batch to 2: one slot for adaptor and another one for base model\n",
    "    \"\"\"\n",
    "python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
    "    --enable-lora \\\n",
    "    --lora-paths lora0=algoprog/fact-generation-llama-3.1-8b-instruct-lora \\\n",
    "    --max-loras-per-batch 2 \\\n",
    "    --log-level warning \\\n",
    "\"\"\"\n",
    ")\n",
    "\n",
    "wait_for_server(f\"http://localhost:{port}\", process=server_process)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = f\"http://127.0.0.1:{port}\"\n",
    "json_data = {\n",
    "    \"text\": [\n",
    "        \"List 3 countries and their capitals.\",\n",
    "        \"List 3 countries and their capitals.\",\n",
    "    ],\n",
    "    \"sampling_params\": {\"max_new_tokens\": 32, \"temperature\": 0},\n",
    "    # The first input uses lora0, and the second input uses the base model\n",
    "    \"lora_path\": [\"lora0\", None],\n",
    "}\n",
    "response = requests.post(\n",
    "    url + \"/generate\",\n",
    "    json=json_data,\n",
    ")\n",
    "print(f\"Output 0: {response.json()[0]['text']}\")\n",
    "print(f\"Output 1: {response.json()[1]['text']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "terminate_process(server_process)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Serving Multiple Adaptors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "server_process, port = launch_server_cmd(\"\"\"\n",
    "python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
    "    --enable-lora \\\n",
    "    --lora-paths lora0=algoprog/fact-generation-llama-3.1-8b-instruct-lora \\\n",
    "    lora1=Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json \\\n",
    "    --max-loras-per-batch 2 \\\n",
    "    --log-level warning \\\n",
    "\"\"\")\n",
    "\n",
    "wait_for_server(f\"http://localhost:{port}\", process=server_process)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = f\"http://127.0.0.1:{port}\"\n",
    "json_data = {\n",
    "    \"text\": [\n",
    "        \"List 3 countries and their capitals.\",\n",
    "        \"List 3 countries and their capitals.\",\n",
    "    ],\n",
    "    \"sampling_params\": {\"max_new_tokens\": 32, \"temperature\": 0},\n",
    "    # The first input uses lora0, and the second input uses lora1\n",
    "    \"lora_path\": [\"lora0\", \"lora1\"],\n",
    "}\n",
    "response = requests.post(\n",
    "    url + \"/generate\",\n",
    "    json=json_data,\n",
    ")\n",
    "print(f\"Output 0: {response.json()[0]['text']}\")\n",
    "print(f\"Output 1: {response.json()[1]['text']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "terminate_process(server_process)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dynamic LoRA loading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of specifying all adapters during server startup via `--lora-paths`. You can also load & unload LoRA adapters dynamically via the `/load_lora_adapter` and `/unload_lora_adapter` API.\n",
    "\n",
    "When using dynamic LoRA loading, it's recommended to explicitly specify both `--max-lora-rank` and `--lora-target-modules` at startup. For backward compatibility, SGLang will infer these values from `--lora-paths` if they are not explicitly provided. However, in that case, you would have to ensure that all dynamically loaded adapters share the same shape (rank and target modules) as those in the initial `--lora-paths` or are strictly \"smaller\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lora0 = \"Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json\"  # rank - 4, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj\n",
    "lora1 = \"algoprog/fact-generation-llama-3.1-8b-instruct-lora\"  # rank - 64, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj\n",
    "lora0_new = \"philschmid/code-llama-3-1-8b-text-to-sql-lora\"  # rank - 256, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj\n",
    "\n",
    "\n",
    "# The `--target-lora-modules` param below is technically not needed, as the server will infer it from lora0 which already has all the target modules specified.\n",
    "# We are adding it here just to demonstrate usage.\n",
    "server_process, port = launch_server_cmd(\"\"\"\n",
    "    python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
    "    --enable-lora \\\n",
    "    --cuda-graph-max-bs 2 \\\n",
    "    --max-loras-per-batch 2 \\\n",
    "    --max-lora-rank 256\n",
    "    --lora-target-modules all\n",
    "    --log-level warning\n",
    "    \"\"\")\n",
    "\n",
    "url = f\"http://127.0.0.1:{port}\"\n",
    "wait_for_server(url, process=server_process)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load adapter lora0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.post(\n",
    "    url + \"/load_lora_adapter\",\n",
    "    json={\n",
    "        \"lora_name\": \"lora0\",\n",
    "        \"lora_path\": lora0,\n",
    "    },\n",
    ")\n",
    "\n",
    "if response.status_code == 200:\n",
    "    print(\"LoRA adapter loaded successfully.\", response.json())\n",
    "else:\n",
    "    print(\"Failed to load LoRA adapter.\", response.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load adapter lora1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.post(\n",
    "    url + \"/load_lora_adapter\",\n",
    "    json={\n",
    "        \"lora_name\": \"lora1\",\n",
    "        \"lora_path\": lora1,\n",
    "    },\n",
    ")\n",
    "\n",
    "if response.status_code == 200:\n",
    "    print(\"LoRA adapter loaded successfully.\", response.json())\n",
    "else:\n",
    "    print(\"Failed to load LoRA adapter.\", response.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check inference output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = f\"http://127.0.0.1:{port}\"\n",
    "json_data = {\n",
    "    \"text\": [\n",
    "        \"List 3 countries and their capitals.\",\n",
    "        \"List 3 countries and their capitals.\",\n",
    "    ],\n",
    "    \"sampling_params\": {\"max_new_tokens\": 32, \"temperature\": 0},\n",
    "    # The first input uses lora0, and the second input uses lora1\n",
    "    \"lora_path\": [\"lora0\", \"lora1\"],\n",
    "}\n",
    "response = requests.post(\n",
    "    url + \"/generate\",\n",
    "    json=json_data,\n",
    ")\n",
    "print(f\"Output from lora0: \\n{response.json()[0]['text']}\\n\")\n",
    "print(f\"Output from lora1 (updated): \\n{response.json()[1]['text']}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unload lora0 and replace it with a different adapter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.post(\n",
    "    url + \"/unload_lora_adapter\",\n",
    "    json={\n",
    "        \"lora_name\": \"lora0\",\n",
    "    },\n",
    ")\n",
    "\n",
    "response = requests.post(\n",
    "    url + \"/load_lora_adapter\",\n",
    "    json={\n",
    "        \"lora_name\": \"lora0\",\n",
    "        \"lora_path\": lora0_new,\n",
    "    },\n",
    ")\n",
    "\n",
    "if response.status_code == 200:\n",
    "    print(\"LoRA adapter loaded successfully.\", response.json())\n",
    "else:\n",
    "    print(\"Failed to load LoRA adapter.\", response.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check output again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = f\"http://127.0.0.1:{port}\"\n",
    "json_data = {\n",
    "    \"text\": [\n",
    "        \"List 3 countries and their capitals.\",\n",
    "        \"List 3 countries and their capitals.\",\n",
    "    ],\n",
    "    \"sampling_params\": {\"max_new_tokens\": 32, \"temperature\": 0},\n",
    "    # The first input uses lora0, and the second input uses lora1\n",
    "    \"lora_path\": [\"lora0\", \"lora1\"],\n",
    "}\n",
    "response = requests.post(\n",
    "    url + \"/generate\",\n",
    "    json=json_data,\n",
    ")\n",
    "print(f\"Output from lora0: \\n{response.json()[0]['text']}\\n\")\n",
    "print(f\"Output from lora1 (updated): \\n{response.json()[1]['text']}\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "terminate_process(server_process)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OpenAI-compatible API usage\n",
    "\n",
    "You can use LoRA adapters via the OpenAI-compatible APIs by specifying the adapter in the `model` field using the `base-model:adapter-name` syntax (for example, `qwen/qwen2.5-0.5b-instruct:adapter_a`). For more details and examples, see the “Using LoRA Adapters” section in the OpenAI API documentation: [openai_api_completions.ipynb](../basic_usage/openai_api_completions.ipynb).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LoRA GPU Pinning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another advanced option is to specify adapters as `pinned` during loading. When an adapter is pinned, it is permanently assigned to one of the available GPU pool slots (as configured by `--max-loras-per-batch`) and will not be evicted from GPU memory during runtime. Instead, it remains resident until it is explicitly unloaded.\n",
    "\n",
    "This can improve performance in scenarios where the same adapter is frequently used across requests, by avoiding repeated memory transfers and reinitialization overhead. However, since GPU pool slots are limited, pinning adapters reduces the flexibility of the system to dynamically load other adapters on demand. If too many adapters are pinned, it may lead to degraded performance, or in the most extreme case (`Number of pinned adapters == max-loras-per-batch`), halt all unpinned requests. Therefore, currently SGLang limits maximal number of pinned adapters to `max-loras-per-batch - 1` to prevent unexpected starvations. \n",
    "\n",
    "In the example below, we start a server with `lora1` loaded as pinned, `lora2` and `lora3` loaded as regular (unpinned) adapters. Please note that, we intentionally specify `lora2` and `lora3` in two different formats to demonstrate that both are supported."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "server_process, port = launch_server_cmd(\"\"\"\n",
    "    python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
    "    --enable-lora \\\n",
    "    --cuda-graph-max-bs 8 \\\n",
    "    --max-loras-per-batch 3 \\\n",
    "    --max-lora-rank 256 \\\n",
    "    --lora-target-modules all \\\n",
    "    --lora-paths \\\n",
    "        {\"lora_name\":\"lora0\",\"lora_path\":\"Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json\",\"pinned\":true} \\\n",
    "        {\"lora_name\":\"lora1\",\"lora_path\":\"algoprog/fact-generation-llama-3.1-8b-instruct-lora\"} \\\n",
    "        lora2=philschmid/code-llama-3-1-8b-text-to-sql-lora\n",
    "    --log-level warning\n",
    "    \"\"\")\n",
    "\n",
    "\n",
    "url = f\"http://127.0.0.1:{port}\"\n",
    "wait_for_server(url, process=server_process)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also specify adapter as pinned during dynamic adapter loading. In the example below, we reload `lora2` as pinned adapter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.post(\n",
    "    url + \"/unload_lora_adapter\",\n",
    "    json={\n",
    "        \"lora_name\": \"lora1\",\n",
    "    },\n",
    ")\n",
    "\n",
    "response = requests.post(\n",
    "    url + \"/load_lora_adapter\",\n",
    "    json={\n",
    "        \"lora_name\": \"lora1\",\n",
    "        \"lora_path\": \"algoprog/fact-generation-llama-3.1-8b-instruct-lora\",\n",
    "        \"pinned\": True,  # Pin the adapter to GPU\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verify that the results are expected:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = f\"http://127.0.0.1:{port}\"\n",
    "json_data = {\n",
    "    \"text\": [\n",
    "        \"List 3 countries and their capitals.\",\n",
    "        \"List 3 countries and their capitals.\",\n",
    "        \"List 3 countries and their capitals.\",\n",
    "    ],\n",
    "    \"sampling_params\": {\"max_new_tokens\": 32, \"temperature\": 0},\n",
    "    # The first input uses lora0, and the second input uses lora1\n",
    "    \"lora_path\": [\"lora0\", \"lora1\", \"lora2\"],\n",
    "}\n",
    "response = requests.post(\n",
    "    url + \"/generate\",\n",
    "    json=json_data,\n",
    ")\n",
    "print(f\"Output from lora0 (pinned): \\n{response.json()[0]['text']}\\n\")\n",
    "print(f\"Output from lora1 (pinned): \\n{response.json()[1]['text']}\\n\")\n",
    "print(f\"Output from lora2 (not pinned): \\n{response.json()[2]['text']}\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "terminate_process(server_process)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Choosing LoRA Backend\n",
    "\n",
    "SGLang supports two LoRA backends that you can choose from using the `--lora-backend` argument:\n",
    "\n",
    "- `triton`: Basic Triton-based backend.\n",
    "- `csgmv`: Default chunked SGMV backend optimized for high concurrency scenarios.\n",
    "\n",
    "The `csgmv` backend was recently introduced to improve performance especially at high-concurrency scenarios. Our benchmark shows that it achieves 20% to 80% latency improvements over the basic triton backend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "server_process, port = launch_server_cmd(\"\"\"\n",
    "    python3 -m sglang.launch_server \\\n",
    "    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
    "    --enable-lora \\\n",
    "    --lora-backend csgmv \\\n",
    "    --max-loras-per-batch 16 \\\n",
    "    --lora-paths lora1=path/to/lora1 lora2=path/to/lora2\n",
    "    \"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "terminate_process(server_process)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LoRA Overlap Loading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By using the `--enable-lora-overlap-loading` server argument, the SGLang engine is able to overlap the loading of LoRA weights with prefill and decode compute, essentially hiding the data movement for LoRA weights behind GPU computation. Our benchmarks show that under adversarial conditions, enabling this feature can result in a ~35% reduction in median TTFT - (see the [LoRA overlap loading PR](https://github.com/sgl-project/sglang/pull/15512) for detailed benchmarks)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lora0 = \"Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json\"\n",
    "lora1 = \"algoprog/fact-generation-llama-3.1-8b-instruct-lora\"\n",
    "lora2 = \"philschmid/code-llama-3-1-8b-text-to-sql-lora\"\n",
    "\n",
    "\n",
    "server_process, port = launch_server_cmd(\"\"\"\n",
    "    python3 -m sglang.launch_server \\\n",
    "    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
    "    --enable-lora \\\n",
    "    --enable-lora-overlap-loading \\\n",
    "    --lora-paths lora0=Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json \\\n",
    "    lora1=algoprog/fact-generation-llama-3.1-8b-instruct-lora \\\n",
    "    lora2=philschmid/code-llama-3-1-8b-text-to-sql-lora \\\n",
    "    --max-lora-rank 256 \\\n",
    "    --max-loras-per-batch 2 \\\n",
    "    --max-loaded-loras 4\n",
    "    \"\"\")\n",
    "\n",
    "url = f\"http://127.0.0.1:{port}\"\n",
    "wait_for_server(url, process=server_process)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "json_data = {\n",
    "    \"text\": [\n",
    "        \"Write a very long fairy-tale.\",\n",
    "        \"List 3 countries and their capitals.\",\n",
    "        \"List 3 countries and their capitals.\",\n",
    "    ],\n",
    "    \"sampling_params\": [\n",
    "        {\"max_new_tokens\": 1024, \"temperature\": 0},\n",
    "        {\"max_new_tokens\": 64, \"temperature\": 0},\n",
    "        {\"max_new_tokens\": 64, \"temperature\": 0},\n",
    "    ],\n",
    "    \"lora_path\": [\"lora0\", \"lora1\", \"lora2\"],\n",
    "}\n",
    "\n",
    "# lora0 and lora1 will be loaded into the memory pool first, and because max_loras_per_batch = 2, lora2's request will remain in the queue.\n",
    "# lora1's request will likely finish first, and once it does, lora2 will be loaded. With --enable-lora-overlap-loading, this loading will\n",
    "# occur asynchronously and thus decoding for lora0's request won't be blocked.\n",
    "response = requests.post(\n",
    "    url + \"/generate\",\n",
    "    json=json_data,\n",
    ")\n",
    "\n",
    "for i in range(3):\n",
    "    print(f\"Output from lora{i}: \\n{response.json()[i]['text']}\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "terminate_process(server_process)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Limitations of LoRA Overlap Loading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, LoRA overlap loading is not free and comes with two important caveats:\n",
    "\n",
    "1. **Pinned CPU memory requirement**:\n",
    "   Asynchronous H2D memory copies require LoRA weights to be pinned in CPU memory, which is a finite system resource. To mitigate excessive pinned-memory usage, SGLang currently restricts `max_loaded_loras` to be at most 2× `max_loras_per_batch` when LoRA overlap loading is enabled.\n",
    "\n",
    "2. **Reduced multi-adapter prefill batching**:\n",
    "   With overlap loading, adapters become available on the GPU at different times because each adapter is loaded asynchronously. This can reduce the scheduler’s ability to form multi-adapter prefill batches, since only requests whose adapters are currently loaded can be grouped together. As a result, requests for different adapters will be scheduled in separate (or smaller) prefill batches, which can increase TTFT when adapter load time is small compared to prefill compute time. This is why LoRA overlap loading is disabled by default: it should only be enabled when users have determined that LoRA weight loading is a bottleneck (EG high adapter churn, heavy adapter weights, or PCIe-bottlenecked workloads).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example When Overlap Loading Results in Higher Latency"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, suppose we have four LoRA adapters: `lora0`, `lora1`, `lora2`, and `lora3`. Loading any adapter takes 2ms, while the prefill step for requests for that adapter takes 20ms.\n",
    "\n",
    "1. **Baseline**:\n",
    "  The engine loads all four adapters synchronously, then runs one combined prefill batch, giving us a total time of ≈ `2 * 4 + 20 = 28ms`\n",
    "\n",
    "2. **With LoRA overlap loading enabled**:\n",
    "  The engine begins loading `lora0` and, once it is ready, schedules a prefill batch containing only `lora0` while `lora1` loads in the background. Then it schedules `lora1`’s prefill while `lora2` loads, and so on. In the worst case where prefill cannot be batched across adapters, total time is ≈ `2 + 4 * 20 = 82ms`\n",
    "\n",
    "In this scenario, overlap loading reduces adapter-load overhead, but the loss of multi-adapter prefill batching dominates and leads to higher TTFT."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Future Works\n",
    "\n",
    "The development roadmap for LoRA-related features can be found in this [issue](https://github.com/sgl-project/sglang/issues/2929). Other features, including Embedding Layer, Unified Paging, Cutlass backend are still under development."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}