Spaces:
Running
Running
| # LiteRT-LM: CMake Overview | |
| The LiteRT-LM CMake build system provides a unified infrastructure for building | |
| all required third-party dependencies, internal libraries, and the primary | |
| litert_lm_main executable. | |
| Additional executable targets (such as litert_lm_advanced_main) are defined but | |
| currently gated behind the `_unverified_targets` flag. They remain in an | |
| unverified state until they can be validated within the standalone CMake | |
| environment. | |
| ## Dependency Management & Project Structure | |
| This project implements a Super-Build pattern to ensure One Definition Rule | |
| (ODR) adherence. This approach is necessary to manage a converging dependency | |
| tree where multiple components rely on different versions of the same core | |
| libraries. | |
| ### Dependency Strategy | |
| The build system leverages a hybrid approach to dependency management: | |
| - **FetchContent**: Used for conventional third-party libraries where | |
| integration is sufficient | |
| - **ExternalProject**: Reserved for dependencies requiring heavy modification. | |
| These are orchestrated to redirect include and library paths to a unified | |
| "source of truth" within the build directory. This prevents the symbol | |
| collisions that occur when multiple dependencies introduce conflicting | |
| versions of the same provider (e.g., Abseil or Protobuf). | |
| ### Package Infrastructure | |
| Orchestration logic for external dependencies is modularized within | |
| `cmake/packages/<name>`. To ensure a hermetic build environment and strict ODR | |
| adherence, these modules implement a Source Transformation and Target Mapping | |
| framework. | |
| This framework does not simply wrap dependencies; it actively transforms them by | |
| "de-nesting" internal third-party code and normalizing source-level paths to | |
| align with the LiteRT-LM unified build structure. | |
| ```bash | |
| cmake/packages/sentencepiece | |
| ├── sentencepiece.cmake # Primary orchestration module (ExternalProject_Add) | |
| ├── sentencepiece_patcher.cmake # Source-level transformation (Path normalization and dependency de-nesting) | |
| ├── sentencepiece_root_shim.cmake # Injected logic for the package-level configuration | |
| ├── sentencepiece_src_shim.cmake # Injected logic for the source-level build definitions | |
| ├── sentencepiece_aggregate.cmake # Logic to consolidate build artifacts into a unified interface | |
| └── sentencepiece_target_map.cmake # Dictionary mapping internal project targets to local static archives | |
| ``` | |
| #### Transformation Highlights | |
| - **Hermeticity**: By removing nested third_party directories within | |
| dependencies, we force all components to resolve a single, verified version | |
| of core libraries (e.g., Abseil, Protobuf). | |
| - **Path Normalization**: Source files are patched in-place to canonicalize | |
| include paths, ensuring compatibility with the standalone CMake layout. | |
| - **Target Redirection**: Using custom mapping logic, internal targets are | |
| transparently redirected to local static archives, maintaining consistency | |
| with the original project structure without requiring a full monorepo | |
| environment. | |
| ### Project Layout | |
| To maintain parity with the internal codebase and facilitate automated | |
| maintenance, LiteRT-LM modeled its CMake target definitions to mirror the source | |
| tree. | |
| - **Source-Locality**: Target definitions for LiteRT-LM components generally | |
| reside in the CMakeLists.txt file located in the same directory as their | |
| respective source files. | |
| - **Exceptions**: Shared resources (such as proto_lib) are consolidated into | |
| centralized configuration files to manage global visibility and reuse | |
| ## Build Guide | |
| This project targets a modern high-performance C++ environment. Currently, the | |
| build system is strictly verified for the GNU toolchain on Debian-based Linux | |
| (e.g., Ubuntu 24.04). | |
| #### Prerequisites | |
| Ensure your local environment meets these minimum version requirements to avoid | |
| compilation errors related to C++20 standards and build-time orchestration. | |
| - **Compiler**: gcc / g++ 13+ (Required for stable C++20 feature support). | |
| - **Build Tools**: cmake (3.25+) and make. | |
| - **Python**: 3.12+ (Required development scripts). | |
| - **Java**: openjdk-17-jre-headless or newer (Required for ANTLR 4). | |
| - **Rust**: The Rust toolchain is required for specific sub-components. | |
| - **System Libraries**: zlib1g-dev, libssl-dev libcurl4-openssl-dev. | |
| -------------------------------------------------------------------------------- | |
| **1. Configuration** | |
| Create a build directory to maintain a clean source tree. Note that while the | |
| build system is currently hard-coded to C++20, future updates will transition | |
| this to a configurable variable. | |
| ```bash | |
| cmake -B cmake/build -G "Unix Makefiles" \ | |
| -DCMAKE_BUILD_TYPE=Release \ | |
| -DCMAKE_CXX_STANDARD=20 | |
| ``` | |
| **2. Executing the Build** | |
| Parallel execution is highly recommended to manage the complex dependency tree, | |
| but it must be balanced against available system resources. | |
| **WARNING** High Memory Usage: Allocating excessive parallel jobs can cause a | |
| SEGFAULT or an OOM-kill (Signal 9). To ensure a stable build, use a conservative | |
| job count. | |
| **Recommended Formula**: Available RAM / 8GB = Max `-j` value | |
| ```bash | |
| # Example for 32GB RAM | |
| cmake --build cmake/build -t litert_lm_main -j4 | |
| ``` | |
| **3. Verification** | |
| Verify the binary integrity and Package Infrastructure mapping via a CPU-based | |
| inference test. This confirms that all internal symbols and external shims | |
| (Abseil, Protobuf, etc.) are correctly linked and functional. | |
| ###### Validation Scope & Environment Notes | |
| - **Primary Target:** The current verification suite focuses on the | |
| CPU-reference implementation, leveraging the high-compute density (48-core) | |
| of the development environment. | |
| - **Future Target:** Verification of hardware-accelerated backends (GPU/NPU) | |
| is deferred to environments with dedicated hardware resource reservation. | |
| Validation requires a physical target or an instance with direct hardware | |
| passthrough to establish the necessary compute-level interface. | |
| ```bash | |
| ./litert_lm_main \ | |
| --model_path=/path/to/gemma-3n-E2B-it-int4.litertlm \ | |
| --backend=cpu \ | |
| --input_prompt="What is the tallest building in the world?" | |
| ``` | |
| **Expected Output**: A successful build will initialize the XNNPACK delegate and | |
| return the model response along with benchmark metrics: | |
| - *Init Phases*: Executor and Tokenizer initialization times. | |
| - *Prefill/Decode Speeds*: Performance stats (tokens/sec) indicating the | |
| backend is optimized. | |
| *Example* | |
| ``` | |
| dev-sh@:LiteRT-LM$ cmake/build/litert_lm_main --model_path=$model_path/gemma-3n-E2B-it-int4.litertlm --backend=cpu --input_prompt="What is the tallest building in the world?" | |
| INFO: Created TensorFlow Lite XNNPACK delegate for CPU. | |
| input_prompt: What is the tallest building in the world? | |
| The tallest building in the world is the **Burj Khalifa** in Dubai, United Arab Emirates. | |
| It stands at a staggering **828 meters (2,717 feet)** tall. | |
| It was completed in 2010 and continues to hold the record. | |
| BenchmarkInfo: | |
| Init Phases (2): | |
| - Executor initialization: 844.54 ms | |
| - Tokenizer initialization: 66.70 ms | |
| Total init time: 911.25 ms | |
| -------------------------------------------------- | |
| Time to first token: 2.40 s | |
| -------------------------------------------------- | |
| Prefill Turns (Total 1 turns): | |
| Prefill Turn 1: Processed 18 tokens in 2.311920273s duration. | |
| Prefill Speed: 7.79 tokens/sec. | |
| -------------------------------------------------- | |
| Decode Turns (Total 1 turns): | |
| Decode Turn 1: Processed 62 tokens in 5.53092314s duration. | |
| Decode Speed: 11.21 tokens/sec. | |
| -------------------------------------------------- | |
| -------------------------------------------------- | |
| ``` | |
| <br> | |
| -------------------------------------------------------------------------------- | |
| This project is licensed under the | |
| [Apache 2.0 License.](https://github.com/google-ai-edge/LiteRT-LM/blob/main/LICENSE) | |
| -------------------------------------------------------------------------------- | |
| ## Getting Started: Running the LiteRT-LM Container | |
| To get the environment up and running, follow these steps from the root | |
| directory of the project. The process is divided into build, create, and attach | |
| phases to ensure container persistence is handled correctly. | |
| ### 1. Build the Image | |
| First, we'll build the image using the configuration in the cmake/ directory. | |
| This might take a moment if it's your first time, as it pulls in our build | |
| dependencies. | |
| ```bash | |
| podman build -f /path/to/repo/cmake/Containerfile -t litert_lm /path/to/repo | |
| ``` | |
| ### 2. Create the Persistent Container | |
| Instead of executing a one-off run, create a named container to preserve the | |
| workspace state for future sessions. Using interactive mode ensures the | |
| container is prepared for a functional terminal. | |
| ```bash | |
| podman container create --interactive --tty --name litert_lm litert_lm:latest | |
| ``` | |
| ### 3. Start and Join the Session | |
| Finally, start the container and attach your shell to it. | |
| ```bash | |
| podman start --attach litert_lm | |
| ``` | |
| **Quick Note:** If you exit the container and want to get back in later, you | |
| don't need to rebuild or recreate it. Just run the podman start --attach | |
| litert_lm command again and you'll be right back where you left off. | |
| <br> | |
| -------------------------------------------------------------------------------- | |
| This project is licensed under the | |
| [Apache 2.0 License.](https://github.com/google-ai-edge/LiteRT-LM/blob/main/LICENSE) | |