# LiteRT-LM Kotlin API The Kotlin API of LiteRT-LM for **Android** and **JVM (Linux, MacOS, Windows)** with features like **GPU and NPU acceleration**, **multi-modality**, and **tools use**. ## Introduction Here is a sample terminal chat app built with the Kotlin API: ```kotlin import com.google.ai.edge.litertlm.* suspend fun main() { Engine.setNativeMinLogSeverity(LogSeverity.ERROR) // Hide log for TUI app val engineConfig = EngineConfig(modelPath = "/path/to/model.litertlm") Engine(engineConfig).use { engine -> engine.initialize() engine.createConversation().use { conversation -> while (true) { print("\n>>> ") conversation.sendMessageAsync(readln()).collect { print(it) } } } } } ``` ![](demo.gif) To try out the above sample, clone the repo and run with [example/Main.kt](../../../kotlin/java/com/google/ai/edge/litertlm/example/Main.kt): ```bazel bazel run -c opt //kotlin/java/com/google/ai/edge/litertlm/example:main -- ``` Available `.litertlm` models are on the [HuggingFace LiteRT Community](https://huggingface.co/litert-community). The above animation was using the [Gemma3-1B-IT](https://huggingface.co/litert-community/Gemma3-1B-IT). For Android sample, check out the [Google AI Edge Gallery](https://github.com/google-ai-edge/gallery) app. ## Getting Started with Gradle While LiteRT-LM is developed with Bazel, we provide the Maven packages for Gradle/Maven users. ### 1. Add the Gradle dependency ``` dependencies { // For Android implementation("com.google.ai.edge.litertlm:litertlm-android:latest.release") // For JVM (Linux, MacOS, Windows) implementation("com.google.ai.edge.litertlm:litertlm-jvm:latest.release") } ``` You can find the available versions on Google Maven in [litertlm-android](https://maven.google.com/web/index.html#com.google.ai.edge.litertlm:litertlm-android) and [litertlm-jvm](https://maven.google.com/web/index.html#com.google.ai.edge.litertlm:litertlm-jvm). `latest.release` can be used to get the latest release. ### 2. Initialize the Engine The `Engine` is the entry point to the API. Initialize it with the model path and configuration. Remember to close the engine to release resources. **Note:** The `engine.initialize()` method can take a significant amount of time (e.g., up to 10 seconds) to load the model. It is strongly recommended to call this on a background thread or coroutine to avoid blocking the UI thread. ```kotlin import com.google.ai.edge.litertlm.Backend import com.google.ai.edge.litertlm.Engine import com.google.ai.edge.litertlm.EngineConfig val engineConfig = EngineConfig( modelPath = "/path/to/your/model.litertlm", // Replace with your model path backend = Backend.CPU(), // Or Backend.GPU() and Backend.NPU("...") // Optional: Pick a writable dir. This can improve 2nd load time. // cacheDir = "/tmp/" or context.cacheDir.path (for Android) ) val engine = Engine(engineConfig) engine.initialize() // ... Use the engine to create a conversation ... // Close the engine when done engine.close() ``` On Android, to use the GPU backend, the app needs to request the depending native libraries explicitly by adding the following to your `AndroidManifest.xml` inside the `` tag: ```xml ``` To use the **NPU** backend, you might need to specify the directory containing the NPU libraries. On Android, if the libraries are bundled with your app, set it to `context.applicationInfo.nativeLibraryDir`. See [LiteRT-LM NPU](https://ai.google.dev/edge/litert/next/litert_lm_npu#NPU) for more details about the NPU native libraries. ```kotlin val engineConfig = EngineConfig( modelPath = modelPath, backend = Backend.NPU(nativeLibraryDir = context.applicationInfo.nativeLibraryDir) ) ``` ### 3. Create a Conversation Once the engine is initialized, create a `Conversation` instance. You can provide a `ConversationConfig` to customize its behavior. ```kotlin import com.google.ai.edge.litertlm.ConversationConfig import com.google.ai.edge.litertlm.Message import com.google.ai.edge.litertlm.SamplerConfig // Optional: Configure the system instruction, initial messages, sampling // parameters, etc. val conversationConfig = ConversationConfig( systemInstruction = Contents.of("You are a helpful assistant."), initialMessages = listOf( Message.user("What is the capital city of the United States?"), Message.model("Washington, D.C."), ), samplerConfig = SamplerConfig(topK = 10, topP = 0.95, temperature = 0.8), ) val conversation = engine.createConversation(conversationConfig) // Or with default config: // val conversation = engine.createConversation() // ... Use the conversation ... // Close the conversation when done conversation.close() ``` `Conversation` implements `AutoCloseable`, so you can use the `use` block for automatic resource management for one-shot or short-lived conversations: ```kotlin engine.createConversation(conversationConfig).use { conversation -> // Interact with the conversation } ``` ### 4. Sending Messages There are three ways to send messages: - **`sendMessage(contents, extraContext): Message`**: Synchronous call that blocks until the model returns a complete response. This is simpler for basic request/response interactions. - **`sendMessageAsync(contents, callback, extraContext)`**: Asynchronous call for streaming responses. This is better for long-running requests or when you want to display the response as it's being generated. - **`sendMessageAsync(contents, extraContext): Flow`**: Asynchronous call that returns a Kotlin Flow for streaming responses. This is the recommended approach for Coroutine users. **Synchronous Example:** ```kotlin import com.google.ai.edge.litertlm.Content import com.google.ai.edge.litertlm.Message print(conversation.sendMessage("What is the capital of France?")) ``` **Asynchronous Example with callback:** Use `sendMessageAsync` to send a message to the model and receive responses through a callback. ```kotlin import com.google.ai.edge.litertlm.Content import com.google.ai.edge.litertlm.Message import com.google.ai.edge.litertlm.MessageCallback import java.util.concurrent.CountDownLatch import java.util.concurrent.TimeUnit val callback = object : MessageCallback { override fun onMessage(message: Message) { print(message) } override fun onDone() { // Streaming completed } override fun onError(throwable: Throwable) { // Error during streaming } } conversation.sendMessageAsync("What is the capital of France?", callback) ``` **Asynchronous Example with Flow:** Use `sendMessageAsync` (without the callback arg) to send a message to the model and receive responses through a Kotlin Flow. ```kotlin import com.google.ai.edge.litertlm.Content import com.google.ai.edge.litertlm.Message import kotlinx.coroutines.flow.catch import kotlinx.coroutines.launch // Within a coroutine scope conversation.sendMessageAsync("What is the capital of France?") .catch { ... } // Error during streaming .collect { print(it.toString()) } ``` ### 5. Multi-Modality Note: This only works with models with multi-modality support, e.g., the [Gemma3n](https://huggingface.co/google/gemma-3n-E2B-it-litert-lm). `Message` objects can contain different types of `Content`, including `Text`, `ImageBytes`, `ImageFile`, and `AudioBytes`, `AudioFile`. ```kotlin // Initialize the `visionBackend` and/or the `audioBackend` val engineConfig = EngineConfig( modelPath = "/path/to/your/model.litertlm", // Replace with your model path backend = Backend.CPU(), // Or Backend.GPU() or Backend.NPU(...) visionBackend = Backend.GPU(), // Or Backend.NPU(...) audioBackend = Backend.CPU(), // Or Backend.NPU(...) ) // Sends a message with multi-modality. // See the Content class for other variants. conversation.sendMessage(Contents.of( Content.ImageFile("/path/to/image"), Content.AudioBytes(audioBytes), // ByteArray of the audio Content.Text("Describe this image and audio."), )) ``` ### 6. Defining and Using Tools Note: This only works with models with tool support, e.g., the [FunctionGemma](https://huggingface.co/google/functiongemma-270m-it). There are two ways to define tools: 1. With Kotlin functions (recommended for most cases) 2. With Open API specification (full control of the tool spec and execution) #### Defining Tools with Kotlin Functions You can define custom Kotlin functions as tools that the model can call to perform actions or fetch information. Create a class implementing `ToolSet` and annotate methods with `@Tool` and parameters with `@ToolParam`. ```kotlin import com.google.ai.edge.litertlm.Tool import com.google.ai.edge.litertlm.ToolParam class SampleToolSet: ToolSet { @Tool(description = "Get the current weather for a city") fun getCurrentWeather( @ToolParam(description = "The city name, e.g., San Francisco") city: String, @ToolParam(description = "Optional country code, e.g., US") country: String? = null, @ToolParam(description = "Temperature unit (celsius or fahrenheit). Default: celsius") unit: String = "celsius" ): Map { // In a real application, you would call a weather API here return mapOf("temperature" to 25, "unit" to unit, "condition" to "Sunny") } @Tool(description = "Get the sum of a list of numbers.") fun sum( @ToolParam(description = "The numbers, could be floating point.") numbers: List, ): Double { return numbers.sum() } } ``` Behind the scenes, the API inspects these annotations and the function signature to generate an OpenAPI-style schema. This schema describes the tool's functionality, parameters (including their types and descriptions from `@ToolParam`), and return type to the language model. ##### Parameter Types The types for parameters annotated with `@ToolParam` can be `String`, `Int`, `Boolean`, `Float`, `Double`, or a `List` of these types (e.g., `List`). Use nullable types (e.g., `String?`) to indicate nullable parameters. Set a default value to indicate that the parameter is optional, and mention the default value in the description in `@ToolParam`. ##### Return Type The return type of your tool function can be any Kotlin type. The result will be converted to a JSON element before being sent back to the model. - `List` types are converted to JSON arrays. - `Map` types are converted to JSON objects. - Primitive types (`String`, `Number`, `Boolean`) are converted to the corresponding JSON primitive. - Other types are converted to strings with the `toString()` method. For structured data, returning `Map` or a data class that will be converted to a JSON object is recommended. #### Defining Tools with OpenAPI Specification Alternatively, you can define a tool by implementing the `OpenApiTool` class and providing the tool's description as a JSON string conforming to the Open API specification. This method is useful if you already have an OpenAPI schema for your tool or if you need fine-grained control over the tool's definition. ```kotlin import com.google.ai.edge.litertlm.OpenApiTool class SampleOpenApiTool : OpenApiTool { override fun getToolDescriptionJsonString(): String { return """ { "name": "addition", "description": "Add all numbers.", "parameters": { "type": "object", "properties": { "numbers": { "type": "array", "items": { "type": "number" } }, "description": "The list of numbers to sum." }, "required": [ "numbers" ] } } """.trimIndent() // Tip: trim to save tokens } override fun execute(paramsJsonString: String): String { // Parse paramsJsonString with your choice of parser/deserializer and // execute the tool. // Return the result as a JSON string return """{"result": 1.4142}""" } } ``` #### Registering Tools Include instances of your tools in the `ConversationConfig`. ```kotlin val conversation = engine.createConversation( ConversationConfig( tools = listOf( tool(SampleToolSet()), tool(SampleOpenApiTool()), ), // ... other configs ) ) // Send messages that might trigger the tool conversation.sendMessageAsync("What's the weather like in London?", callback) ``` The model will decide when to call the tool based on the conversation. The results from the tool execution are automatically sent back to the model to generate the final response. #### Manual Tool Calling By default, tool calls generated by the model are automatically executed by LiteRT-LM and the results from the tool execution are automatically sent back to the model to generate the next response. If you want to manually execute tools and send results back to the model, you can set `automaticToolCalling` in `ConversationConfig` to `false`. ```kotlin val conversation = engine.createConversation( ConversationConfig( tools = listOf( tool(SampleOpenApiTool()), ), automaticToolCalling = false, ) ) ``` If you disable automatic tool calling, you will need to manually execute tools and send results back to the model in your application code. The `execute` method of `OpenApiTool` will **not** be called automatically when `automaticToolCalling` is set to `false`. ```kotlin // Send a message that triggers a tool call. val responseMessage = conversation.sendMessage("What's the weather like in London?") // The model returns a Message with `toolCalls` populated. if (responseMessage.toolCalls.isNotEmpty()) { val toolResponses = mutableListOf() // There can be multiple tool calls in a single response. for (toolCall in responseMessage.toolCalls) { println("Model wants to call: ${toolCall.name} with arguments: ${toolCall.arguments}") // Execute the tool manually with your own logic. `executeTool` is just an example here. val toolResponseJson = executeTool(toolCall.name, toolCall.arguments) // Collect tool responses. toolResponses.add(Content.ToolResponse(toolCall.name, toolResponseJson)) } // Use Message.tool to create the tool response message. val toolResponseMessage = Message.tool(Contents.of(toolResponses)) // Send the tool response message to the model. val finalMessage = conversation.sendMessage(toolResponseMessage) println("Final answer: ${finalMessage.text}") // e.g., "The weather in London is 25c." } ``` #### Example To try out tool use, clone the repo and run with [example/ToolMain.kt](../../../kotlin/java/com/google/ai/edge/litertlm/example/ToolMain.kt): ```bazel bazel run -c opt //kotlin/java/com/google/ai/edge/litertlm/example:tool -- ``` ### 7. Extra Template Context Variables You can pass extra context variables to the prompt template for rendering. This allows you to customize the model's behavior based on dynamic values. `extraContext` is an optional `Map` that can be passed to `sendMessage` and `sendMessageAsync`. These variables are merged with the extra context provided in the `Preface` (if any), with keys in the message-level context overwriting those in the `Preface`. ```kotlin val extraContext = mapOf( "user_name" to "Alice", "enable_thinking" to true ) // Synchronous val response = conversation.sendMessage("Hello!", extraContext = extraContext) // Asynchronous with Flow conversation.sendMessageAsync("Hello!", extraContext = extraContext) .collect { ... } ``` These variables are used within the Jinja-style prompt templates, e.g., `{{ user_name }}` or `{% if enable_thinking %}`. ## Error Handling API methods can throw `LiteRtLmJniException` for errors from the native layer or standard Kotlin exceptions like `IllegalStateException` for lifecycle issues. Always wrap API calls in try-catch blocks. The `onError` callback in `MessageCallback` will also report errors during asynchronous operations.