Instructions to use efficiencyx/Jun-LoRA-E2B-LiteRT with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use efficiencyx/Jun-LoRA-E2B-LiteRT with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="efficiencyx/Jun-LoRA-E2B-LiteRT")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("efficiencyx/Jun-LoRA-E2B-LiteRT", device_map="auto")

LiteRT

How to use efficiencyx/Jun-LoRA-E2B-LiteRT with LiteRT:

# No code snippets available yet for this library.

# To use this model, check the repository files and the library's documentation.

# Want to help? PRs adding snippets are welcome at:
# https://github.com/huggingface/huggingface.js

LiteRT-LM

How to use efficiencyx/Jun-LoRA-E2B-LiteRT with LiteRT-LM:

# LiteRT-LM runs on various platforms (Android, iOS, Windows, Linux, macOS, IoT, Web/WASM)
# and supports many APIs (C++, Python, Kotlin, Swift, JavaScript, Flutter).
# For platform-specific integration guides, please refer to the official developer website:
# https://ai.google.dev/edge/litert-lm

# To try LiteRT-LM, the easiest way is to use our CLI tool.
# 1. Install the LiteRT-LM CLI tool:
pip install -U litert-lm

# 2. Download and run this model locally:
# See: https://ai.google.dev/edge/litert-lm/cli
litert-lm run \
  --from-huggingface-repo=efficiencyx/Jun-LoRA-E2B-LiteRT \
  --prompt="Write me a poem"

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use efficiencyx/Jun-LoRA-E2B-LiteRT with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "efficiencyx/Jun-LoRA-E2B-LiteRT"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "efficiencyx/Jun-LoRA-E2B-LiteRT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/efficiencyx/Jun-LoRA-E2B-LiteRT

SGLang

How to use efficiencyx/Jun-LoRA-E2B-LiteRT with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "efficiencyx/Jun-LoRA-E2B-LiteRT" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "efficiencyx/Jun-LoRA-E2B-LiteRT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "efficiencyx/Jun-LoRA-E2B-LiteRT" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "efficiencyx/Jun-LoRA-E2B-LiteRT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use efficiencyx/Jun-LoRA-E2B-LiteRT with Docker Model Runner:
```
docker model run hf.co/efficiencyx/Jun-LoRA-E2B-LiteRT
```

Jun-LoRA-E2B-LiteRT

File size: 6,665 Bytes

2d5ee3e

---
license: apache-2.0
base_model: unsloth/gemma-4-E2B-it-qat-q4_0-unquantized
tags:
  - gemma4
  - lora
  - character
  - roleplay
  - litert
  - litert-lm
  - on-device
  - conversational
language:
  - en
library_name: transformers
pipeline_tag: text-generation
model-index:
  - name: Jun-E2B-LiteRT
    results: []
---

# Jun-E2B-LiteRT

`.litertlm` build of the Jun LoRA on [Gemma 4 E2B (QAT)](https://huggingface.co/unsloth/gemma-4-E2B-it-qat-q4_0-unquantized), for [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) — Google AI Edge's on-device runtime (Android, desktop, embedded). Same fine-tune as the [GGUF builds](https://huggingface.co/efficiencyx/Jun-LoRA-E2B-GGUF): a compact, heavily curated synthetic conversational dataset derived from the visual novel *My Dystopian Robot Girlfriend*, capturing the personality, speech patterns, and emotional nuance of the character **Jun** while preserving the base model's general reasoning and instruction-following.

The adapter is merged into the base weights — this is a standalone model. LiteRT-LM has no runtime adapter path for E2B anyway (see [Notes](#notes-and-known-behaviour)).

## Model Variants & Repositories

| Repository | Format | Description |
|:-----------|:-------|:------------|
| [`efficiencyx/Jun-LoRA-E2B-LiteRT`](https://huggingface.co/efficiencyx/Jun-LoRA-E2B-LiteRT) | `.litertlm` | This repo — merged, quantized, for LiteRT-LM |
| [`efficiencyx/Jun-LoRA-E2B-GGUF`](https://huggingface.co/efficiencyx/Jun-LoRA-E2B-GGUF) | GGUF (Q8_0 / Q6_K / Q4_K_M) | Same checkpoint, for llama.cpp |
| [`efficiencyx/Jun-LoRA-E2B-Adapter`](https://huggingface.co/efficiencyx/Jun-LoRA-E2B-Adapter) | LoRA Adapter | The adapter merged into this build, currently private |
| [`efficiencyx/Jun-LoRA-12B-GGUF`](https://huggingface.co/efficiencyx/Jun-LoRA-12B-GGUF) | GGUF | Larger sibling, same dataset |

## The Build

| | |
|:--|:--|
| File | `model.litertlm` |
| Size | 2.58 GB |
| Weights | int4 channelwise (embeddings + FFN/attention), int8 for the per-layer embedding projections |
| Activations | fp32 |
| Embedder | externalized |
| Backend | CPU (XNNPack) |
| Exported with | `litert-torch` 0.9.3 |

This matches Google's own **CPU** distribution of E2B in size and recipe. Their 2.0 GB GPU/web variants use a 2/4/8-bit mixture that no published recipe reproduces; this build is 4/8-bit only.

Text-only. The LiteRT export covers the language tower — no vision or audio encoder, unlike the GGUF `mmproj`.

## Usage

```bash
litert-lm run model.litertlm --prompt "Ciao Jun"
```

Or from the [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) C++/Android APIs, pointing at the same file.

**Supply a system prompt.** Without one the model answers as stock Gemma — the persona lives behind the Jun OS system instruction, not in the weights alone. With the CLI:

```python
# preset.py
system_instruction = open("system_prompt.txt").read()
```

```bash
litert-lm run model.litertlm --preset preset.py --prompt "Chi sei?"
```

The symbol must be lowercase `system_instruction`; `SYSTEM_INSTRUCTION` is silently ignored.

The bundled chat template is the stock Google Gemma 4 one, not Unsloth's — LiteRT-LM renders templates with minijinja, which has no `map.get()`, and Unsloth's template calls it 23 times.

## Intended Use

Conversational backend for **Jun OS**, an AI companion webapp — specifically its on-device path:

- Character-consistent multi-turn conversation, offline
- Mobile / embedded deployment where a GGUF runtime is not an option
- Research into character-faithful fine-tuning on small, high-quality datasets

### Limitations

- Specialized for a single character persona; **not** a general-purpose assistant.
- Outputs reflect fictional narrative tropes and are not factual information or advice.
- Performance degrades far outside the training distribution.
- Inherits any biases present in the Gemma 4 E2B base weights.
- At E2B scale the model is noticeably less coherent than [Jun-12B](https://huggingface.co/efficiencyx/Jun-LoRA-12B-GGUF): it can contradict itself inside a single reply and tends to lose the reply-length rule when asked for detailed explanations. The output contract (action tags, mood tags, tool calls) holds up well, including through int4 quantization.
- Text only — no image or audio input.

## Training Details

| Parameter | Value |
|:----------|:------|
| Base model | `unsloth/gemma-4-E2B-it-qat-q4_0-unquantized` |
| Method | LoRA (rsLoRA) |
| LoRA rank | 32 |
| LoRA alpha | 32 |
| LoRA dropout | 0.0 |
| Target modules | q/k/v/o + gate/up/down (language tower) |
| Checkpoint | **step 60** |
| Framework | [Unsloth](https://github.com/unslothai/unsloth) |

### Why step 60 and not 3 epochs

Training was planned for 3 epochs but the released checkpoint is **step 60**. Evaluation loss bottomed out around **1.03** at step 60 and degraded afterwards — roughly **1.38** at step 70, recovering only partially to **1.16** at step 80. Later checkpoints did not recover the step-60 quality.

Behavioural probing agreed with the loss curve. The step-70 checkpoint in particular stopped responding to the live gauge values on physical-contact turns, emitting an identical mood update whether affection was 8 or 92 and whether tension was 10 or 90 — a collapse the step-60 checkpoint does not show. Step 60 was released on that basis.

## Notes and Known Behaviour

**The merge drops 40 of 245 LoRA pairs, by necessity.** Gemma 4 E2B shares KV projections across layers 15–34, so the checkpoint contains `k_proj`/`v_proj` only for layers 0–14. Training wrapped the unused modules anyway; those deltas were never reachable at inference and are dropped at merge time. Every other pair is applied. This is the same structural fact that makes runtime `llama.cpp --lora` impossible on E2B, and why both this and the GGUF builds ship pre-merged.

**Export needed patches.** E2B's `head_dim` alternates 256/512 per layer, and `litert-torch` 0.9.3 infers a single global value — every KV cache came out 256 and attention failed. The export used a patched cache-shape inference reading `per_layer_config[i]`.

## Dataset

Synthetic conversational data derived from the visual novel *My Dystopian Robot Girlfriend*, curated for character consistency and for a structured output contract: inline `[A:...]` action tags, a trailing `[A:mood_shift|...]` bookkeeping tag, and tool calls. Roughly half of the assistant turns carry an explicit reasoning trace.

## License

Apache 2.0, inherited from the base model. The character and source material belong to their respective owners; this fine-tune is a non-commercial fan project.