Image-Text-to-Text
LiteRT-LM
LiteRT
litertlm
on-device
vision-language
object-detection
referring-expression-comprehension
object-counting
point-grounding
ocr
Instructions to use litert-community/WeVisOne-VL-4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT-LM
How to use litert-community/WeVisOne-VL-4B with LiteRT-LM:
# LiteRT-LM runs on various platforms (Android, iOS, Windows, Linux, macOS, IoT, Web/WASM) # and supports many APIs (C++, Python, Kotlin, Swift, JavaScript, Flutter). # For platform-specific integration guides, please refer to the official developer website: # https://ai.google.dev/edge/litert-lm # To try LiteRT-LM, the easiest way is to use our CLI tool. # 1. Install the LiteRT-LM CLI tool: pip install -U litert-lm # 2. Download and run this model locally: # See: https://ai.google.dev/edge/litert-lm/cli # A single .litertlm file in the repo is picked automatically; otherwise the CLI asks which one to run # (or pass its name right after the repo id). litert-lm run \ --from-huggingface-repo=litert-community/WeVisOne-VL-4B \ --prompt="Write me a poem"
- LiteRT
How to use litert-community/WeVisOne-VL-4B with LiteRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Correct three details in the card and MEASUREMENTS.md and make their wording clearer
Browse files- MEASUREMENTS.md +7 -7
- README.md +11 -11
- litertlm_manifest.json +4 -4
MEASUREMENTS.md
CHANGED
|
@@ -45,11 +45,11 @@ The variant (5,358,955,008 bytes, not uploaded) has the decoder and the token em
|
|
| 45 |
| 20:53 | this file | 109.74 tok/s | 14.83 tok/s | 9.40 s | 1.87 GB |
|
| 46 |
| 20:56 | the int8 variant | 167.05 tok/s | 11.63 tok/s | 6.22 s | 1.04 GB |
|
| 47 |
|
| 48 |
-
Peak memory is the process's peak resident memory (VmHWM). In each of the three pairs
|
| 49 |
|
| 50 |
### The int8 file on the CPU (2026-11-10)
|
| 51 |
|
| 52 |
-
The int8 file has the variant's weights; the variant is the int8 file without its 1,024-token prefill graph.
|
| 53 |
|
| 54 |
| Start | File | Picture: first token | of it, vision encoder | Picture: decode | Peak memory |
|
| 55 |
|---|---|---:|---:|---:|---:|
|
|
@@ -91,7 +91,7 @@ The int8 file gave the same answers on the CPU with LiteRT-LM 0.19.1 and 0.18.1,
|
|
| 91 |
|
| 92 |
### Agreement with the original model
|
| 93 |
|
| 94 |
-
These numbers say how far the file drifts from the original model's own answers, not whether an answer is right. The reference is the source package at `40f735b8` in float32 on the CPU, greedy, given the same 512 × 512 pictures. On 29 questions over 17 pictures (2 synthetic test pictures, 4 NASA pictures, 10 crops of the source package's example figures and its demo photo) across 9 tasks (the first seven rows of the card's template table, plus segmentation and matting, whose boxes are scored), a run is scored by how many of the reference's boxes it reproduces (IoU ≥ 0.75), its counts, its points (within 2 % of the side) and its OCR character error rate against the reference's text. The first row is the original model itself in bfloat16. The pass rule, fixed on 2026-10-27 before any 4B file was scored: at least 90 % of the reference's boxes, counts and points, and an OCR character error rate of at most 0.05; where the bfloat16 original itself misses more than 10 % of a column, that column's
|
| 95 |
|
| 96 |
| Run | Boxes (IoU ≥ 0.75) | Counts | Points (within 2 % of the side) | OCR character error rate |
|
| 97 |
|---|---:|---:|---:|---:|
|
|
@@ -104,7 +104,7 @@ These numbers say how far the file drifts from the original model's own answers,
|
|
| 104 |
|
| 105 |
The OCR difference comes from one picture, a crop of a sign from the source package's example figures that reads "HAMBURGERS": this file reads "HAMBURGERS" and the original "HAMBURGER". The variant's OCR difference is the same picture: the original reads "Image", "THOMAS", "World", "Famous", "Chili", "HAMBURGER", and the variant leaves out "Image", a small word in the picture's top right corner, and reads the other five as the original does.
|
| 106 |
|
| 107 |
-
The int8 file's OCR difference on the CPU is the variant's: it leaves out "Image" and reads the other five words as the original does. Three fresh processes on the phone's CPU gave this answer token for token: on LiteRT-LM 0.18.1 (2026-10-27 22:19, the phone at thermal status 2 with its CPU clocks capped) and on 0.19.1 (2026-11-11 00:02, thermal status 3, capped; and 01:21, thermal status 0, in the first process of a session, which built the CPU caches). The original is not near a tie there: in float32 it
|
| 108 |
|
| 109 |
Every score was computed through the source package's `parse_prediction` (`evaluation/utils/parser.py`). On the referring question whose figure answer is "No target", the file answered `<box>none</box>`, as the original does; asked to count airplanes with boxes in a radar picture that holds none, it answered one small box and `Total number: 1.`, as the original does. The answers in the card's "On-device answers" table are from 2026-11-10; the model got each original NASA file (KSC-08pd3809, KSC-20170424-PH_SWW01_0007, S67-34267) resized to 512 × 512.
|
| 110 |
|
|
@@ -112,15 +112,15 @@ Every score was computed through the source package's `parse_prediction` (`evalu
|
|
| 112 |
|
| 113 |
- **Text-only conversations past 271 tokens.** In a conversation without a picture, every token after the 271st of the conversation, prompt and answer together (the system turn and the chat template count), gets a rotary position 240 lower than the original gives it, because the decoder computes picture and text positions with one formula. On a 353-token summary request, the original model run in PyTorch with these positions followed its own answer for 11 tokens and then wrote "The bicycle evolved into the same time, …". Conversations with a picture get the original's positions.
|
| 114 |
- **The second message of a conversation.** From the second message on, LiteRT-LM passes the previous answer's stop token to the model with a zero embedding, so the second answer starts from a context the original never sees. LiteRT-LM 0.19.1 has the same code as 0.18.1 (`runtime/executor/llm_litert_compiled_model_executor.cc`, L473–509, L661–672 and L1488–1504 at v0.19.1); by that source it affects every file with a separate embedder section, as this one has. The cause and a replay that reproduces it on WeVisOne-VL-2B are in the section "A runtime defect that every file with an embedder section meets" of [`docs/deepstack-wide-embedding.md`](https://github.com/john-rocky/hf-to-litertlm/blob/main/docs/deepstack-wide-embedding.md). On this file one two-message test was run: after "My name is Ada.", the second answer to "What is my name?" was "Your name is Ada.", as the original answers.
|
| 115 |
-
- **Memory.** On 2026-11-10 the 12 GB Galaxy S26 ran 57 processes of this file through `litert_lm_advanced_main` and none was stopped. The highest peak resident memory was 4.02 GB (VmHWM 3,922,820 kB, on a picture question), and the phone's free memory (MemAvailable) fell to 0.92 GB at the lowest. The int8 file ran 91 processes on the phone's CPU (44 on LiteRT-LM 0.18.1 on 2026-10-27, 47 on 0.19.1 on 2026-11-10 and 11) and none was stopped; its highest peak was 8.00 GB (VmHWM 7,815,316 kB, on a referring question, 0.18.1), and the free memory fell to 4.
|
| 116 |
-
- **Apps and sampling.** Every run was greedy
|
| 117 |
- **Repetition penalty.** The source package generates with a repetition penalty of 1.05 (`wevisone_vl/engine.py`). No run used one: the file's sampler settings have none, and the `--repetition_penalty` option of `litert_lm_advanced_main` (default 1.0) was not tested with this file. The reference answers are plain greedy.
|
| 118 |
|
| 119 |
## The file and its use, in detail
|
| 120 |
|
| 121 |
- **Activations.** The file declares float32 activations for the decoder, so on a GPU LiteRT-LM 0.19.1 runs the decoder in float32 (`MaybeOverrideActivationType` in `runtime/engine/engine_settings.cc`) and the vision encoder at its default, also float32. The int8 file declares no activation type.
|
| 122 |
- **Sampling.** The file carries no sampler settings, so LiteRT-LM 0.19.1 decodes it greedily (top-k 1, `runtime/engine/engine_settings.cc`).
|
| 123 |
-
- **The Android program.** The v0.19.1 release has no Android binary; the card's numbers come from `litert_lm_advanced_main` built from the v0.19.1 tag for `android_arm64`, with the tag's `prebuilt/android_arm64` libraries in the same folder and `LD_LIBRARY_PATH` set to it, one process per answer.
|
| 124 |
- **Cache files.** The first process writes cache files next to the model, about 0.46 GB: 436,314,752 bytes for the vision adapter (XNNPACK) and 24,204,112 bytes of GPU programs. The int8 file's first process on the CPU writes about 5.69 GB, all XNNPACK caches: 4,039,354,952 bytes for the decoder, 1,211,999,392 for the vision encoder and 436,314,752 for the vision adapter.
|
| 125 |
- **Prompts and answers.** The source package's parsers read the answers (`parse_predictions` in `visualization.py`, and `parse_prediction` in `evaluation/utils/parser.py`). The fixed words of the card's templates are the package's, grammar included. The chat template places only the first picture of the first message; a second picture was not tried with this file. The system turn is fixed to "You are a helpful assistant.", as in the source package.
|
| 126 |
|
|
|
|
| 45 |
| 20:53 | this file | 109.74 tok/s | 14.83 tok/s | 9.40 s | 1.87 GB |
|
| 46 |
| 20:56 | the int8 variant | 167.05 tok/s | 11.63 tok/s | 6.22 s | 1.04 GB |
|
| 47 |
|
| 48 |
+
Peak memory is the process's peak resident memory (VmHWM). In each of the three pairs (this file first in two, the variant first in one), the variant reached the first token sooner and this file decoded faster. The variant's scores are under Correctness.
|
| 49 |
|
| 50 |
### The int8 file on the CPU (2026-11-10)
|
| 51 |
|
| 52 |
+
The int8 file has the variant's weights; the variant is the int8 file without its 1,024-token prefill graph. The int8 file ran with build A on the same Galaxy S26's CPU, with the card's command, its own `--model_path`, and `--backend=cpu --vision_backend=cpu --num_cpu_threads=4`, in one session (23:41–23:55): the picture question once, the text benchmark three times, then the picture question three more times. "(first)" marks the process that built the CPU caches; the card's values are the medians of the other processes (picture 3, text 3). Each process started at thermal status 1 with no CPU clock cap, after a rest of 78–174 s. During each one the phone capped the CPU clocks in most of the samples taken every 0.5 s (picture: 15–21 of 22–25 samples, lowest cap 45–47 % of a cluster's top frequency; text: 62–66 of 68–71, 32 %), so these speeds include that slowdown. Engine creation took 4.9 s in the first process and 0.28–0.41 s after it; opening the picture conversation took another 1.26 s and 0.48–1.10 s.
|
| 53 |
|
| 54 |
| Start | File | Picture: first token | of it, vision encoder | Picture: decode | Peak memory |
|
| 55 |
|---|---|---:|---:|---:|---:|
|
|
|
|
| 91 |
|
| 92 |
### Agreement with the original model
|
| 93 |
|
| 94 |
+
These numbers say how far the file drifts from the original model's own answers, not whether an answer is right. The reference is the source package at `40f735b8` in float32 on the CPU, greedy, given the same 512 × 512 pictures. On 29 questions over 17 pictures (2 synthetic test pictures, 4 NASA pictures, 10 crops of the source package's example figures and its demo photo) across 9 tasks (the first seven rows of the card's template table, plus segmentation and matting, whose boxes are scored), a run is scored by how many of the reference's boxes it reproduces (IoU ≥ 0.75), its counts, its points (within 2 % of the side) and its OCR character error rate against the reference's text. The first row is the original model itself in bfloat16. The pass rule, fixed on 2026-10-27 before any 4B file was scored: at least 90 % of the reference's boxes, counts and points, and an OCR character error rate of at most 0.05; where the bfloat16 original itself misses more than 10 % of a column, that column's threshold is 1 − 2 × its miss rate (counts: 2/3). This file passes in all three runs; the variant and the int8 file pass every column but OCR.
|
| 95 |
|
| 96 |
| Run | Boxes (IoU ≥ 0.75) | Counts | Points (within 2 % of the side) | OCR character error rate |
|
| 97 |
|---|---:|---:|---:|---:|
|
|
|
|
| 104 |
|
| 105 |
The OCR difference comes from one picture, a crop of a sign from the source package's example figures that reads "HAMBURGERS": this file reads "HAMBURGERS" and the original "HAMBURGER". The variant's OCR difference is the same picture: the original reads "Image", "THOMAS", "World", "Famous", "Chili", "HAMBURGER", and the variant leaves out "Image", a small word in the picture's top right corner, and reads the other five as the original does.
|
| 106 |
|
| 107 |
+
The int8 file's OCR difference on the CPU is the variant's: it leaves out "Image" and reads the other five words as the original does. Three fresh processes on the phone's CPU gave this answer token for token: on LiteRT-LM 0.18.1 (2026-10-27 22:19, the phone at thermal status 2 with its CPU clocks capped) and on 0.19.1 (2026-11-11 00:02, thermal status 3, capped; and 01:21, thermal status 0, in the first process of a session, which built the CPU caches). The original is not near a tie there: in float32 it gives "Image" a probability of 0.996, 6.09 logits above its second choice, "Imagine", and the int8 file's next token, "TH" of "THOMAS", is the float32 original's fourth choice, at probability 0.0003, 8.05 logits below "Image"; in bfloat16 the original puts "Image" 4.25 logits above "Imagine", and "TH" is its seventh choice, 8.25 logits below "Image". On the Mac CPU (LiteRT-LM 0.18.1 from PyPI, 2026-10-27) the int8 file read "Image" as the original does in three processes with 4 threads (one of them the 29-question run: OCR error 0.000) and left it out in two processes with 1 thread, as the phone's CPU does with 4.
|
| 108 |
|
| 109 |
Every score was computed through the source package's `parse_prediction` (`evaluation/utils/parser.py`). On the referring question whose figure answer is "No target", the file answered `<box>none</box>`, as the original does; asked to count airplanes with boxes in a radar picture that holds none, it answered one small box and `Total number: 1.`, as the original does. The answers in the card's "On-device answers" table are from 2026-11-10; the model got each original NASA file (KSC-08pd3809, KSC-20170424-PH_SWW01_0007, S67-34267) resized to 512 × 512.
|
| 110 |
|
|
|
|
| 112 |
|
| 113 |
- **Text-only conversations past 271 tokens.** In a conversation without a picture, every token after the 271st of the conversation, prompt and answer together (the system turn and the chat template count), gets a rotary position 240 lower than the original gives it, because the decoder computes picture and text positions with one formula. On a 353-token summary request, the original model run in PyTorch with these positions followed its own answer for 11 tokens and then wrote "The bicycle evolved into the same time, …". Conversations with a picture get the original's positions.
|
| 114 |
- **The second message of a conversation.** From the second message on, LiteRT-LM passes the previous answer's stop token to the model with a zero embedding, so the second answer starts from a context the original never sees. LiteRT-LM 0.19.1 has the same code as 0.18.1 (`runtime/executor/llm_litert_compiled_model_executor.cc`, L473–509, L661–672 and L1488–1504 at v0.19.1); by that source it affects every file with a separate embedder section, as this one has. The cause and a replay that reproduces it on WeVisOne-VL-2B are in the section "A runtime defect that every file with an embedder section meets" of [`docs/deepstack-wide-embedding.md`](https://github.com/john-rocky/hf-to-litertlm/blob/main/docs/deepstack-wide-embedding.md). On this file one two-message test was run: after "My name is Ada.", the second answer to "What is my name?" was "Your name is Ada.", as the original answers.
|
| 115 |
+
- **Memory.** On 2026-11-10 the 12 GB Galaxy S26 ran 57 processes of this file through `litert_lm_advanced_main` and none was stopped. The highest peak resident memory was 4.02 GB (VmHWM 3,922,820 kB, on a picture question), and the phone's free memory (MemAvailable) fell to 0.92 GB at the lowest. The int8 file ran 91 processes on the phone's CPU (44 on LiteRT-LM 0.18.1 on 2026-10-27, 47 on 0.19.1 on 2026-11-10 and 11) and none was stopped; its highest peak was 8.00 GB (VmHWM 7,815,316 kB, on a referring question, 0.18.1), and the free memory fell to 4.65 GB at the lowest (MemAvailable 4,536,636 kB, on 0.19.1). On the phone's GPU (LiteRT-LM 0.18.1, 2026-10-27) the test harness stopped a run of the int8 file when the free memory fell below its limit of 700,000 kB (MemAvailable 696,352 kB); the variant, without the 1,024-token prefill graph, is the int8 form that ran there.
|
| 116 |
+
- **Apps and sampling.** Every run was greedy. The runs in these tables went through `litert_lm_advanced_main` on the phone and the PyPI `litert-lm` package on the Mac. On 2026-11-10 one Android test app on the Maven library `com.google.ai.edge.litertlm:litertlm-android:0.19.1` (decoder and vision encoder on the GPU, `maxNumTokens = 2048`, one conversation per question, no sampler settings) asked the person-and-hat question about S67-34267, resized to 512 × 512 as on the card, in five processes on the Galaxy S26; each gave the card's answer byte for byte. No other app was tried. On the phone the int8 file ran through `litert_lm_advanced_main` only.
|
| 117 |
- **Repetition penalty.** The source package generates with a repetition penalty of 1.05 (`wevisone_vl/engine.py`). No run used one: the file's sampler settings have none, and the `--repetition_penalty` option of `litert_lm_advanced_main` (default 1.0) was not tested with this file. The reference answers are plain greedy.
|
| 118 |
|
| 119 |
## The file and its use, in detail
|
| 120 |
|
| 121 |
- **Activations.** The file declares float32 activations for the decoder, so on a GPU LiteRT-LM 0.19.1 runs the decoder in float32 (`MaybeOverrideActivationType` in `runtime/engine/engine_settings.cc`) and the vision encoder at its default, also float32. The int8 file declares no activation type.
|
| 122 |
- **Sampling.** The file carries no sampler settings, so LiteRT-LM 0.19.1 decodes it greedily (top-k 1, `runtime/engine/engine_settings.cc`).
|
| 123 |
+
- **The Android program.** The v0.19.1 release has no Android command-line binary; the card's numbers come from `litert_lm_advanced_main` built from the v0.19.1 tag for `android_arm64`, with the tag's `prebuilt/android_arm64` libraries in the same folder and `LD_LIBRARY_PATH` set to it, one process per answer.
|
| 124 |
- **Cache files.** The first process writes cache files next to the model, about 0.46 GB: 436,314,752 bytes for the vision adapter (XNNPACK) and 24,204,112 bytes of GPU programs. The int8 file's first process on the CPU writes about 5.69 GB, all XNNPACK caches: 4,039,354,952 bytes for the decoder, 1,211,999,392 for the vision encoder and 436,314,752 for the vision adapter.
|
| 125 |
- **Prompts and answers.** The source package's parsers read the answers (`parse_predictions` in `visualization.py`, and `parse_prediction` in `evaluation/utils/parser.py`). The fixed words of the card's templates are the package's, grammar included. The chat template places only the first picture of the first message; a second picture was not tried with this file. The system turn is fixed to "You are a helpful assistant.", as in the source package.
|
| 126 |
|
README.md
CHANGED
|
@@ -23,7 +23,7 @@ This repository is a conversion of [`tencent/WeVisOne-VL-4B`](https://huggingfac
|
|
| 23 |
|
| 24 |
Speed still being tuned (updates follow).
|
| 25 |
|
| 26 |
-
[`tencent/WeVisOne-VL-4B`](https://huggingface.co/tencent/WeVisOne-VL-4B) is a vision-language model by WeChat Vision, Tencent, built on Qwen3-VL. It answers detection, referring, pointing, counting and OCR requests as structured text: boxes and points are the coordinate tokens `<0>` … `<1000>`, thousandths of the picture's width and height.
|
| 27 |
|
| 28 |
| File | Bytes | sha256 | Weights |
|
| 29 |
|---|---:|---|---|
|
|
@@ -41,7 +41,7 @@ pip install pillow
|
|
| 41 |
python3 -c "from PIL import Image, ImageOps; ImageOps.exif_transpose(Image.open('photo.jpg')).convert('RGB').resize((512, 512), Image.BICUBIC).save('photo_512.png')"
|
| 42 |
```
|
| 43 |
|
| 44 |
-
On Android, build `litert_lm_advanced_main`, a command-line program of LiteRT-LM, from its v0.19.1 tag for `android_arm64` (the v0.19.1 release has no Android
|
| 45 |
|
| 46 |
```
|
| 47 |
litert_lm_advanced_main --model_path=WeVisOne-VL-4B_int4.litertlm --backend=gpu --vision_backend=gpu --max_num_tokens=2048 --max_output_tokens=1024 --async=false --use_session=false --benchmark --benchmark_prefill_tokens=0 --benchmark_decode_tokens=0 --report_peak_memory_footprint=true --input_prompt_file=<prompt file>
|
|
@@ -51,7 +51,7 @@ The first run writes about 0.46 GB of cache files next to the model. The file ca
|
|
| 51 |
|
| 52 |
The int8 file ran on the CPU with the same command, its own `--model_path`, and `--backend=cpu --vision_backend=cpu --num_cpu_threads=4` in place of the two GPU flags; its first run writes about 5.69 GB of cache files next to the model.
|
| 53 |
|
| 54 |
-
In an Android app, the Maven library `com.google.ai.edge.litertlm:litertlm-android:0.19.1` runs the file with `backend = Backend.GPU()`, `visionBackend = Backend.GPU()`, `maxNumTokens = 2048` and `maxNumImages = 1` in its `EngineConfig`, a new conversation per question, and the message `Contents.of(Content.ImageBytes(png), Content.Text(question))` with the picture resized as above. A test app built this way
|
| 55 |
|
| 56 |
Word each request with the source package's fixed template for its task ([Tencent/WeVisOne-VL](https://github.com/Tencent/WeVisOne-VL), built by its `build_prompt`); `{…}` is your text. Free-form questions about a picture were not tested.
|
| 57 |
|
|
@@ -74,7 +74,7 @@ Answers name each label in `<ref>…</ref>` and follow it with `<box>` (left, to
|
|
| 74 |
|
| 75 |
## Performance
|
| 76 |
|
| 77 |
-
Measured on a Galaxy S26 (SM-S942Q, Adreno GPU, Android 16) on 2026-11-10 with LiteRT-LM 0.19.1
|
| 78 |
|
| 79 |
| LiteRT-LM 0.19.1 build | Engine creation | Picture: first token | Picture: decode | Text: prefill | Peak memory (picture) |
|
| 80 |
|---|---:|---:|---:|---:|---:|
|
|
@@ -83,14 +83,14 @@ Measured on a Galaxy S26 (SM-S942Q, Adreno GPU, Android 16) on 2026-11-10 with L
|
|
| 83 |
|
| 84 |
Each time and speed is the median of two processes, run after a first process built the GPU caches; memory is the range of the two. After engine creation, opening the picture conversation took another 4.9 s (A) and 5.4–5.7 s (B). On the Galaxy S26, in both pairs of processes (run in the order A, B, B, A), build A prefilled the text faster and build B, which loads the prebuilt `libLiteRtGpuAccelerator.so`, reached the picture's first token sooner; the answers were checked with build A only. Method, raw values and how build B is made: [MEASUREMENTS.md](https://huggingface.co/litert-community/WeVisOne-VL-4B/blob/main/MEASUREMENTS.md).
|
| 85 |
|
| 86 |
-
In two later sessions on the same phone with build A, this file
|
| 87 |
|
| 88 |
| File | Picture: first token | Picture: decode | Text: prefill | Text: decode |
|
| 89 |
|---|---:|---:|---:|---:|
|
| 90 |
| This file | 4.61 s | 15.2 tok/s | 109.74 tok/s | 14.83 tok/s |
|
| 91 |
| The int8 variant (not uploaded) | 3.53 s | 11.3 tok/s | 167.05 tok/s | 11.63 tok/s |
|
| 92 |
|
| 93 |
-
In all three pairs the variant reached the first token sooner and this file decoded faster. This file stays
|
| 94 |
|
| 95 |
On the same phone's CPU, with build A, 4 threads and the vision encoder on the CPU too, the int8 file ran in one session (2026-11-10, 23:41–23:55):
|
| 96 |
|
|
@@ -98,7 +98,7 @@ On the same phone's CPU, with build A, 4 threads and the vision encoder on the C
|
|
| 98 |
|---|---:|---:|---:|---:|---:|---:|
|
| 99 |
| `WeVisOne-VL-4B_int8.litertlm`, CPU | 0.39 s | 4.49 s | 10.25 tok/s | 213.12 tok/s | 8.24 tok/s | 7.26–7.48 GB |
|
| 100 |
|
| 101 |
-
Each
|
| 102 |
|
| 103 |
## Correctness
|
| 104 |
|
|
@@ -116,7 +116,7 @@ These are 21 questions with a known answer: 11 about two drawn test pictures and
|
|
| 116 |
| This file (the same three runs) | 57/58 | 6/6 | 13/14 | 0.016 |
|
| 117 |
| The int8 file (the same two runs) | 57/58 | 6/6 | 14/14 | 0.098 |
|
| 118 |
|
| 119 |
-
This second table shows how closely the answers to 29 picture questions follow the original model's own answers in float32, not whether they are right. The OCR difference is one sign that reads "HAMBURGERS": this file reads it right and the original drops the S. The int8 file's OCR difference is
|
| 120 |
|
| 121 |
## On-device answers (Galaxy S26 GPU, LiteRT-LM 0.19.1)
|
| 122 |
|
|
@@ -133,8 +133,8 @@ The model got each original NASA file resized to 512 × 512; the pictures shown
|
|
| 133 |
- **512 × 512 only.** The file always reads 512 × 512, and every comparison on this card is at that size; the source package chooses the resolution per picture, and how far the original's answers at its own resolution differ was not measured for the 4B. Resize your picture as in Use it.
|
| 134 |
- **Keep text-only conversations within 271 tokens.** Without a picture, every token after the 271st of the conversation (prompt, answer and chat template together) gets a wrong position, and from there the answer can differ from the original's. Conversations with a picture are not affected.
|
| 135 |
- **One question per conversation.** From the second message on, LiteRT-LM passes the previous answer's stop token to the model with a zero embedding, so the second answer starts from a context the original never sees (the cause, and a replay on WeVisOne-VL-2B: [`docs/deepstack-wide-embedding.md`](https://github.com/john-rocky/hf-to-litertlm/blob/main/docs/deepstack-wide-embedding.md#a-runtime-defect-that-every-file-with-an-embedder-section-meets)).
|
| 136 |
-
- **Memory.**
|
| 137 |
-
- **Not tested:** this file on a CPU, the int8 file in an app (it ran through `litert_lm_advanced_main` only), desktop runs on LiteRT-LM 0.19.1, apps other than the one test app in Use it, sampled answers, and the source package's repetition penalty of 1.05. There is no NPU build.
|
| 138 |
|
| 139 |
Details: [MEASUREMENTS.md](https://huggingface.co/litert-community/WeVisOne-VL-4B/blob/main/MEASUREMENTS.md).
|
| 140 |
|
|
@@ -144,7 +144,7 @@ Details: [MEASUREMENTS.md](https://huggingface.co/litert-community/WeVisOne-VL-4
|
|
| 144 |
- **Qwen3-VL's positions, computed in the graph** from the runtime's one position per token and an image flag. With a picture they equal the original's at every position of the test questions.
|
| 145 |
- **The vision tower rewritten in plain PyTorch** for a fixed 512 × 512 input, without a gather op, and converted to [LiteRT](https://github.com/google-ai-edge/litert) graphs with float16 weights. On the Galaxy S26 the whole encoder runs on the GPU.
|
| 146 |
- **int4 decoder.** ai-edge-quantizer 0.10.0 put the decoder's weights, its output layer included, in int4 blocks of 32 with one scale per block, every block scale kept at or above 2^-14, and the token embedder in int8 (one scale per output channel). The file declares float32 activations for the decoder: without the declaration the prefill on the Galaxy S26 GPU was faster, but one of the ten text questions was answered wrong on the Mac GPU.
|
| 147 |
-
- **int8 file.** The same export with ai-edge-quantizer 0.10.0's dynamic int8 recipe (one scale per output channel) on the decoder, its output layer and the token embedder, without the activation declaration and the GPU graph step.
|
| 148 |
- **GPU graph.** After the export, [`gpu_graph_retrofit.py`](https://github.com/john-rocky/hf-to-litertlm/blob/main/tools/gpu_graph/gpu_graph_retrofit.py) in hf-to-litertlm rewrote the decoder's KV-cache writes and attention products into composite ops that LiteRT-LM's GPU path recognizes (`odml.cache_update`, `odml.runtime_bmm`), with every weight byte the same. On the Galaxy S26 GPU the decode and the picture's first token got faster; on the Mac GPU the answers stayed the same.
|
| 149 |
|
| 150 |
The conversion started from the source's float32 release (revision `4a7bed92`); its current bfloat16 release (`6b44a458`) holds the same weights. It follows the WeVisOne-VL-2B recipe in [hf-to-litertlm](https://github.com/john-rocky/hf-to-litertlm) ([REPRODUCE.md](https://github.com/john-rocky/hf-to-litertlm/blob/main/REPRODUCE.md), the WeVisOne-VL-2B entry, and [`docs/deepstack-wide-embedding.md`](https://github.com/john-rocky/hf-to-litertlm/blob/main/docs/deepstack-wide-embedding.md)) at the 4B's dimensions, with the int4 decoder and the GPU graph step above. REPRODUCE.md has no entry for the 4B itself yet.
|
|
|
|
| 23 |
|
| 24 |
Speed still being tuned (updates follow).
|
| 25 |
|
| 26 |
+
[`tencent/WeVisOne-VL-4B`](https://huggingface.co/tencent/WeVisOne-VL-4B) is a vision-language model by WeChat Vision, Tencent, built on Qwen3-VL. It answers detection, referring, pointing, counting and OCR requests as structured text: boxes and points are the coordinate tokens `<0>` … `<1000>`, thousandths of the picture's width and height. The files here do not do segmentation or matting: those need the source model's 9.6M-parameter pixel decoder, which they do not contain.
|
| 27 |
|
| 28 |
| File | Bytes | sha256 | Weights |
|
| 29 |
|---|---:|---|---|
|
|
|
|
| 41 |
python3 -c "from PIL import Image, ImageOps; ImageOps.exif_transpose(Image.open('photo.jpg')).convert('RGB').resize((512, 512), Image.BICUBIC).save('photo_512.png')"
|
| 42 |
```
|
| 43 |
|
| 44 |
+
On Android, build `litert_lm_advanced_main`, a command-line program of LiteRT-LM, from its v0.19.1 tag for `android_arm64` (the v0.19.1 release has no Android build of it), put the tag's `prebuilt/android_arm64` libraries in the same folder and set `LD_LIBRARY_PATH` to it. Each answer on this card from this program is one process with these flags (the int8 file's CPU flags are below). The prompt file holds `[image:<path to photo_512.png>]` and then the question, with nothing between them and no newline at the end:
|
| 45 |
|
| 46 |
```
|
| 47 |
litert_lm_advanced_main --model_path=WeVisOne-VL-4B_int4.litertlm --backend=gpu --vision_backend=gpu --max_num_tokens=2048 --max_output_tokens=1024 --async=false --use_session=false --benchmark --benchmark_prefill_tokens=0 --benchmark_decode_tokens=0 --report_peak_memory_footprint=true --input_prompt_file=<prompt file>
|
|
|
|
| 51 |
|
| 52 |
The int8 file ran on the CPU with the same command, its own `--model_path`, and `--backend=cpu --vision_backend=cpu --num_cpu_threads=4` in place of the two GPU flags; its first run writes about 5.69 GB of cache files next to the model.
|
| 53 |
|
| 54 |
+
In an Android app, the Maven library `com.google.ai.edge.litertlm:litertlm-android:0.19.1` runs the int4 file with `backend = Backend.GPU()`, `visionBackend = Backend.GPU()`, `maxNumTokens = 2048` and `maxNumImages = 1` in its `EngineConfig`, a new conversation per question, and the message `Contents.of(Content.ImageBytes(png), Content.Text(question))` with the picture resized as above. A test app built this way reproduced, byte for byte, the person-and-hat answer in the On-device answers table below, on the same Galaxy S26 (2026-11-10).
|
| 55 |
|
| 56 |
Word each request with the source package's fixed template for its task ([Tencent/WeVisOne-VL](https://github.com/Tencent/WeVisOne-VL), built by its `build_prompt`); `{…}` is your text. Free-form questions about a picture were not tested.
|
| 57 |
|
|
|
|
| 74 |
|
| 75 |
## Performance
|
| 76 |
|
| 77 |
+
Measured on a Galaxy S26 (SM-S942Q, Adreno GPU, Android 16) on 2026-11-10 with LiteRT-LM 0.19.1; the first two tables ran the decoder and the vision encoder on the GPU. Picture: a 512 × 512 picture and a detection question (297 prompt tokens, 60 answer tokens). Text: the runtime's speed benchmark with 1,024 prompt tokens; it measures speed only (a text-only prompt that long gets wrong positions, see Limits).
|
| 78 |
|
| 79 |
| LiteRT-LM 0.19.1 build | Engine creation | Picture: first token | Picture: decode | Text: prefill | Peak memory (picture) |
|
| 80 |
|---|---:|---:|---:|---:|---:|
|
|
|
|
| 83 |
|
| 84 |
Each time and speed is the median of two processes, run after a first process built the GPU caches; memory is the range of the two. After engine creation, opening the picture conversation took another 4.9 s (A) and 5.4–5.7 s (B). On the Galaxy S26, in both pairs of processes (run in the order A, B, B, A), build A prefilled the text faster and build B, which loads the prebuilt `libLiteRtGpuAccelerator.so`, reached the picture's first token sooner; the answers were checked with build A only. Method, raw values and how build B is made: [MEASUREMENTS.md](https://huggingface.co/litert-community/WeVisOne-VL-4B/blob/main/MEASUREMENTS.md).
|
| 85 |
|
| 86 |
+
In two later sessions on the same phone with build A, this file took turns with an int8 variant made for the GPU (the int8 file without its 1,024-token prefill graph; 5.36 GB, not uploaded): this file, the variant, the variant, this file, this file, the variant (2026-11-10). Each value is the median of three processes (the variant's picture values: four). This file's first token took 4.61 s in these sessions and 4.86 s in the table above.
|
| 87 |
|
| 88 |
| File | Picture: first token | Picture: decode | Text: prefill | Text: decode |
|
| 89 |
|---|---:|---:|---:|---:|
|
| 90 |
| This file | 4.61 s | 15.2 tok/s | 109.74 tok/s | 14.83 tok/s |
|
| 91 |
| The int8 variant (not uploaded) | 3.53 s | 11.3 tok/s | 167.05 tok/s | 11.63 tok/s |
|
| 92 |
|
| 93 |
+
In all three pairs the variant reached the first token sooner and this file decoded faster. This file stays this repository's file for a GPU: it is 1.85 GB smaller and follows the original's answers more closely (boxes 57 of 58 against the variant's 54, OCR error 0.016 against 0.098). Each value and how the variant's answers scored: [MEASUREMENTS.md](https://huggingface.co/litert-community/WeVisOne-VL-4B/blob/main/MEASUREMENTS.md).
|
| 94 |
|
| 95 |
On the same phone's CPU, with build A, 4 threads and the vision encoder on the CPU too, the int8 file ran in one session (2026-11-10, 23:41–23:55):
|
| 96 |
|
|
|
|
| 98 |
|---|---:|---:|---:|---:|---:|---:|
|
| 99 |
| `WeVisOne-VL-4B_int8.litertlm`, CPU | 0.39 s | 4.49 s | 10.25 tok/s | 213.12 tok/s | 8.24 tok/s | 7.26–7.48 GB |
|
| 100 |
|
| 101 |
+
Each time and speed is the median of three processes, run after a first one that built the CPU caches (4.9 s of engine creation), and memory is the range of the three; after engine creation, opening the picture conversation took another 0.61 s. Each process started with no limit on the CPU's frequency, and the phone set one during each run, so these speeds include that slowdown.
|
| 102 |
|
| 103 |
## Correctness
|
| 104 |
|
|
|
|
| 116 |
| This file (the same three runs) | 57/58 | 6/6 | 13/14 | 0.016 |
|
| 117 |
| The int8 file (the same two runs) | 57/58 | 6/6 | 14/14 | 0.098 |
|
| 118 |
|
| 119 |
+
This second table shows how closely the answers to 29 picture questions follow the original model's own answers in float32, not whether they are right. The OCR difference is one sign that reads "HAMBURGERS": this file reads it right and the original drops the S. The int8 file's OCR difference is in the same picture: on the Galaxy S26's CPU it leaves out "Image", a small label in the picture's corner that the original reads first, and reads the other five words as the original does; three separate processes gave this answer. This one answer, 1 of the 29, puts the int8 file's OCR error at 0.098, above the 0.05 that this table's pass rule allows. The questions, the pass rule and each run: [MEASUREMENTS.md](https://huggingface.co/litert-community/WeVisOne-VL-4B/blob/main/MEASUREMENTS.md).
|
| 120 |
|
| 121 |
## On-device answers (Galaxy S26 GPU, LiteRT-LM 0.19.1)
|
| 122 |
|
|
|
|
| 133 |
- **512 × 512 only.** The file always reads 512 × 512, and every comparison on this card is at that size; the source package chooses the resolution per picture, and how far the original's answers at its own resolution differ was not measured for the 4B. Resize your picture as in Use it.
|
| 134 |
- **Keep text-only conversations within 271 tokens.** Without a picture, every token after the 271st of the conversation (prompt, answer and chat template together) gets a wrong position, and from there the answer can differ from the original's. Conversations with a picture are not affected.
|
| 135 |
- **One question per conversation.** From the second message on, LiteRT-LM passes the previous answer's stop token to the model with a zero embedding, so the second answer starts from a context the original never sees (the cause, and a replay on WeVisOne-VL-2B: [`docs/deepstack-wide-embedding.md`](https://github.com/john-rocky/hf-to-litertlm/blob/main/docs/deepstack-wide-embedding.md#a-runtime-defect-that-every-file-with-an-embedder-section-meets)).
|
| 136 |
+
- **Memory.** On LiteRT-LM 0.19.1 the 12 GB Galaxy S26 ran 57 processes of this file (highest peak memory 4.02 GB); on 0.18.1 and 0.19.1 it ran 91 of the int8 file on its CPU (8.00 GB); none was stopped. On that phone's GPU, the test harness stopped a run of the int8 file when the free memory fell to 0.71 GB (LiteRT-LM 0.18.1). Phones with less memory were not tried.
|
| 137 |
+
- **Not tested:** this file on a CPU, the int8 file in an app (on the phone it ran through `litert_lm_advanced_main` only), desktop runs on LiteRT-LM 0.19.1, apps other than the one test app in Use it, sampled answers, and the source package's repetition penalty of 1.05. There is no NPU build.
|
| 138 |
|
| 139 |
Details: [MEASUREMENTS.md](https://huggingface.co/litert-community/WeVisOne-VL-4B/blob/main/MEASUREMENTS.md).
|
| 140 |
|
|
|
|
| 144 |
- **Qwen3-VL's positions, computed in the graph** from the runtime's one position per token and an image flag. With a picture they equal the original's at every position of the test questions.
|
| 145 |
- **The vision tower rewritten in plain PyTorch** for a fixed 512 × 512 input, without a gather op, and converted to [LiteRT](https://github.com/google-ai-edge/litert) graphs with float16 weights. On the Galaxy S26 the whole encoder runs on the GPU.
|
| 146 |
- **int4 decoder.** ai-edge-quantizer 0.10.0 put the decoder's weights, its output layer included, in int4 blocks of 32 with one scale per block, every block scale kept at or above 2^-14, and the token embedder in int8 (one scale per output channel). The file declares float32 activations for the decoder: without the declaration the prefill on the Galaxy S26 GPU was faster, but one of the ten text questions was answered wrong on the Mac GPU.
|
| 147 |
+
- **int8 file.** The same export with ai-edge-quantizer 0.10.0's dynamic int8 recipe (one scale per output channel) on the decoder, its output layer and the token embedder, without the activation declaration and the GPU graph step below.
|
| 148 |
- **GPU graph.** After the export, [`gpu_graph_retrofit.py`](https://github.com/john-rocky/hf-to-litertlm/blob/main/tools/gpu_graph/gpu_graph_retrofit.py) in hf-to-litertlm rewrote the decoder's KV-cache writes and attention products into composite ops that LiteRT-LM's GPU path recognizes (`odml.cache_update`, `odml.runtime_bmm`), with every weight byte the same. On the Galaxy S26 GPU the decode and the picture's first token got faster; on the Mac GPU the answers stayed the same.
|
| 149 |
|
| 150 |
The conversion started from the source's float32 release (revision `4a7bed92`); its current bfloat16 release (`6b44a458`) holds the same weights. It follows the WeVisOne-VL-2B recipe in [hf-to-litertlm](https://github.com/john-rocky/hf-to-litertlm) ([REPRODUCE.md](https://github.com/john-rocky/hf-to-litertlm/blob/main/REPRODUCE.md), the WeVisOne-VL-2B entry, and [`docs/deepstack-wide-embedding.md`](https://github.com/john-rocky/hf-to-litertlm/blob/main/docs/deepstack-wide-embedding.md)) at the 4B's dimensions, with the int4 decoder and the GPU graph step above. REPRODUCE.md has no entry for the 4B itself yet.
|
litertlm_manifest.json
CHANGED
|
@@ -184,8 +184,8 @@
|
|
| 184 |
},
|
| 185 |
{
|
| 186 |
"file": "WeVisOne-VL-4B_int8.litertlm",
|
| 187 |
-
"size_bytes": 5358955008,
|
| 188 |
"sha256": "277492d63a16615152d3445e3748126f4c7b60adea224365da1505f90d211881",
|
|
|
|
| 189 |
"sections": [
|
| 190 |
{
|
| 191 |
"type": "LlmMetadataProto",
|
|
@@ -231,7 +231,7 @@
|
|
| 231 |
"platform": "android",
|
| 232 |
"device_class": "12 GB flagship (Galaxy S26, Qualcomm SM8850 CPU)",
|
| 233 |
"backend": "cpu",
|
| 234 |
-
"reason": "Galaxy S26 CPU (4 threads, vision on the CPU), litert_lm_advanced_main built from the LiteRT-LM v0.19.1 tag: a 512x512 picture + question to first token 4.49 s, decode 10.25 tok/s; text 1,024 / 256: 213.12 / 8.24 tok/s (medians of 3;
|
| 235 |
}
|
| 236 |
],
|
| 237 |
"requirements": {
|
|
@@ -264,7 +264,7 @@
|
|
| 264 |
"cache": "disk",
|
| 265 |
"runs": 3,
|
| 266 |
"date": "2026-11-10",
|
| 267 |
-
"source": "a 512x512 drawn test picture + a detection question; one session, the 3 picture processes after the first one (which built the XNNPACK caches: engine creation 4.9 s); ttft_s = the vision encoder's 1.24-2.07 s + the runtime's time to first token 2.16-3.25 s; opening the picture conversation took another 0.48-1.10 s (Init Conversation); each process started at thermal status 1 with no CPU clock cap, and the phone capped the CPU clocks in 15-21 of 22-25 samples (every 0.5 s) of each run, down to 45-47 % of a cluster's top frequency
|
| 268 |
},
|
| 269 |
{
|
| 270 |
"device": "Galaxy S26 (SM-S942Q, Android 16, Qualcomm SM8850)",
|
|
@@ -281,7 +281,7 @@
|
|
| 281 |
"cache": "disk",
|
| 282 |
"runs": 3,
|
| 283 |
"date": "2026-11-10",
|
| 284 |
-
"source": "forced 1,024-token prefill and 256 decoded tokens (--benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256); the same session, 3 processes after the picture question's first process built the caches; each started at thermal status 1 with no CPU clock cap, and the phone capped the CPU clocks in 62-66 of 68-71 samples of each run, down to 32 % of a cluster's top frequency
|
| 285 |
}
|
| 286 |
],
|
| 287 |
"checked": [
|
|
|
|
| 184 |
},
|
| 185 |
{
|
| 186 |
"file": "WeVisOne-VL-4B_int8.litertlm",
|
|
|
|
| 187 |
"sha256": "277492d63a16615152d3445e3748126f4c7b60adea224365da1505f90d211881",
|
| 188 |
+
"size_bytes": 5358955008,
|
| 189 |
"sections": [
|
| 190 |
{
|
| 191 |
"type": "LlmMetadataProto",
|
|
|
|
| 231 |
"platform": "android",
|
| 232 |
"device_class": "12 GB flagship (Galaxy S26, Qualcomm SM8850 CPU)",
|
| 233 |
"backend": "cpu",
|
| 234 |
+
"reason": "Galaxy S26 CPU (4 threads, vision on the CPU), litert_lm_advanced_main built from the LiteRT-LM v0.19.1 tag: a 512x512 picture + question to first token 4.49 s, decode 10.25 tok/s; text 1,024 / 256: 213.12 / 8.24 tok/s (medians of 3; the phone capped the CPU clocks during each run, and these speeds include that slowdown). Engine creation 0.39 s after the first process, then 0.61 s to open a picture conversation. Known-answer test: the same score as the float32 original (11 picture and 10 text questions); against the original's own answers it leaves out one word ('Image') on 1 of 29 picture questions (OCR character error rate 0.098)."
|
| 235 |
}
|
| 236 |
],
|
| 237 |
"requirements": {
|
|
|
|
| 264 |
"cache": "disk",
|
| 265 |
"runs": 3,
|
| 266 |
"date": "2026-11-10",
|
| 267 |
+
"source": "a 512x512 drawn test picture + a detection question; one session, the 3 picture processes after the first one (which built the XNNPACK caches: engine creation 4.9 s); ttft_s = the vision encoder's 1.24-2.07 s + the runtime's time to first token 2.16-3.25 s; opening the picture conversation took another 0.48-1.10 s (Init Conversation); each process started at thermal status 1 with no CPU clock cap, and the phone capped the CPU clocks in 15-21 of 22-25 samples (every 0.5 s) of each run, down to 45-47 % of a cluster's top frequency, and these speeds include that slowdown; peak = the larger VmHWM / 1024"
|
| 268 |
},
|
| 269 |
{
|
| 270 |
"device": "Galaxy S26 (SM-S942Q, Android 16, Qualcomm SM8850)",
|
|
|
|
| 281 |
"cache": "disk",
|
| 282 |
"runs": 3,
|
| 283 |
"date": "2026-11-10",
|
| 284 |
+
"source": "forced 1,024-token prefill and 256 decoded tokens (--benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256); the same session, 3 processes after the picture question's first process built the caches; each started at thermal status 1 with no CPU clock cap, and the phone capped the CPU clocks in 62-66 of 68-71 samples of each run, down to 32 % of a cluster's top frequency, and these speeds include that slowdown; peak = the larger VmHWM / 1024"
|
| 285 |
}
|
| 286 |
],
|
| 287 |
"checked": [
|