mlboydaisuke commited on
Commit
0fd5532
·
verified ·
1 Parent(s): e8bac17

Card: verified one-command usage, measured Performance table (backends gated on real generation), accuracy note

Browse files
Files changed (1) hide show
  1. README.md +37 -1
README.md CHANGED
@@ -37,7 +37,43 @@ tiny, fast on-device VLM — give it an image and a question, get a grounded ans
37
  | **Image input** | resized to 448×448 (ImageNet normalization baked into the vision encoder) |
38
  | **Base model** | OpenGVLab/InternVL3-1B |
39
 
40
- ## Quality
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
41
 
42
  Output is coherent and image-grounded (CPU-verified; the vision tower converts bit-faithfully to the
43
  reference, float CPU-parity corr ≈ 1.0). On-device behavior mirrors the larger InternVL3-2B build
 
37
  | **Image input** | resized to 448×448 (ImageNet normalization baked into the vision encoder) |
38
  | **Base model** | OpenGVLab/InternVL3-1B |
39
 
40
+ ## How to use
41
+
42
+ **1. Install the runtime**
43
+
44
+ ```bash
45
+ pip install litert-lm
46
+ ```
47
+
48
+ **2. Run it in one command** — this downloads the bundle, encodes your image and answers:
49
+
50
+ ```bash
51
+ litert-lm run --from-huggingface-repo litert-community/InternVL3-1B InternVL3-1B.litertlm \
52
+ --attachment photo.jpg \
53
+ --prompt "Describe this image in one sentence."
54
+ ```
55
+
56
+ On the COCO sample image `huggingface/documentation-images/coco_sample.png` (two tabby cats on a pink blanket, remote controls beside them) this prints:
57
+
58
+ ```
59
+ The image shows two cats lying on a pink bed with two remote controls.
60
+ ```
61
+
62
+ Drop `--prompt` for an interactive chat, and pass `--attachment` more than once for several images. `litert-lm serve` exposes the same bundle as a local OpenAI-compatible API. The same file runs on macOS, Linux and Windows.
63
+
64
+ ## Performance
65
+
66
+ `litert-lm benchmark` (litert-lm 0.15.0) on an Apple M4 Max, `-p 256 -d 256 --runs 3` (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine. These figures cover the text path; the vision encoder runs once per image and is not included.
67
+
68
+ | Device | Backend | Prefill (256) | Decode | TTFT |
69
+ |---|---|---|---|---|
70
+ | Apple M4 Max (macOS) | CPU | 454 tok/s | 94.0 tok/s | 0.62 s |
71
+
72
+ Every desktop backend listed above was checked by actually generating a caption on it, not just by reading the benchmark tool's output.
73
+
74
+ **The macOS GPU backend is not usable for this bundle on litert-lm 0.15.0.** Asked to describe an image with `--backend gpu`, it returns a run of `!` characters instead of a caption. `litert-lm benchmark` still reports GPU numbers for it — 5,583 tok/s prefill and 342 tok/s decode — but they measure a path that produces no usable text, so no GPU row is quoted here. Use the CPU backend on the desktop. This is a desktop-runtime observation and says nothing about the iPhone or Android GPU paths described elsewhere on this card.
75
+
76
+ ### Accuracy note
77
 
78
  Output is coherent and image-grounded (CPU-verified; the vision tower converts bit-faithfully to the
79
  reference, float CPU-parity corr ≈ 1.0). On-device behavior mirrors the larger InternVL3-2B build