Ben Graville commited on
Commit
4d4168e
Β·
verified Β·
1 Parent(s): c55cf96

docs: descriptive model card

Browse files
Files changed (1) hide show
  1. README.md +149 -0
README.md ADDED
@@ -0,0 +1,149 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: gemma
3
+ base_model: google/embeddinggemma-300m
4
+ library_name: coreml
5
+ pipeline_tag: sentence-similarity
6
+ tags:
7
+ - coreml
8
+ - core-ml
9
+ - apple-silicon
10
+ - on-device
11
+ - ane
12
+ - quantized
13
+ - int8
14
+ - embeddings
15
+ - sentence-embedding
16
+ ---
17
+
18
+ # embeddinggemma-300m β€” Core ML (int8, seq 128, ANE)
19
+
20
+ `google/embeddinggemma-300m` as a compiled Core ML encoder for Apple silicon, mirrored and
21
+ independently rebuild-verified by the [Visible](https://visible.cx) project. **This is the
22
+ embedding model Visible actually serves.** Core ML is the production embedding path in the
23
+ app; the LiteRT artifact is retained as a benchmark instrument and never as a serving
24
+ fallback, and the Core AI embedding bundle only runs in a specific build configuration on
25
+ macOS 27+.
26
+
27
+ It is also the one artifact in this entire catalog whose weights are **provably
28
+ reproducible**: a rebuild from the recipe on Linux produced a `weight.bin` that is
29
+ byte-for-byte identical to the published one.
30
+
31
+ ## Provenance
32
+
33
+ | | |
34
+ |---|---|
35
+ | Base checkpoint | `google/embeddinggemma-300m` |
36
+ | Upstream bundle | `mlboydaisuke/embeddinggemma-300m-coreml` @ `3fa12f0b97b8afe23264f76800afe14af4615ca5` |
37
+ | Recipe | `john-rocky/CoreML-LLM` β†’ `conversion/build_embeddinggemma_bundle.py --max-seq-len 128 --quantize int8` |
38
+ | Format | Core ML `.mlmodelc` (compiled), int8 weights |
39
+ | Sequence length | **128** (static) |
40
+ | Output | 768-d embedding |
41
+ | ANE residency | **~99.80%**, 1950/1954 ops (published claim; not re-measured by Visible) |
42
+ | Pinned in the app as | `CoreMLEmbeddingGemmaBundle` |
43
+
44
+ The mirror was verified byte-exact against upstream on all six files of the published
45
+ artifact before this card was written.
46
+
47
+ ## Contents
48
+
49
+ Two top-level directories. The first is the artifact; the second is the evidence.
50
+
51
+ **`3fa12f0b97b8afe23264f76800afe14af4615ca5/`** β€” the pinned upstream artifact, byte for
52
+ byte, under its own upstream revision as the directory name. 309,358,096 B total.
53
+
54
+ | File | Bytes |
55
+ |---|---:|
56
+ | `encoder.mlmodelc/weights/weight.bin` | 308,616,576 |
57
+ | `encoder.mlmodelc/model.mil` | 735,948 |
58
+ | `encoder.mlmodelc/metadata.json` | 2,570 |
59
+ | `encoder.mlmodelc/coremldata.bin` | 408 |
60
+ | `encoder.mlmodelc/analytics/coremldata.bin` | 243 |
61
+ | `model_config.json` | 2,351 |
62
+
63
+ **`rebuild-verification/2026-08-17/`** β€” an independent rebuild of the same recipe on a
64
+ different operating system and CPU architecture, published so the reproducibility claim
65
+ below can be checked rather than taken on trust. 309,346,242 B total.
66
+
67
+ | File | Bytes |
68
+ |---|---:|
69
+ | `encoder.mlpackage/Data/com.apple.CoreML/weights/weight.bin` | 308,616,576 |
70
+ | `encoder.mlpackage/Data/com.apple.CoreML/model.mlmodel` | 723,393 |
71
+ | `encoder.mlpackage/Manifest.json` | 617 |
72
+ | `model_config.json` | 2,351 |
73
+ | `SHA256SUMS` | 427 |
74
+ | `VERIFICATION.md` | 2,878 |
75
+
76
+ Note the shapes differ deliberately: the pinned artifact is a **compiled** `.mlmodelc`, the
77
+ rebuild is the **uncompiled** `.mlpackage` the recipe emits. Compilation is a macOS-only
78
+ step (`xcrun coremlcompiler`) and was not faked on Linux.
79
+
80
+ ## Requirements
81
+
82
+ - **Apple silicon**, Core ML runtime. The encoder is shaped for the Neural Engine.
83
+ - **Static sequence length 128.** Inputs must be padded or truncated to 128 tokens; this is
84
+ a compile-time property of the artifact, not a runtime option.
85
+ - Weights β‰ˆ 0.31 GB resident. This is a small model by every measure in this catalog and
86
+ imposes no meaningful memory tier.
87
+ - `model_config.json` beside the encoder carries the pooling/dense/normalisation contract
88
+ the host must honour: mean pooling β†’ dense stack β†’ L2 normalise, 768-d out.
89
+
90
+ ## Measured performance
91
+
92
+ **No throughput or latency figure is published here.** The rebuild verification was run on
93
+ a Linux conversion box, where Core ML cannot execute at all
94
+ (`coremltools.libcoremlpython` does not load), so no vector was computed and no timing was
95
+ taken during that work.
96
+
97
+ The ANE residency figure (99.80%, 1950/1954 ops) is the **upstream published claim**, not a
98
+ Visible measurement. A re-measure via `MLComputePlan` is an open follow-up.
99
+
100
+ The last cross-runtime quality datapoint on record is a cosine of **~0.966** between the
101
+ Core ML leg and the LiteRT leg on short text, measured 2026-08-11 against the previously
102
+ installed copies β€” not against the artifacts in this repo.
103
+
104
+ ## Qualification status
105
+
106
+ | Artifact | Status |
107
+ |---|---|
108
+ | `3fa12f0b…/encoder.mlmodelc` + `model_config.json` | **SHIP** β€” this is the production embedding path in the Visible app, pinned by `CoreMLEmbeddingGemmaBundle`, with the compiled-in `weight.bin` SHA-256 matching. |
109
+ | `rebuild-verification/2026-08-17/encoder.mlpackage` | **VERIFICATION EVIDENCE, not a runtime artifact.** It is uncompiled and has never been executed. Compiling it and diffing the result against the pinned `.mlmodelc` is a named open gate (below). |
110
+
111
+ ## Verification β€” the bit-exact result, stated precisely
112
+
113
+ The recipe was re-run on **Debian 13 / x86_64 / torch 2.7.1+cpu / coremltools 9.0 /
114
+ transformers 5.15.0**, against a bundle originally built on **macOS**. What reproduced:
115
+
116
+ | File class | Verdict |
117
+ |---|---|
118
+ | `weights/weight.bin` (308,616,576 B) | **IDENTICAL** β€” same SHA-256 `f81f60eb…`, `cmp -l` reports **0 differing bytes**. Also equal to the app's compiled-in `CoreMLEmbeddingGemmaBundle.weightSha256`. |
119
+ | `model_config.json` (2,351 B) | **IDENTICAL** β€” SHA-256 `0b949875…` |
120
+ | tokenizer / config JSON emitted by the recipe | **IDENTICAL** β€” all files |
121
+ | `encoder.mlmodelc/model.mil`, `coremldata.bin` Γ—2, `metadata.json` | **not produced on Linux** β€” these are products of `xcrun coremlcompiler`, which is Xcode/macOS-only |
122
+
123
+ So the honest claim is: **everything the recipe produces reproduces exactly, across
124
+ operating systems and CPU architectures; the remaining four files are a macOS compile step
125
+ that could not be run and was not faked.**
126
+
127
+ This is worth stating against the rest of the catalog, because it is the exception. Core AI
128
+ `.aimodel` bundles are **not** byte-reproducible β€” the exporter is not deterministic even
129
+ against itself, with two runs of an identical command nine minutes apart on the same box
130
+ differing by 32 bytes in `main.mlirb`. For those artifacts, integrity has to rest on
131
+ per-file SHA-256 of the exact published bytes. For this one, "verify by rebuilding" is
132
+ genuinely available for the weights.
133
+
134
+ ### The open gate
135
+
136
+ Compile `rebuild-verification/2026-08-17/encoder.mlpackage` with
137
+ `xcrun coremlcompiler compile` on a Mac and diff the resulting `encoder.mlmodelc` against
138
+ the pinned root artifact. That closes the four remaining files and would establish whether
139
+ the *whole* Core ML artifact is reproducible or only its weights. Until then the claim is
140
+ scoped to the weights and the config, as written above.
141
+
142
+ ## License
143
+
144
+ EmbeddingGemma is Gemma-family. These files are a derivative of
145
+ `google/embeddinggemma-300m` and use is subject to the
146
+ [Gemma Terms of Use](https://ai.google.dev/gemma/terms) and the
147
+ [Gemma Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy). Those
148
+ terms travel with the artifact and with any redistribution of it. Visible's contribution is
149
+ the mirror and the rebuild verification, not the weights.