Upload README.md with huggingface_hub

README.md

@@ -12,59 +12,85 @@ library_name: llama.cpp
 
 # Gemma LITE (E2B) — a smaller dictionary, the same mind
 
-A **Model Derivative** of Google's Gemma 3n E2B, made for phones that cannot
-hold the original in memory.
+A **Model Derivative** of Google's Gemma 3n E2B, made for phones that cannot hold
+the original in memory.
 
-**Nothing about the model's reasoning was changed.** All 35 transformer blocks
-are byte-for-byte the quantisation they started as. The only modification is to
-one tensor.
+**Nothing about the model's reasoning was changed.** All 35 transformer blocks are
+byte-for-byte the quantisation they started as. The only modification is to one
+tensor.
 
 ## What was modified
 
-`per_layer_token_embd.weight` — requantised from **q6_K to q2_K**.
+`per_layer_token_embd.weight` — requantised from **q6_K to q4_0**, matching the
+precision of every layer that does the actual reasoning.
 
 That single tensor is **57% of the original file**: 1,837 MiB of a 3,179 MiB
-model, shaped `[8960, 262144]`. It is a *lookup table* — one row of about 7 KB
-is read per token — and it was being stored at **higher precision than the
-layers that do the reasoning**, every one of which is q4_0.
+model, shaped `[8960, 262144]`. It is a *lookup table* — one row of about 7 KB is
+read per token — and it was stored at **higher precision than the layers doing the
+thinking**, every one of which is q4_0.
 
 ```
 llama-quantize --allow-requantize \
-  --tensor-type per_layer_token_embd=q2_K \
-  gemma-3n-E2B-it-q4_0.gguf gemma-lite-e2b-q2.gguf Q4_0
+  --tensor-type per_layer_token_embd=q4_0 \
+  gemma-3n-E2B-it-q4_0.gguf gemma-lite-e2b-q4.gguf Q4_0
 ```
 
 | | file | `per_layer_token_embd` |
 |---|---|---|
 | source (E2B q4_0) | 3.12 GiB | 1,837 MiB (q6_K) |
-| **LITE** | **2.04 GiB** | **735 MiB (q2_K)** |
+| **LITE** | **2.56 GiB** | 1,270 MiB (q4_0) |
 
-## Why it exists
+## ⚠️ A q2_K version of this model was published here and has been withdrawn
 
-Measured on a Galaxy S22 Ultra (8 GB): the unmodified model was killed by
-Android's low-memory killer **eleven times**, never finishing load. It needs
-roughly every spare byte the phone has.
+An earlier build took the same tensor down to **q2_K**, reaching 2.04 GiB. It
+passed a single-turn audition and **failed badly over a real conversation.** It
+was replaced on 2026-08-12 and should not be used.
 
-Measured on a ROG Phone 8 Pro with llama.cpp and OpenCL, 8192 context:
+Over 30 turns, q2_K against this q4_0 build:
+
+| | q2_K (withdrawn) | **q4_0 (this file)** |
+|---|---|---|
+| repetition, second half (4-gram overlap) | 0.175, peaking **0.43** | **0.000** |
+| replies containing malformed control tokens | **17 of 30** | 3 of 30 |
+| corrupted words (`It't what it is`) | throughout | none |
+| latency, second half | 5,600 ms | 3,963 ms |
+
+q2_K collapsed into a verbal tic — *"That's what it is"* in nearly every reply
+after turn 12 — emitted malformed control tokens such as `<end_of_of_turn>` into
+visible output, and confidently misremembered the conversation it was in.
+
+☠️ **The lesson, if you are quantising embeddings yourself: a single-turn test
+cannot see this.** The damage only appears once enough context has to be held at
+once. Test over a long conversation, not one reply.
+
+## Measured
+
+Galaxy S22 Ultra (8 GB): the **unmodified** model was killed by Android's
+low-memory killer eleven times and never finished loading.
+
+ROG Phone 8 Pro, llama.cpp with OpenCL, 8192 context:
 
 | | |
 |---|---|
-| system memory held | **~510 MiB** |
-| unevictable (anonymous) | **0.17 GiB** |
+| unevictable (anonymous) memory | **~178 MiB** |
+| GPU / shmem | ~80 MiB |
 | prompt eval | **211 tok/s** |
 | generation | **10.7 tok/s** |
 
 A ~3,800-token system prompt prefills in about **18 seconds**.
 
+☠️ The GPU path is not an optimisation — it is why this fits. On CPU the same
+model holds **1,642 MiB** of unevictable memory, roughly 9× more, which is the
+shape of failure that killed the unmodified model on 8 GB.
+
 ## Quality
 
-Auditioned against the unmodified model — same seed, same temperature, same
-2,400-token character prompt. Both replies were in character and of comparable
-quality; the LITE build was, if anything, more faithful to the prompt's register.
+☠️ **No perplexity or standardised benchmark has been run.** What exists is a
+30-turn conversational soak and a 17-prompt content probe, both single-sample. If
+you need measured quality, measure it.
 
-☠️ **This is one sample, not a benchmark.** No perplexity or standardised
-evaluation has been run. q2_K is applied to token embeddings — how the model
-represents word *meaning* — so if you need measured quality, measure it.
+Note the numbers above are for this file at 4,300 tokens of context; behaviour at
+the full 8,192 has not been characterised.
 
 ## What it cannot do
 
@@ -76,8 +102,8 @@ The source GGUF carries **no vision or audio encoder tensors**, so this is a
 Gemma 3n is provided under and subject to the **Gemma Terms of Use**:
 https://ai.google.dev/gemma/terms
 
-This is a **modified** version. The modification is described above and is
-limited to the requantisation of a single tensor. The Gemma Terms, including the
+This is a **modified** version. The modification is described above and is limited
+to the requantisation of a single tensor. The Gemma Terms, including the
 Prohibited Use Policy, apply to this derivative and to anything derived from it.
 
 Base model: `google/gemma-3n-E2B-it`.