ReLU-CLIP

Frozen CLIP image encoders distilled into small ReLU/ReLU6 + BatchNorm CNNs that quantise to int8 and run as a single graph on low-power edge accelerators, whose operator coverage is strongest for convolutional networks.

Code, figures and the full result set: https://github.com/jiaheguo521/relu-clip

This repository holds the weights: the deployable model, and the int8 graphs for all 18 runs of the accompanying study so its central claim can be checked independently.

Headline model — efflite4 distilled from CLIP ViT-L/14

• 68.25% ImageNet-1k zero-shot top-1 in int8 — 68.35% in fp32, so quantisation costs 0.10 pp
• 12.71M parameters, 13.6 MiB as an int8 graph
• 26.48 ms/frame (37.8 FPS) measured, with 7.5 MiB of weights streamed off-chip
• 116 of 116 operators on the accelerator, 0 on CPU — the whole image tower runs as one graph

Evaluated on the complete 50,000-image ImageNet-1k validation set. Latency measured on a Coral USB Edge TPU over a 300-frame pack, first frame dropped.

Files

efflite4-vitl14/
  student.safetensors           fp32 weights, 12.71M parameters -- prefer this
  student.pt                    the same model as a pickled nn.Module (see note below)
  student_int8.tflite           per-channel int8, runs on any TFLite CPU runtime
  student_int8_edgetpu.tflite   compiled with edgetpu_compiler 16.0
text/
  imagenet1k_text_emb_vitl14.npz  1000 x 768 class embeddings (keys: embs, labels)
  demo_prompt_emb_vitl14.npz      open-vocabulary demo prompts, encoded verbatim (keys: text_emb, labels)
sweep-int8/
  <teacher>__<student>_int8.tflite   all 18 runs (2 teachers x 9 students)
example_infer.py                runnable example -- numpy + Pillow + a TFLite runtime, nothing else
preprocessing.json              machine-readable input/output spec
matrix.csv, results_full.json, activation_ranges.csv, derisk_table.csv

Only want the deployable model? sweep-int8/ is 384 of the 500 MB:

from huggingface_hub import snapshot_download
snapshot_download("jiaheguo521/relu-clip", allow_patterns=["efflite4-vitl14/*", "text/*", "*.json", "*.py"])

Two things that will silently give wrong results if missed:

• imagenet1k_text_emb_vitl14.npz is not L2-normalised. Each row is an average over the 80 OpenAI ImageNet prompt templates, and averaging unit vectors gives a norm below 1 (here 0.81-0.93). Normalise it yourself before taking cosines. demo_prompt_emb_vitl14.npz is normalised — its prompts are encoded verbatim, without templating.
• The image graph emits an unnormalised embedding. L2 normalisation is deliberately left out of the int8 graph, so the accelerator subgraph stays free of reductions. Do it on the host after dequantising.

student.pt is a pickled nn.Module, so loading it runs torch.load(..., weights_only=False) and needs this project's sources/ importable. It is included because the repository's own conversion and evaluation scripts consume that format. For anything else, use student.safetensors, which carries the identical weights (verified bit-exact) and loads without executing pickled code:

from safetensors.torch import load_file
from students import build_student          # from the GitHub repo
model = build_student("efflite4", embed_dim=768, pretrained=False)
model.load_state_dict(load_file("efflite4-vitl14/student.safetensors"))
model.eval()

Usage

pip install numpy pillow tflite-runtime      # no torch, no CLIP
python example_infer.py your_photo.jpg

example_infer.py is the whole thing end to end, and its preprocessing is verified bit-equivalent to the transform the student was trained with across 40 aspect ratios. Two details in it are easy to get wrong and cost accuracy silently, so preprocessing.json states them machine-readably: Resize(224) truncates the long side, and CenterCrop rounds the crop offset. Off by one pixel in either and the crop shifts.

The image tower is a plain int8 TFLite graph; zero-shot classification is a cosine between its L2-normalised output and a stored text-embedding matrix. No CLIP text encoder is needed at inference.

from tflite_runtime.interpreter import Interpreter   # example_infer.py also falls back to
                                                     # ai_edge_litert or full tensorflow
import numpy as np

itp = Interpreter("efflite4-vitl14/student_int8.tflite")
itp.allocate_tensors()
inp, out = itp.get_input_details()[0], itp.get_output_details()[0]

text = np.load("text/imagenet1k_text_emb_vitl14.npz")
T, labels = text["embs"], text["labels"]                   # [1000, 768]
T = T / np.linalg.norm(T, axis=1, keepdims=True)           # NOT normalised on disk -- see above

# img: [1,224,224,3], resized to 224 bicubic + CLIP-normalised, then quantised with inp["quantization"]
s_in, z_in = inp["quantization"]
itp.set_tensor(inp["index"], np.clip(np.round(img / s_in + z_in), -128, 127).astype(inp["dtype"]))
itp.invoke()
emb = itp.get_tensor(out["index"]).astype(np.float32)
s, z = out["quantization"]
emb = (emb - z) * s                                        # dequantise
emb /= np.linalg.norm(emb, axis=-1, keepdims=True)         # the graph emits UNNORMALISED embeddings
print(labels[(emb @ T.T).argmax(-1)])

Your own classes

Changing the sentences changes the classifier; nothing is retrained. Write the phrases one per line and encode them once:

pip install torch --index-url https://download.pytorch.org/whl/cpu   # CPU wheel: skips ~2.7 GB of CUDA libraries
pip install open_clip_torch

python sources/make_prompt_emb.py --teacher vitl14 \
    --labels-file my_labels.txt --out my_labels.npz               # downloads ViT-L/14 once, 1.6 GB

python example_infer.py your_photo.jpg --text my_labels.npz       # back to numpy + tflite only

This is a build step, not a runtime dependency: run it once on any laptop and copy the resulting few-KB .npz to the device. Text encoding never happens on the accelerator — that is what makes the deployed graph a single int8 CNN.

text/demo_prompt_emb_vitl14.npz is a worked example of the output.

For the Edge TPU binary, load student_int8_edgetpu.tflite with the libedgetpu.so.1 delegate. It was produced by edgetpu_compiler 16.0 and needs a matched-generation runtime; the GitHub README documents which builds work.

The sweep, and why the int8 graphs are published

Across 2 teachers x 9 students, int8 does not preserve the fp32 ranking:

teacher / student	activation	fp32	int8	Δ
vitl14 / efflite4	ReLU6	68.35	68.25	-0.10
vitl14 / r50	ReLU	67.94	60.03	-7.90
vitl14 / r50_relu6	ReLU6	68.13	67.62	-0.50
vitl14 / r101	ReLU	70.70	12.09	-58.62
vitl14 / r101_relu6	ReLU6	69.09	67.27	-1.82

r50_relu6 / r101_relu6 are the same ResNets with every nn.ReLU replaced by nn.ReLU6 — same depth, same parameter count, same pretrained weights, same recipe. Bounding the activation is what recovers the accuracy.

The mechanism is measurable directly from these files. int8 spans a tensor's range in 255 steps, so the step is range / 255; every ReLU6 run has a median activation range of exactly 6.00 (step 0.0235), while the plain-ReLU ResNets sit at 59.2–209.8 with worst tensors reaching 4547.1 (step 17.83). sweep-int8/ is published so that anyone can read those quantisation scales out of the same graphs that produced the accuracy numbers, rather than taking a CSV on trust:

python sources/activation_ranges.py      # from the GitHub repo

All nine students compile with 0 CPU operators (derisk_table.csv), so none of this is an operator-coverage artifact.

Limitations

• Single seed, no variance estimate. The bounded-vs-unbounded gap is far larger than run-to-run noise, but rankings within the frontier should not be over-interpreted.
• Latency is measured per architecture, not per run: the 768-d models were timed and the numbers reused for their 1024-d counterparts, a sub-1% approximation.
• ReLU6 is not demonstrably fp32-free — r50_relu6 (68.13) is close to r50 (67.94), but the two trained for different epoch counts under the convergence schedule. The int8 conclusions are unaffected, since each drop is relative to that model's own fp32.
• r34 and r50 may be undertrained, as early stopping was driven by a 5k validation subset.
• Bounded activations quantising better than unbounded ones is a known result in the quantisation literature. What is measured here is its magnitude and consequences under a specific deployment constraint, with same-parameter controls; no novelty is claimed for the mechanism.
• Operator coverage and the compiler behaviour described on GitHub are specific to one toolchain generation. The activation-range result is toolchain-independent; the compile details are not.
• The unbounded-ReLU graphs in sweep-int8/ are published as evidence, not as usable models — rn50__r101_int8.tflite scores 0.72%.

License

MIT.