Add Dense Mobile & Edge Suite with ONNX and LiteRT models

README.md

@@ -9,8 +9,12 @@ tags:
 - western-australia
 - dinov3
 - mixture-of-experts
+- litert
+- onnx
 - adaround
 - fp8
+- int8
+- android
 - biodiversity
 - flora
 datasets:
@@ -33,148 +37,61 @@ pipeline_tag: image-classification
 
 ---
 
-## Available Model Weights
+## Model Suite Breakdown
 
-| Model Identifier | Architecture | Precision Format | File Size | Top-1 Accuracy | Top-5 Accuracy | Usage Requirements & Mode |
-| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
-| **`PlantDetect-FP8-AdaRound.safetensors`** | MoE++ (16 FFNs) | **Mixed AdaRound FP8 + BF16** | **`104.3 MB`** | **89.21%** | **96.19%** | **Standard 1-Photo Mode** (Fastest GPU inference, 3.95× smaller). |
-| **`PlantDetect-BF16.safetensors`** | MoE++ (16 FFNs) | **Full `bfloat16`** | **`205.2 MB`** | **89.21%** | **96.19%** | **Standard 1-Photo Mode** (Full precision reference). |
-| **`PlantDetect-4View-FP8-AdaRound.safetensors`** | MoE++ 4-View | **Mixed AdaRound FP8 + BF16** | **`104.3 MB`** | **99.20%** | **99.85%** | ⚠️ **Requires 2–4 complementary photos** of the same plant (Flower + Leaf + Habit + Bark). |
-| **`PlantDetect-4View-BF16.safetensors`** | MoE++ 4-View | **Full `bfloat16`** | **`205.2 MB`** | **99.20%** | **99.85%** | ⚠️ **Requires 2–4 complementary photos** of the same plant (Flower + Leaf + Habit + Bark). |
-| **`PlantDetect-FP32.pt`** | MoE++ (16 FFNs) | **Full `float32`** | `412.4 MB` | 89.21% | 96.19% | PyTorch training checkpoint. |
-
-> [!IMPORTANT]
-> **Single-Photo vs. Multi-Photo (4-View) Note:**
-> * If you only have **1 single photograph**, use **`PlantDetect-FP8-AdaRound.safetensors`** (**89.2% Top-1 / 96.2% Top-5**).
-> * The **`4View` models** are trained specifically for multi-photo sets. Feeding **2 to 4 angles** (e.g. flower close-up + leaf serration + whole shrub) resolves fine-grained sister-species ambiguity and boosts Top-1 accuracy to **99.2%**!
-
----
+### 1. Server / Desktop GPU Models (Mixture-of-Experts)
 
-## Multi-Photo (4-View) Accuracy Scaling
-
-When surveying wild plants in Western Australia, taking multiple complementary photos dramatically increases confidence:
-
-| Number of Photos Provided | Angles Captured | Validation Groups | Top-1 Accuracy | Top-5 Accuracy |
-| :--- | :--- | :--- | :--- | :--- |
-| **1 Photo ($K=1$)** | Single shot (e.g. whole plant) | 27,673 photos | **`89.04%`** | **`96.24%`** |
-| **2 Photos ($K=2$)** | Flower close-up + Leaf margin | 13,593 pairs | **`97.37%`** | **`99.83%`** |
-| **3 Photos ($K=3$)** | Flower + Leaf + Habit / Shrub | 8,903 triplets | **`98.92%`** | **`99.97%`** |
-| **4 Photos ($K=4$)** | Flower + Leaf + Habit + Bark | 6,535 4-sets | **`99.33%`** | **`100.0%`** |
+| Model Identifier | Architecture | Precision Format | File Size | Top-1 Accuracy | Top-5 Accuracy | Recommended Use Case |
+| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
+| **`PlantDetect-FP8-AdaRound.safetensors`** | MoE++ (16 FFNs) | **Mixed AdaRound FP8 + BF16** | **`104.3 MB`** | **89.21%** | **96.19%** | **Fastest, ultra-compact GPU inference (3.95× smaller)** |
+| **`PlantDetect-BF16.safetensors`** | MoE++ (16 FFNs) | **Full `bfloat16`** | **`205.2 MB`** | **89.21%** | **96.19%** | **Lossless baseline reference** |
+| **`PlantDetect-4View-FP8-AdaRound.safetensors`** | MoE++ 4-View | **Mixed AdaRound FP8 + BF16** | **`104.3 MB`** | **99.20%** | **99.85%** | ⚠️ **Requires 2–4 complementary photos** (Flower + Leaf + Habit + Bark) |
+| **`PlantDetect-4View-BF16.safetensors`** | MoE++ 4-View | **Full `bfloat16`** | **`205.2 MB`** | **99.20%** | **99.85%** | ⚠️ **Requires 2–4 complementary photos** (Lossless multi-photo baseline) |
 
 ---
 
-## How AdaRound-FP8 Was Created
-
-Standard Post-Training Quantization (PTQ) uses **Round-to-Nearest (`RTN`)**, which snaps each weight independently to the closest discrete value on the non-uniform FP8 (`E4M3`) grid. On sparse Mixture-of-Experts architectures, naive `RTN` causes quantization noise and routing jitter.
+### 2. Mobile Edge & Embedded Models (Dense ViT-Base)
 
-To eliminate quantization loss while maintaining **zero external dependencies**, **AdaRound-FP8** was developed using a two-stage process:
+Optimized for **mobile NPUs (Samsung S24, Pixel, iOS) and edge devices (Raspberry Pi, Jetson)** using dense matrix layers without routing scatter-gather overhead.
 
-```
-                  ┌─────────────────────────────────────────────────────────┐
-                  │ 1. Layer-by-Layer Sensitivity Analysis                   │
-                  │    - LayerNorms & Routers: 1.71M params (Kept in BF16)  │
-                  │    - Attention & MoE Experts: 100.85M params (FP8 E4M3) │
-                  └────────────────────────────┬────────────────────────────┘
-                                               │
-                                               ▼
-                  ┌─────────────────────────────────────────────────────────┐
-                  │ 2. Discrete Adaptive Rounding (AdaRound)                │
-                  │    - For each weight, bound between [Q_floor, Q_ceil]   │
-                  │    - Optimize continuous mask: min ||Wx - W_fp8 x||²    │
-                  │    - Discretize to pure native torch.float8_e4m3fn      │
-                  └────────────────────────────┬────────────────────────────┘
-                                               │
-                                               ▼
-                  ┌─────────────────────────────────────────────────────────┐
-                  │ 3. Output: 104.3 MB Single Safetensors File             │
-                  │    - 100% Native PyTorch (zero custom C++ wheels needed)│
-                  │    - 97.0% exact FP32 agreement, 0.9998 cosine sim      │
-                  └─────────────────────────────────────────────────────────┘
-```
-
-1. **Layer Sensitivity Analysis:**
-   * Probing demonstrated that 71 sensitive layers (all LayerNorm scale/biases, MoE router gates, and final classification heads) accounted for 90% of naive quantization error.
-   * These 1.71M parameters are preserved in native `torch.bfloat16` (+3.4 MB).
-2. **Adaptive Non-Uniform Discrete Optimization:**
-   * For the remaining 237 heavy Linear, Attention QKV, and SwiGLU expert layers, discrete rounding decisions were optimized via AdamW to minimize the quadratic feature reconstruction loss over field validation samples.
-3. **Results:**
-   * **Exact FP32 Top-1 Match:** Retains identical **90.5%** validation Top-1 accuracy.
-   * **Cosine Similarity:** **`0.999840`** on output logits.
-   * **Size:** Compressed by **3.95×** from 412 MB down to **104.3 MB**.
+| File Path in Repo | Runtime Target | Quantization / Format | Size | Top-1 Accuracy | Top-5 Accuracy | Target Hardware |
+| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
+| **`onnx/PlantDetect-Dense-INT8-AdaRound.onnx`** | ONNX Runtime Mobile | **Mixed-Precision INT8** | **`106.3 MB`** | **83.08%** | **94.93%** | Android / iOS / Windows DirectML / CoreML |
+| **`onnx/PlantDetect-Dense-FP32.onnx`** | ONNX Runtime Standard | **Full FP32** | `363.3 MB` | 83.08% | 94.93% | Reference ONNX graph |
+| **`litert/PlantDetect-Dense-INT8.tflite`** | Google LiteRT (TFLite) | **Full Integer INT8** | **`387.0 MB`** | **83.08%** | **94.93%** | Samsung Galaxy S24 NPU (Exynos / Snapdragon) |
+| **`litert/PlantDetect-Dense-FP16.tflite`** | Google LiteRT (TFLite) | **Float16** | **`196.2 MB`** | **83.08%** | **94.93%** | Mobile GPU / XNNPACK CPU acceleration |
+| **`PlantDetect-Dense-FP8-AdaRound.safetensors`** | PyTorch / Python | **Mixed AdaRound FP8** | **`92.7 MB`** | **83.08%** | **94.93%** | Lightweight Python deployment |
+| **`PlantDetect-Dense-BF16.safetensors`** | PyTorch / Python | **Full BF16** | **`181.5 MB`** | **83.08%** | **94.93%** | Dense PyTorch reference |
 
 ---
 
-## Quickstart / Inference Examples
+## Android & ONNX Runtime Quickstart
 
-### 1. Single-Photo Inference
+### ONNX Runtime Inference in Python / C++ / C#
 ```python
-import torch
-import torchvision.transforms as T
+import onnxruntime as ort
+import numpy as np
 from PIL import Image
-from huggingface_hub import hf_hub_download
-from safetensors.torch import load_file
-import json
-
-# Download PlantDetect model and species index
-repo_id = "thenukegun10x/wa-plant-identifier"
-weights_file = hf_hub_download(repo_id=repo_id, filename="PlantDetect-FP8-AdaRound.safetensors")
-species_file = hf_hub_download(repo_id=repo_id, filename="species_index.json")
-
-with open(species_file, "r", encoding="utf-8") as f:
-    species_index = json.load(f)
-
-transform = T.Compose([
-    T.Resize(int(336 * 1.14)),
-    T.CenterCrop(336),
-    T.ToTensor(),
-    T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
-])
-
-img = Image.open("sample_plant.jpg").convert("RGB")
-x = transform(img).unsqueeze(0)
-
-from src.models.plant_vit import PlantViT
-model = PlantViT(stem_name="vit_base_patch16_dinov3", n_classes=999, use_moe=True, num_ffn=16)
-
-raw_weights = load_file(weights_file)
-sd = {k: v.float() if v.is_floating_point() else v for k, v in raw_weights.items()}
-model.load_state_dict(sd, strict=False)
-model.eval()
-
-with torch.no_grad():
-    logits, embedding, aux = model(x)
-    probs = torch.softmax(logits, dim=-1)[0]
-    top5_probs, top5_indices = torch.topk(probs, 5)
-
-for rank, (prob, idx) in enumerate(zip(top5_probs, top5_indices), 1):
-    spp_info = species_index[idx.item()]
-    print(f"#{rank}: {spp_info['scientific_name']} ({spp_info['genus']}) - {prob*100:.2f}%")
-```
 
-### 2. Multi-Photo (4-View) Inference (99.2% Accuracy)
-```python
-# Pass 4 complementary photos (e.g. flower, leaf, habit, bark)
-image_paths = ["flower.jpg", "leaf.jpg", "habit.jpg", "bark.jpg"]
-batch_tensors = torch.cat([transform(Image.open(p).convert("RGB")).unsqueeze(0) for p in image_paths], dim=0)
-
-weights_4view = hf_hub_download(repo_id=repo_id, filename="PlantDetect-4View-FP8-AdaRound.safetensors")
-raw_4view = load_file(weights_4view)
-model.load_state_dict({k: v.float() if v.is_floating_point() else v for k, v in raw_4view.items()}, strict=False)
-model.eval()
-
-with torch.no_grad():
-    logits_4view, _, _ = model(batch_tensors)
-    # Average logits across all 4 angles
-    mean_logits = logits_4view.mean(dim=0, keepdim=True)
-    probs = torch.softmax(mean_logits, dim=-1)[0]
-    top_pred = species_index[probs.argmax().item()]
-    print(f"4-View Identified Species: {top_pred['scientific_name']} ({probs.max()*100:.2f}% confidence)")
+# Initialize ONNX Runtime session
+session = ort.InferenceSession("onnx/PlantDetect-Dense-INT8-AdaRound.onnx", providers=["CPUExecutionProvider"])
+
+# Preprocess image to [1, 3, 336, 336]
+img = Image.open("wild_flower.jpg").convert("RGB").resize((383, 383))
+crop = img.crop((23, 23, 359, 359))
+arr = (np.array(crop, dtype=np.float32) / 255.0 - [0.485, 0.456, 0.406]) / [0.229, 0.224, 0.225]
+tensor = np.expand_dims(arr.transpose(2, 0, 1), axis=0)
+
+# Run inference
+logits = session.run(None, {"input": tensor})[0]
+predicted_class = np.argmax(logits, axis=-1)[0]
+print(f"Predicted Species Index: {predicted_class}")
 ```
 
 ---
 
 ## Attribution & Licensing
 
-* **Built with DINOv3:** This project uses the DINOv3 vision backbone (`vit_base_patch16_dinov3.lvd1689m`) developed by Meta AI.
-* **Dataset Attribution:** Observation data sourced under open licenses (CC BY, CC BY-NC, CC0) from the **Global Biodiversity Information Facility (GBIF)** and **Western Australian Herbarium (FloraBase / DBCA)**.
+* **Built with DINOv3:** Uses the DINOv3 vision backbone (`vit_base_patch16_dinov3.lvd1689m`) developed by Meta AI.
+* **Dataset Attribution:** Observation data sourced from the **Global Biodiversity Information Facility (GBIF)** and **Western Australian Herbarium (FloraBase / DBCA)**.
 * **License:** Released under **Creative Commons Non-Commercial (CC BY-NC 4.0)** for scientific and non-commercial research.