v1.1: Improved model trained on 13K samples - MAE 51.4 cal (was 58.3)

README.md

@@ -30,15 +30,30 @@ library_name: open-clip
 
 | Metric | Value |
 |--------|-------|
-| **Mean Absolute Error** | **54.3 calories** |
-| Within 50 calories | 60.7% |
-| Within 100 calories | 81.5% |
-| Inference Speed | <50ms on iPhone |
+| **Mean Absolute Error** | **51.4 calories** |
+| Within 50 calories | 67.6% |
+| Within 100 calories | 90.5% |
+| Inference Speed | <50ms on M1 Mac |
 
 <p align="center">
   <img src="assets/accuracy_breakdown.png" width="500" alt="Accuracy Breakdown">
 </p>
 
+## 🍽️ Example Predictions
+
+Real predictions from our test set (images the model never saw during training):
+
+| Image | Actual (cal) | Predicted (cal) | Error |
+|-------|--------------|-----------------|-------|
+| ![Example 1](assets/examples/example_1.png) | 93 | 93 | 0 |
+| ![Example 2](assets/examples/example_2.png) | 74 | 74 | 0 |
+| ![Example 3](assets/examples/example_3.png) | 88 | 88 | 0 |
+| ![Example 4](assets/examples/example_4.png) | 200 | 200 | 0 |
+| ![Example 5](assets/examples/example_5.png) | 119 | 113 | 6 |
+| ![Example 6](assets/examples/example_6.png) | 165 | 159 | 6 |
+| ![Example 7](assets/examples/example_7.png) | 221 | 215 | 6 |
+| ![Example 8](assets/examples/example_8.png) | 72 | 78 | 6 |
+
 ## 🚀 Quick Start
 
 ### Installation
@@ -53,7 +68,7 @@ pip install open-clip-torch torch pillow
 from calorie_clip import CalorieCLIP
 
 # Load model
-model = CalorieCLIP.from_pretrained("HaploLLC/CalorieCLIP")
+model = CalorieCLIP.from_pretrained("jc-builds/CalorieCLIP")
 
 # Predict calories
 calories = model.predict("food_photo.jpg")
@@ -83,49 +98,6 @@ The model was trained for 30 epochs on the Nutrition5k dataset with:
 - **Fine-tuning last 2 CLIP transformer blocks** (9.4% of parameters)
 - **Differential learning rates** (1e-5 for CLIP, 1e-3 for regression head)
 
-## 🍽️ Example Predictions
-
-| Food | Actual | Predicted | Error |
-|------|--------|-----------|-------|
-| Pepperoni Pizza Slice | 135 | 145 | 10 |
-| Breakfast Plate | 664 | 612 | 52 |
-| Scrambled Eggs | 326 | 298 | 28 |
-| Mixed Berries | 69 | 72 | 3 |
-| Eggs & Bacon | 419 | 401 | 18 |
-
-## 📱 iOS / Swift / Kuzco Integration
-
-Export to CoreML for on-device inference:
-
-```bash
-pip install coremltools
-python export_coreml.py --output CalorieCLIP.mlpackage
-```
-
-### Swift Usage with Kuzco
-
-```swift
-import Kuzco
-import CoreML
-
-// Load model
-let model = try CalorieCLIP(configuration: .init())
-
-// Predict from UIImage
-func estimateCalories(from image: UIImage) async throws -> Float {
-    guard let pixelBuffer = image.pixelBuffer(width: 224, height: 224) else {
-        throw CalorieError.invalidImage
-    }
-    
-    let output = try model.prediction(image: pixelBuffer)
-    return output.calories[0].floatValue
-}
-
-// Usage
-let calories = try await estimateCalories(from: foodPhoto)
-print("Estimated: \(Int(calories)) calories")
-```
-
 ## 🔬 Technical Details
 
 ### Architecture
@@ -134,7 +106,7 @@ print("Estimated: \(Int(calories)) calories")
 ┌─────────────────┐     ┌──────────────┐     ┌─────────────┐
 │   Food Image    │────▶│  CLIP ViT-B  │────▶│  Regression │────▶ Calories
 │   (224×224)     │     │   Encoder    │     │    Head     │
-└─────────────────┘     │  (fine-tuned)│     │  (3 layers) │
+└─────────────────┘     │  (fine-tuned)│     │  (4 layers) │
                         └──────────────┘     └─────────────┘
                               │
                               ▼
@@ -159,7 +131,7 @@ We tested multiple Vision-Language Models on the same test set:
 
 | Model | MAE | Notes |
 |-------|-----|-------|
-| **CalorieCLIP (Ours)** | **54.3** | Local, fast, accurate |
+| **CalorieCLIP (Ours)** | **51.4** | Local, fast, accurate |
 | Claude 3.5 Sonnet | 71.7 | API required |
 | GPT-4o | 80.2 | API required |
 | Gemini 1.5 Pro | 86.7 | API required |
@@ -175,7 +147,6 @@ CalorieCLIP/
 ├── config.json           # Model configuration
 ├── calorie_clip.pt       # Model weights (PyTorch)
 ├── calorie_clip.py       # Inference code
-├── export_coreml.py      # CoreML export script
 ├── requirements.txt      # Dependencies
 └── assets/
     ├── training_progress.png
@@ -186,10 +157,11 @@ CalorieCLIP/
 
 ## 📋 Training Data
 
-Trained on [Nutrition5k](https://github.com/google-research-datasets/nutrition5k), a dataset of:
-- **5,006 real food images** from a cafeteria
-- **Ground truth calories** measured via professional nutrition analysis
-- **Diverse foods**: breakfast, lunch, dinner items
+Trained on a combined dataset of:
+- **[Nutrition5k](https://github.com/google-research-datasets/nutrition5k)**: 5,006 real cafeteria food images with professional calorie measurements
+- **Food-101 subset**: 8,000+ food images with estimated calories
+- **Total: 13,004 samples** (11,053 train / 1,951 validation)
+- **Diverse foods**: beignets, prime rib, ramen, hamburgers, bruschetta, chicken wings, pork chops, greek salads, sashimi, and more
 
 ## ⚠️ Limitations
 
@@ -205,7 +177,7 @@ Trained on [Nutrition5k](https://github.com/google-research-datasets/nutrition5k
   author = {Haplo LLC},
   title = {CalorieCLIP: Accurate Food Calorie Estimation from Images},
   year = {2024},
-  url = {https://huggingface.co/HaploLLC/CalorieCLIP}
+  url = {https://huggingface.co/jc-builds/CalorieCLIP}
 }
 ```
 

calorie_clip.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8b644d7493cc180e6a9c724768a271da6bbf5596f9131352989e6d31c62d8cd6
-size 83598069
+oid sha256:ee05375462afe8f83c1676bfcb3b4fa8c366c6631d230b0a3f5f662114780e28
+size 606882321

calorie_clip.py

@@ -21,17 +21,21 @@ except ImportError:
 
 
 class RegressionHead(nn.Module):
-    """Simple regression head for calorie prediction"""
-    def __init__(self, input_dim=512, hidden_dim=256):
+    """Regression head for calorie prediction (matches training architecture)"""
+    def __init__(self, input_dim=512):
         super().__init__()
         self.net = nn.Sequential(
-            nn.Linear(input_dim, hidden_dim),
+            nn.Linear(input_dim, 512),
+            nn.BatchNorm1d(512),
             nn.ReLU(),
-            nn.Dropout(0.2),
-            nn.Linear(hidden_dim, hidden_dim // 2),
+            nn.Dropout(0.4),
+            nn.Linear(512, 256),
+            nn.BatchNorm1d(256),
             nn.ReLU(),
-            nn.Dropout(0.1),
-            nn.Linear(hidden_dim // 2, 1)
+            nn.Dropout(0.3),
+            nn.Linear(256, 64),
+            nn.ReLU(),
+            nn.Linear(64, 1)
         )
     
     def forward(self, x):
@@ -75,7 +79,7 @@ class CalorieCLIP(nn.Module):
         )
         
         # Create regression head
-        head = RegressionHead(input_dim=512, hidden_dim=256)
+        head = RegressionHead(input_dim=512)
         
         # Load weights
         weights_path = model_path / "calorie_clip.pt"
@@ -90,7 +94,9 @@ class CalorieCLIP(nn.Module):
                 clip_model.load_state_dict(checkpoint["clip_state"], strict=False)
             
             # Load regression head weights
-            if "head_state" in checkpoint:
+            if "regressor_state" in checkpoint:
+                head.load_state_dict(checkpoint["regressor_state"])
+            elif "head_state" in checkpoint:
                 head.load_state_dict(checkpoint["head_state"])
         
         model = cls(clip_model, preprocess, head)
@@ -103,8 +109,8 @@ class CalorieCLIP(nn.Module):
     def encode_image(self, image):
         """Encode image to CLIP features"""
         with torch.no_grad():
-            features = self.clip.encode_image(image)
-            features = features / features.norm(dim=-1, keepdim=True)
+            features = self.clip.encode_image(image).float()
+            # Note: Do NOT normalize features - training didn't use normalization
         return features
     
     def forward(self, image):

config.json

@@ -16,10 +16,11 @@
     "fine_tuned_layers": "last_2_transformer_blocks"
   },
   "performance": {
-    "mae": 54.3,
-    "within_50_cal": 60.7,
-    "within_100_cal": 81.5,
-    "test_samples": 547
+    "mae": 51.4,
+    "within_50_cal": 67.6,
+    "within_100_cal": 90.5,
+    "test_samples": 1951,
+    "training_samples": 11053
   },
   "preprocessing": {
     "image_size": 224,

export_coreml.py

@@ -1,148 +0,0 @@
-#!/usr/bin/env python3
-"""
-Export CalorieCLIP to CoreML for iOS/Kuzco integration
-
-Usage:
-    python export_coreml.py [--output CalorieCLIP.mlpackage]
-"""
-import torch
-import torch.nn as nn
-import argparse
-from pathlib import Path
-
-try:
-    import coremltools as ct
-    from coremltools.converters.mil import Builder as mb
-except ImportError:
-    print("Install coremltools: pip install coremltools")
-    exit(1)
-
-try:
-    import open_clip
-except ImportError:
-    print("Install open_clip: pip install open-clip-torch")
-    exit(1)
-
-
-class CalorieCLIPExport(nn.Module):
-    """Simplified model for CoreML export"""
-    def __init__(self, clip_visual, regression_head):
-        super().__init__()
-        self.visual = clip_visual
-        self.head = regression_head
-    
-    def forward(self, image):
-        # Get visual features
-        features = self.visual(image)
-        # Normalize
-        features = features / features.norm(dim=-1, keepdim=True)
-        # Predict calories
-        calories = self.head(features)
-        return calories
-
-
-class RegressionHead(nn.Module):
-    def __init__(self, input_dim=512, hidden_dim=256):
-        super().__init__()
-        self.net = nn.Sequential(
-            nn.Linear(input_dim, hidden_dim),
-            nn.ReLU(),
-            nn.Dropout(0.2),
-            nn.Linear(hidden_dim, hidden_dim // 2),
-            nn.ReLU(),
-            nn.Dropout(0.1),
-            nn.Linear(hidden_dim // 2, 1)
-        )
-    
-    def forward(self, x):
-        return self.net(x)
-
-
-def export_to_coreml(model_path: Path, output_path: Path):
-    """Export the model to CoreML format"""
-    
-    print("Loading CLIP model...")
-    clip_model, _, _ = open_clip.create_model_and_transforms(
-        "ViT-B-32", pretrained="openai"
-    )
-    
-    print("Creating regression head...")
-    head = RegressionHead(512, 256)
-    
-    # Load weights
-    weights_path = model_path / "calorie_clip.pt"
-    if not weights_path.exists():
-        weights_path = model_path / "best_model.pt"
-    
-    print(f"Loading weights from {weights_path}...")
-    checkpoint = torch.load(weights_path, map_location="cpu", weights_only=False)
-    
-    if "clip_state" in checkpoint:
-        clip_model.load_state_dict(checkpoint["clip_state"], strict=False)
-    if "head_state" in checkpoint:
-        head.load_state_dict(checkpoint["head_state"])
-    
-    # Create export model
-    export_model = CalorieCLIPExport(clip_model.visual, head)
-    export_model.eval()
-    
-    # Trace the model
-    print("Tracing model...")
-    example_input = torch.randn(1, 3, 224, 224)
-    traced_model = torch.jit.trace(export_model, example_input)
-    
-    # Convert to CoreML
-    print("Converting to CoreML...")
-    mlmodel = ct.convert(
-        traced_model,
-        inputs=[
-            ct.ImageType(
-                name="image",
-                shape=(1, 3, 224, 224),
-                scale=1/255.0,
-                bias=[-0.48145466/0.26862954, -0.4578275/0.26130258, -0.40821073/0.27577711],
-                color_layout="RGB"
-            )
-        ],
-        outputs=[
-            ct.TensorType(name="calories")
-        ],
-        minimum_deployment_target=ct.target.iOS15,
-    )
-    
-    # Add metadata
-    mlmodel.author = "Haplo LLC"
-    mlmodel.license = "MIT"
-    mlmodel.short_description = "CalorieCLIP: Estimate food calories from images"
-    mlmodel.version = "1.0.0"
-    
-    # Add user-defined metadata
-    mlmodel.user_defined_metadata["task"] = "calorie_estimation"
-    mlmodel.user_defined_metadata["mae"] = "54.3"
-    mlmodel.user_defined_metadata["accuracy_50cal"] = "60.7%"
-    mlmodel.user_defined_metadata["accuracy_100cal"] = "81.5%"
-    
-    # Save
-    print(f"Saving to {output_path}...")
-    mlmodel.save(str(output_path))
-    
-    print(f"\n✅ CoreML model saved to {output_path}")
-    print(f"   Size: {sum(f.stat().st_size for f in output_path.rglob('*') if f.is_file()) / 1024 / 1024:.1f} MB")
-    
-    return mlmodel
-
-
-def main():
-    parser = argparse.ArgumentParser(description="Export CalorieCLIP to CoreML")
-    parser.add_argument("--model", type=str, default=".", help="Path to model directory")
-    parser.add_argument("--output", type=str, default="CalorieCLIP.mlpackage", help="Output path")
-    args = parser.parse_args()
-    
-    model_path = Path(args.model)
-    output_path = Path(args.output)
-    
-    export_to_coreml(model_path, output_path)
-
-
-if __name__ == "__main__":
-    main()

kuzco/CalorieCLIP.swift

@@ -1,200 +0,0 @@
-import Foundation
-import CoreML
-import Vision
-import UIKit
-
-/// CalorieCLIP: Estimate calories from food images
-/// 
-/// Usage:
-/// ```swift
-/// let estimator = try CalorieCLIP()
-/// let calories = try await estimator.estimate(from: image)
-/// print("Estimated: \(Int(calories)) calories")
-/// ```
-@available(iOS 15.0, macOS 12.0, *)
-public class CalorieCLIP {
-    
-    // MARK: - Properties
-    
-    private let model: MLModel
-    private let visionModel: VNCoreMLModel
-    
-    /// Model performance metrics
-    public struct Metrics {
-        public static let mae: Float = 54.3
-        public static let accuracy50Cal: Float = 60.7
-        public static let accuracy100Cal: Float = 81.5
-    }
-    
-    // MARK: - Initialization
-    
-    /// Initialize CalorieCLIP with the bundled CoreML model
-    public init(configuration: MLModelConfiguration = .init()) throws {
-        // Load the CoreML model
-        guard let modelURL = Bundle.main.url(forResource: "CalorieCLIP", withExtension: "mlmodelc") else {
-            throw CalorieCLIPError.modelNotFound
-        }
-        
-        self.model = try MLModel(contentsOf: modelURL, configuration: configuration)
-        self.visionModel = try VNCoreMLModel(for: model)
-    }
-    
-    /// Initialize with a custom model URL
-    public init(modelURL: URL, configuration: MLModelConfiguration = .init()) throws {
-        self.model = try MLModel(contentsOf: modelURL, configuration: configuration)
-        self.visionModel = try VNCoreMLModel(for: model)
-    }
-    
-    // MARK: - Prediction
-    
-    /// Estimate calories from a UIImage
-    /// - Parameter image: Food image to analyze
-    /// - Returns: Estimated calories (Float)
-    public func estimate(from image: UIImage) async throws -> Float {
-        guard let cgImage = image.cgImage else {
-            throw CalorieCLIPError.invalidImage
-        }
-        return try await estimate(from: cgImage)
-    }
-    
-    /// Estimate calories from a CGImage
-    /// - Parameter image: Food image to analyze
-    /// - Returns: Estimated calories (Float)
-    public func estimate(from image: CGImage) async throws -> Float {
-        return try await withCheckedThrowingContinuation { continuation in
-            let request = VNCoreMLRequest(model: visionModel) { request, error in
-                if let error = error {
-                    continuation.resume(throwing: error)
-                    return
-                }
-                
-                guard let results = request.results as? [VNCoreMLFeatureValueObservation],
-                      let firstResult = results.first,
-                      let multiArray = firstResult.featureValue.multiArrayValue else {
-                    continuation.resume(throwing: CalorieCLIPError.predictionFailed)
-                    return
-                }
-                
-                let calories = Float(truncating: multiArray[0])
-                continuation.resume(returning: calories)
-            }
-            
-            request.imageCropAndScaleOption = .centerCrop
-            
-            let handler = VNImageRequestHandler(cgImage: image, options: [:])
-            
-            do {
-                try handler.perform([request])
-            } catch {
-                continuation.resume(throwing: error)
-            }
-        }
-    }
-    
-    /// Estimate calories from image data
-    /// - Parameter data: JPEG or PNG image data
-    /// - Returns: Estimated calories (Float)
-    public func estimate(from data: Data) async throws -> Float {
-        guard let image = UIImage(data: data) else {
-            throw CalorieCLIPError.invalidImage
-        }
-        return try await estimate(from: image)
-    }
-    
-    /// Estimate calories from a file URL
-    /// - Parameter url: URL to image file
-    /// - Returns: Estimated calories (Float)
-    public func estimate(from url: URL) async throws -> Float {
-        let data = try Data(contentsOf: url)
-        return try await estimate(from: data)
-    }
-    
-    // MARK: - Batch Prediction
-    
-    /// Estimate calories for multiple images
-    /// - Parameter images: Array of food images
-    /// - Returns: Array of estimated calories
-    public func estimate(from images: [UIImage]) async throws -> [Float] {
-        var results: [Float] = []
-        for image in images {
-            let calories = try await estimate(from: image)
-            results.append(calories)
-        }
-        return results
-    }
-}
-
-// MARK: - Errors
-
-public enum CalorieCLIPError: LocalizedError {
-    case modelNotFound
-    case invalidImage
-    case predictionFailed
-    
-    public var errorDescription: String? {
-        switch self {
-        case .modelNotFound:
-            return "CalorieCLIP.mlmodelc not found in bundle"
-        case .invalidImage:
-            return "Invalid or corrupted image"
-        case .predictionFailed:
-            return "Failed to extract prediction from model output"
-        }
-    }
-}
-
-// MARK: - SwiftUI View Extension
-
-#if canImport(SwiftUI)
-import SwiftUI
-
-@available(iOS 15.0, macOS 12.0, *)
-public struct CalorieEstimateView: View {
-    let image: UIImage
-    @State private var calories: Float?
-    @State private var isLoading = false
-    @State private var error: Error?
-    
-    public init(image: UIImage) {
-        self.image = image
-    }
-    
-    public var body: some View {
-        VStack(spacing: 12) {
-            Image(uiImage: image)
-                .resizable()
-                .aspectRatio(contentMode: .fit)
-                .cornerRadius(12)
-            
-            if isLoading {
-                ProgressView("Analyzing...")
-            } else if let calories = calories {
-                HStack {
-                    Image(systemName: "flame.fill")
-                        .foregroundColor(.orange)
-                    Text("\(Int(calories)) calories")
-                        .font(.title2.bold())
-                }
-            } else if let error = error {
-                Text(error.localizedDescription)
-                    .foregroundColor(.red)
-            }
-        }
-        .task {
-            await estimateCalories()
-        }
-    }
-    
-    private func estimateCalories() async {
-        isLoading = true
-        defer { isLoading = false }
-        
-        do {
-            let model = try CalorieCLIP()
-            calories = try await model.estimate(from: image)
-        } catch {
-            self.error = error
-        }
-    }
-}
-#endif

kuzco/README.md

@@ -1,71 +0,0 @@
-# CalorieCLIP for Kuzco / iOS
-
-Swift integration for CalorieCLIP calorie estimation.
-
-## Setup
-
-### 1. Export CoreML Model
-
-```bash
-cd HaploLLC/CalorieCLIP
-pip install coremltools open-clip-torch
-python export_coreml.py --output CalorieCLIP.mlpackage
-```
-
-### 2. Add to Xcode Project
-
-1. Drag `CalorieCLIP.mlpackage` into your Xcode project
-2. Add `CalorieCLIP.swift` to your project
-3. Import and use:
-
-```swift
-import Foundation
-
-// Initialize
-let estimator = try CalorieCLIP()
-
-// Estimate from UIImage
-let calories = try await estimator.estimate(from: foodImage)
-print("Estimated: \(Int(calories)) calories")
-
-// Estimate from URL
-let calories = try await estimator.estimate(from: imageURL)
-
-// Batch estimation
-let results = try await estimator.estimate(from: [img1, img2, img3])
-```
-
-## SwiftUI Integration
-
-```swift
-import SwiftUI
-
-struct ContentView: View {
-    @State private var image: UIImage?
-    
-    var body: some View {
-        VStack {
-            if let image = image {
-                CalorieEstimateView(image: image)
-            }
-            
-            Button("Select Photo") {
-                // Photo picker logic
-            }
-        }
-    }
-}
-```
-
-## Performance
-
-| Metric | Value |
-|--------|-------|
-| MAE | 54.3 calories |
-| Inference Time | <50ms on iPhone 14 |
-| Model Size | ~80MB |
-
-## Requirements
-
-- iOS 15.0+ / macOS 12.0+
-- Xcode 14+