File size: 3,232 Bytes
4ecaaf8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 | ---
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- lfm2
- vision
- multimodal
- food
- calories
- nutrition
- onnx
- transformers.js
license: apache-2.0
base_model: LiquidAI/LFM2.5-VL-450M
model-type: lfm2_vl
---
# OpenCal Base
**OpenCal Base** is a LFM2.5-VL-450M fine-tune that reads food photos (and text) and
outputs structured ingredient + gram-weight extractions aligned to the USDA food
database. It is exported to ONNX for the `@huggingface/transformers.js` browser
runtime (WebGPU / WASM / CPU).
## Model details
| Field | Value |
| --- | --- |
| Base model | [`LiquidAI/LFM2.5-VL-450M`](https://huggingface.co/LiquidAI/LFM2.5-VL-450M) |
| Fine-tune | LoRA, OpenCal v6 (gram-weight + kcal/macro target) |
| Architecture | `Lfm2VlForConditionalGeneration` (`model_type: lfm2_vl`) |
| Runtime | ONNX / transformers.js (WebGPU, WASM, CPU) |
| License | Apache-2.0 |
## Fine-tune summary
LoRA on top of the frozen base, merged weights. Target format is a JSON list of
`{name, grams, kcal, protein_g, carbs_g, fat_g}` extracted per item, with grams
normalized per 100 g and macros sourced from USDA data. The model is used by the
OpenCal app to turn a photo or a text description of a meal into per-item
nutrition.
### Benchmark results (OpenCal internal evals)
- **F101** (full meals, median gold kcal ≈ 293): kcal MAE **243**, WAPE **41.5%**,
within-50% of gold **70.4%**.
- **N5k** (lab samples, median gold kcal ≈ 43): ingredient-identity recall
baseline **76.4%**, this model **84.4%** (reference LLM **73.6%**).
## Usage (transformers.js)
```js
import { AutoModelForImageTextToText, AutoProcessor, RawImage } from '@huggingface/transformers';
const processor = await AutoProcessor.from_pretrained('OpenCal/opencal-base');
const model = await AutoModelForImageTextToText.from_pretrained('OpenCal/opencal-base', { dtype: 'auto', device: 'webgpu' });
const image = await RawImage.fromURL('https://example.com/meal.jpg');
const texts = ['List the ingredients and their amounts in grams'];
const { inputs } = await processor(image, texts, { return_tensor: false });
const { logits } = await model(inputs);
const decoded = processor.batch_decode(logits[1].id); // [1] for single sample
console.log(decoded[0]);
```
## File layout (ONNX)
The model is split into the standard LFM2.5-VL ONNX components. The suffix of each
weight file selects the runtime/precision:
| File | Precision | Use |
| --- | --- | --- |
| `embed_tokens{,_fp16,_q4,_quantized}.onnx` | fp32 / fp16 / q4 / q8 | input embeddings (`input_ids → inputs_embeds`) |
| `decoder_model_merged{,_fp16,_q4,_q4f16}.onnx` | fp32 / fp16 / q4 / q4f16 | text decoder (main body) |
| `vision_encoder{,_fp16,_q4,_quantized}.onnx` | fp32 / fp16 / q4 / q8 | vision encoder |
- `q4f16` — MatMulNBits q4 weights with fp16 scales / KV cache (WebGPU production path).
- `quantized` (= q8, `MatMulNBits`) — WASM path.
- `fp16` — 16-bit WebGPU path.
`tokenizer.json`, `preprocessor_config.json`, `config.json`, and
`generation_config.json` are at the repository root.
## Citation / base model
This model derives from `LiquidAI/LFM2.5-VL-450M`. Please see the base repository
for the original weights and paper.
|