---
library_name: transformers
pipeline_tag: image-text-to-text
tags:
  - lfm2
  - vision
  - multimodal
  - food
  - calories
  - nutrition
  - onnx
  - transformers.js
license: apache-2.0
base_model: LiquidAI/LFM2.5-VL-450M
model-type: lfm2_vl
---

# OpenCal Base

**OpenCal Base** is a LFM2.5-VL-450M fine-tune that reads food photos (and text) and
outputs structured ingredient + gram-weight extractions aligned to the USDA food
database. It is exported to ONNX for the `@huggingface/transformers.js` browser
runtime (WebGPU / WASM / CPU).

## Model details

| Field | Value |
| --- | --- |
| Base model | [`LiquidAI/LFM2.5-VL-450M`](https://huggingface.co/LiquidAI/LFM2.5-VL-450M) |
| Fine-tune | LoRA, OpenCal v6 (gram-weight + kcal/macro target) |
| Architecture | `Lfm2VlForConditionalGeneration` (`model_type: lfm2_vl`) |
| Runtime | ONNX / transformers.js (WebGPU, WASM, CPU) |
| License | Apache-2.0 |

## Fine-tune summary

LoRA on top of the frozen base, merged weights. Target format is a JSON list of
`{name, grams, kcal, protein_g, carbs_g, fat_g}` extracted per item, with grams
normalized per 100 g and macros sourced from USDA data. The model is used by the
OpenCal app to turn a photo or a text description of a meal into per-item
nutrition.

### Benchmark results (OpenCal internal evals)

- **F101** (full meals, median gold kcal ≈ 293): kcal MAE **243**, WAPE **41.5%**,
  within-50% of gold **70.4%**.
- **N5k** (lab samples, median gold kcal ≈ 43): ingredient-identity recall
  baseline **76.4%**, this model **84.4%** (reference LLM **73.6%**).

## Usage (transformers.js)

```js
import { AutoModelForImageTextToText, AutoProcessor, RawImage } from '@huggingface/transformers';

const processor = await AutoProcessor.from_pretrained('OpenCal/opencal-base');
const model = await AutoModelForImageTextToText.from_pretrained('OpenCal/opencal-base', { dtype: 'auto', device: 'webgpu' });

const image = await RawImage.fromURL('https://example.com/meal.jpg');
const texts = ['List the ingredients and their amounts in grams'];

const { inputs } = await processor(image, texts, { return_tensor: false });
const { logits } = await model(inputs);
const decoded = processor.batch_decode(logits[1].id); // [1] for single sample
console.log(decoded[0]);
```

## File layout (ONNX)

The model is split into the standard LFM2.5-VL ONNX components. The suffix of each
weight file selects the runtime/precision:

| File | Precision | Use |
| --- | --- | --- |
| `embed_tokens{,_fp16,_q4,_quantized}.onnx` | fp32 / fp16 / q4 / q8 | input embeddings (`input_ids → inputs_embeds`) |
| `decoder_model_merged{,_fp16,_q4,_q4f16}.onnx` | fp32 / fp16 / q4 / q4f16 | text decoder (main body) |
| `vision_encoder{,_fp16,_q4,_quantized}.onnx` | fp32 / fp16 / q4 / q8 | vision encoder |

- `q4f16` — MatMulNBits q4 weights with fp16 scales / KV cache (WebGPU production path).
- `quantized` (= q8, `MatMulNBits`) — WASM path.
- `fp16` — 16-bit WebGPU path.

`tokenizer.json`, `preprocessor_config.json`, `config.json`, and
`generation_config.json` are at the repository root.

## Citation / base model

This model derives from `LiquidAI/LFM2.5-VL-450M`. Please see the base repository
for the original weights and paper.