d1 models on the Decision Index 0.3

Complete runs of the public 0.3 suite for two Liquid AI decision models, made with the Decision Index kit at 62d2f51 (branch v0.3) and scored by it.

Each run directory holds the kit's own files: results.jsonl.gz (compact), scores.json, index.json, benchmark-summary.json, environment.json and status.json.

How each model was run

d1_engine:D1 loads the model from the Hub with its own code (trust_remote_code=True) and answers every request with model.system_one(state, questions), the call documented on the model card. Nothing is shortened: a request the model cannot read whole is recorded as unsupported.

• d1-omni-600M reads at most 16,384 positions, and its encoder keeps a question's instructions and option texts to a fixed token budget. A request that its encoder would cut in any way (state, instructions or an option text) is unsupported: 37,141 of 140,178 scored requests. Its default batching needs about 52 GB of GPU memory on the longest requests.
• d1-3B reads every request whole. --option compile=true (CUDA graphs, NVIDIA only) changes 0.1% of top answers and is no faster on the varied shapes of this suite.

Reproduce

With the suite built as the kit's README describes:

git clone -b v0.3 https://github.com/apolinario/decision-index && cd decision-index && git checkout 62d2f51
pip install -e . "transformers==5.19.0" torch
cp <this repo>/code/d1_engine.py .
python -m decision_index pipeline --edition 0.3 --engine d1_engine:D1 --compact \
    --option model=LiquidAI/d1-3B --option revision=da1fe36a861f24690f27f622dca1d8688503d113 \
    --option dtype=bfloat16 --out runs/d1-3B
python -m decision_index pipeline --edition 0.3 --engine d1_engine:D1 --compact \
    --option model=LiquidAI/d1-omni-600M --option revision=414f8d6438174f5b2133a9c21a478fc42625e308 \
    --option dtype=float16 --out runs/d1-omni-600M

Images

Both models read images. D1(...)(state, questions, images=[...]) takes PIL images, file paths, raw bytes or data: URLs. With images, d1-omni-600M keeps 896 text tokens, and the engine applies the same rule as above.