zeus / README.md
Heliusnetwork's picture
Zeus v22: adapter and model card from measured files
25638ae verified
|
Raw History Blame Contribute Delete
3.52 kB
metadata
license: apache-2.0
base_model: mistralai/Mistral-7B-Instruct-v0.2
library_name: peft
pipeline_tag: text-generation
tags:
  - fact-checking
  - claim-verification
  - lora
  - olo
language:
  - en

Zeus v22: open claim verification

Zeus reads a claim and a few passages and says whether the passages support it, contradict it or do not cover it. It quotes the exact words that carry the evidence. It is the checker behind Olo, which tests every sentence of an answer against sources from the open Olo Commons. It is built in the open: the data pipeline, training code and every measured score are in Hashim57/olo-commons under zeus/.

Measured scores

Measured on 2026-11-16 for this exact version by zeus/eval.py. Nothing here is estimated. The held-out sets come first because the frozen first split has been at or near 100 percent for weeks and no longer shows progress.

Set Label accuracy
VitaminC dev (300 human-written claims, held out) 73.3%
Commons eval v3 (frozen, hard patterns, n=200) 92.0%
Commons eval v2 (frozen, harder split, n=200) 100.0%
Commons eval (frozen first split, n=200) 99.5%

Evidence hit on the frozen split (quoted span found in the right passage): 99.5%. JSON valid: 100.0%.

Training

  • Base model: mistralai/Mistral-7B-Instruct-v0.2 (Apache 2.0).
  • Method: LoRA rank 8 on q_proj, v_proj, learning rate 0.0002, batch 16, 1034 optimiser steps in 60.0 minutes on one A100-40GB.
  • Training examples: 36,255.
  • Trained by the public GitHub workflow zeus-train.yml on a cloud GPU; the full log of every version is zeus/STATUS.md.

Data and licences

Three parts, described in zeus/DATA_CARD.md: examples made by rule from Olo Commons atoms (each atom keeps its own source licence), hand-written gold examples checked against the atoms they quote, and human-written claim and evidence pairs from VitaminC (Schuster, Fisch and Barzilay, NAACL 2021, CC BY-SA 3.0). Text from Wikipedia and VitaminC is CC BY-SA: attribute Wikipedia and the VitaminC authors. The adapter is released under Apache 2.0, like its base model.

Use

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2", torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(base, "Heliusnetwork/zeus")
tok = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")

Send one user turn with the system prompt from zeus/cloud/recipe.py followed by:

Claim: <claim>

Passages:
[p1] (<source>) <text>

[p2] (<source>) <text>

The answer is one JSON object: {"label": "supported" | "contradicted" | "not_covered", "evidence": [{"passage": "<id>", "quote": "<exact words>"}], "rationale": "<one sentence>"}.

Limits

Most training contradictions are made by rule (one changed relation, year, number or name), so subtle contradictions are harder for it than clear ones; the weakest pattern in each version is reported in the weekly learning report. It checks claims against the passages it is given and knows nothing beyond them. It is not a substitute for reading the source: every Olo answer carries its passages so a person can.

Card generated by zeus/cloud/publish_hf.py on 2026-11-17 from zeus/out/zeus-v22/.