Lev

Lev is a 9B decision model for case law. You give it an excerpt of a court decision and one or more typed questions, and it returns a probability for every answer option. It does not generate text. It was built for citator work: deciding how a decision treats a case it cites, whether the cited case is an earlier stage of the same proceeding, and what happened to it on appeal. It also answers ten other classification questions about decisions and passages.

Lev is a fine-tune of d3-flash (Qwen3.5-9B backbone, Apache-2.0) and loads with the same runtime.

• Treatment of a cited case: 90.8% on agreed labels, against 80.7% for Jev 1.13 and 66.8% for the OpenAI Decisions API, on 238 held-out citations from decisions dated after the training data ends.
• Four citator questions: 96.4% (Jev 94.2%, OpenAI 89.2%, d3-flash 90.4%); all 13 questions: 96.4% (Jev 95.2%, OpenAI 93.1%, d3-flash 91.9%).
• Negative treatment: at a 0.2 threshold on the summed negative probability, recall 100% at precision 72% (Jev 100% at 57%, OpenAI 77% at 83%).
• Invented treatments: 1.9% of citations that are not the court's own receive a treatment (Jev 15.1%, OpenAI 7.5%).
• Calibration: expected calibration error 3.3% (Jev 4.1%, OpenAI 3.5%); 97% of citator answers can be accepted at 98% accuracy (Jev 96%, OpenAI 84%).
• Area of law (49 options): 93.2% (Jev 92.6%, OpenAI 91.3%). The difference from Jev is within the 95% interval (-2.5 to +3.7 points).
• Held-out legal tasks: 87.8% mean over 20 held-out LegalBench tasks from other areas of law (d3-flash 86.9%, Jev 83.7%, OpenAI 82.4%).
• Speed: a median of 591 ms to answer all questions about one excerpt on one A100 through the stock runtime.
• Open: Apache-2.0 weights, adapters, training and evaluation code, and a technical report.

Parameters	8.39B (the d3-flash checkpoint with merged adapters and a tuned readout)
Input	Text: an excerpt of a decision, up to a few thousand tokens
Questions	Choice (named options with definitions) and yes/no
Trained for	13 questions on Canadian court decisions in English, plus public legal classification tasks
License	Apache-2.0

Results

Scores are on a held-out test set of 1,028 excerpts from decisions dated after the training data ends, with 4,550 judgments in all. No version of Lev was scored on it until the weights were frozen, and the frozen model was scored once. Every system got the same excerpt, instructions and option definitions. The first table uses the 4,030 judgments on which two independent annotators agreed; the second uses every judgment, labelled by the first annotator only. Both annotators are models, not people. Details and caveats are in the technical report.

Agreed labels (two annotators agree), accuracy %

Task	n	Lev	Jev 1.13	OpenAI Decisions (gpt-6-luna)	d3-flash (base)
Treatment of the cited case	238	90.8	80.7	66.8	75.2
Appeal history	361	97.2	97.5	92.5	94.2
Reported negative history	371	97.6	98.7	97.6	97.0
Cited in dissent	386	100.0	100.0	100.0	95.3
Mean of the four citator tasks	1356	96.4	94.2	89.2	90.4
Disposition	297	89.9	90.2	91.6	88.2
Decision type	362	98.1	94.5	97.0	87.8
Proceeding	366	96.2	97.5	93.4	94.3
Area of law (49 options)	311	93.2	92.6	91.3	80.7
Publication ban	389	99.7	99.5	99.5	99.5
Self-represented party	371	99.2	98.4	97.6	98.7
Separate opinions	393	100.0	100.0	99.7	100.0
Mean of seven decision-level tasks	2489	96.6	96.1	95.7	92.7
Standard of review	77	94.8	96.1	94.8	93.5
Test-element finding	108	96.3	91.7	88.0	90.7
Mean of two passage-level tasks	185	95.6	93.9	91.4	92.1
Mean of all 13	4030	96.4	95.2	93.1	91.9

First annotator's labels, accuracy %

Task	n	Lev	Jev 1.13	OpenAI Decisions (gpt-6-luna)	d3-flash (base)
Treatment of the cited case	386	74.1	58.3	50.0	55.2
Appeal history	386	93.3	93.3	87.8	89.9
Reported negative history	386	95.1	95.6	95.6	94.8
Cited in dissent	386	100.0	100.0	100.0	95.3
Mean of the four citator tasks	1544	90.6	86.8	83.4	83.8
Disposition	394	81.2	76.4	75.1	73.4
Decision type	394	92.4	89.6	92.9	83.8
Proceeding	394	91.6	92.4	89.8	89.3
Area of law (49 options)	394	79.9	76.4	77.7	70.6
Publication ban	394	98.7	98.5	98.5	98.5
Self-represented party	394	94.7	93.7	93.1	93.7
Separate opinions	394	99.7	99.7	99.5	99.7
Mean of seven decision-level tasks	2758	91.2	89.5	89.5	87.0
Standard of review	128	78.9	62.5	60.9	59.4
Test-element finding	120	90.8	85.8	80.8	83.3
Mean of two passage-level tasks	248	84.9	74.2	70.9	71.4
Mean of all 13	4550	90.0	86.3	84.8	83.6

Averaged over the 13 questions on agreed labels, Lev scores 96.4%, Jev 95.2% and the OpenAI Decisions API 93.1%; the base model before fine-tuning scores 91.9%. Pooled over all agreed judgments, Lev is +0.9 points from Jev (95% interval +0.2 to +1.5) and +2.6 from OpenAI (+1.9 to +3.4). On the four citator questions the differences are +1.4 (+0.3 to +2.6) and +5.5 (+4.0 to +6.9).

• Treatment of the cited case is where Lev is strongest: 90.8% against 80.7% for Jev and 66.8% for OpenAI.
• Area of law: Lev scores 93.2%, ahead of Jev by 0.6 points and ahead of OpenAI by 1.9 points.
• Negative treatment (questioned, not followed, overruled, superseded by statute): flagging a citation when the four negative options sum to a probability of 0.2 or more, Lev catches 100% of the 13 negative treatments at 72% precision; Jev 100% at 57%; OpenAI 77% at 83%. Use the threshold, not the top answer, and set it on your own data.
• Invented treatments: for citations that are not the court's own (inside a quotation, in a party's argument, or not named), Lev assigns a treatment 1.9% of the time, Jev 15.1% and OpenAI 7.5%.

Held-out legal tasks

20 LegalBench tasks from other areas of law (contract clauses, privacy policies, legal-aid triage, supply chain disclosures, federal jurisdiction, statutory entailment), 1,890 items, each asked as a yes/no question with one plain instruction. The suite was assembled before the released model was trained, its tasks were left out of the training data, and the released model was scored on it once.

LegalBench task	n	Lev	Jev 1.13	OpenAI Decisions (gpt-6-luna)	d3-flash (base)
contract_nli_confidentiality_of_agreement	82	95.1	91.5	84.1	95.1
contract_nli_explicit_identification	100	92.0	92.0	86.0	91.0
contract_nli_notice_on_compelled_disclosure	100	98.0	99.0	97.0	97.0
contract_nli_permissible_copy	87	79.3	77.0	80.5	80.5
contract_nli_survival_of_obligations	100	95.0	92.0	94.0	96.0
cuad_anti-assignment	100	94.0	85.0	85.0	89.0
cuad_audit_rights	100	99.0	90.0	87.0	92.0
cuad_insurance	100	95.0	88.0	87.0	94.0
cuad_license_grant	100	91.0	90.0	92.0	90.0
cuad_termination_for_convenience	100	95.0	97.0	96.0	96.0
diversity_6	100	71.0	66.0	39.0	74.0
learned_hands_benefits	66	90.9	87.9	83.3	90.9
learned_hands_consumer	100	83.0	78.0	72.0	83.0
learned_hands_health	100	83.0	87.0	89.0	81.0
learned_hands_immigration	100	99.0	98.0	99.0	99.0
legal_reasoning_causality	55	81.8	60.0	80.0	83.6
opp115_data_retention	100	76.0	66.0	74.0	71.0
opp115_user_choice_control	100	82.0	79.0	82.0	79.0
sara_entailment	100	77.0	77.0	78.0	78.0
supply_chain_disclosure_best_practice_certification	100	78.0	74.0	64.0	77.0
Mean		87.8	83.7	82.4	86.9

Lev's mean is 87.8%, against 86.9% for its base model, 83.7% for Jev and 82.4% for OpenAI.

Judicial decisions from other courts

The Australian set has treatment labels written by the deciding judges; the others are LegalBench tasks about judicial decisions. None were used in training. This suite was also scored for several candidate models during development.

Task	n	Lev	Jev 1.13	OpenAI Decisions (gpt-6-luna)	d3-flash (base)
Treatment, Federal Court of Australia (judge-labelled)	818	44.4	40.8	38.0	39.7
Overruling (LegalBench)	300	94.7	93.3	92.0	95.3
Tax Court of Canada outcomes (LegalBench)	244	96.7	99.2	96.7	98.4
Function of decision section (LegalBench)	300	40.7	44.0	45.7	41.7
Oral argument question purpose (LegalBench)	300	43.0	44.3	47.7	42.0
SCALR holdings (LegalBench)	300	75.3	79.7	83.6	75.3
Mean of six		65.8	66.9	67.3	65.4

Nobody does well on the Australian labels. Every system is right about 85% of the time on whether the Australian court limited or rejected the case, and close to guessing on which positive label the judge chose. The report explains why: two careful readers of the same decision agree that a case was followed or applied only about half the time.

Quickstart

pip install "transformers==5.17.0" torch safetensors accelerate
pip install flash-linear-attention  # optional, faster linear-attention layers on GPU

from transformers import AutoModel

model = AutoModel.from_pretrained("jfloydian/lev", trust_remote_code=True)

excerpt = """Citing decision: Smith v. Jones, 2025 ONCA 100 | Court of Appeal for Ontario (appeal court, Ontario) | decided 2025-02-03
Cited case: Brown v. Green, 2010 ONSC 55

Paragraphs that cite the case, with neighbours:
[41] The respondent relies on Brown v. Green. With respect, that decision cannot stand with the later
guidance of this court in Grey, and I decline to follow it.
[42] The governing test is the one stated in Grey at para. 17."""

result = model.system_one(
    state=excerpt,
    questions={
        "treatment": {
            "type": "choice",
            "instructions": "How does the citing decision treat the cited case? If it treats the case in more than one way, choose the most negative treatment.",
            "criteria": {
                "followed": "The court treats the cited case as binding or adopts its rule as the governing rule, and decides the point with it.",
                "applied": "The court applies the cited case's rule or test to the facts before it.",
                "distinguished": "The court says the cited case does not govern because the facts or the law differ.",
                "not_followed": "The court declines to follow the cited case.",
                "referred_to": "The court cites the case in its own reasoning for a point, without more.",
            },
        },
        "dissent": {"type": "noul", "instructions": "Do dissenting reasons in the citing decision cite or treat the cited case?"},
    },
)
print(result["answers"]["treatment"]["choice"], result["answers"]["treatment"]["probabilities"])

The 13 questions Lev was trained on, with the exact instructions and option definitions, are in questions.py. Use that wording for the trained tasks. You can also ask your own questions with your own options, as with d3-flash; the public-task table above gives a sense of how far that carries.

How to build the excerpt

The model was trained on excerpts with a fixed layout, and it does best when you keep to it.

• For a citation pair: a line naming the citing decision, its court and date; a line naming the cited case; up to 500 characters of front matter; the first paragraph; the paragraphs that cite the case with one neighbour on each side; the last two paragraphs.
• For a decision-level question: the header line, up to 900 characters of front matter, the first three paragraphs and the last five.
• For a passage-level question: the header line, the issue or the test element, and the paragraphs that deal with it.

excerpts.py has the functions used to build all three.

The questions

Group	Question	Options
Citator	treatment	followed, applied, considered, distinguished, questioned, not_followed, overruled, superseded_by_statute, referred_to, inside_quotation, other_court_or_party, same_matter, not_named
Citator	history	not_same_matter, affirmed, reversed, reversed_in_part, varied, set_aside, restored, remitted, earlier_ruling
Citator	reported	none, overruled, reversed, superseded_by_statute, not_followed, doubted
Citator	dissent	yes / no
Decision	outcome	16 dispositions
Decision	decision_type	10 kinds of document
Decision	proceeding	11 kinds of proceeding
Decision	area	49 areas of law
Decision	publication_ban, self_represented, separate_opinions	yes / no
Passage	standard_of_review	correctness, reasonableness, palpable_and_overriding_error, patent_unreasonableness, other, not_stated
Passage	element_finding	met, not_met, not_decided

Three of the treatment options exist to keep a citator honest. inside_quotation is for a case that appears only in text the court is quoting. other_court_or_party is for a case that appears only in a party's argument or another court's reasoning. reported is a separate question for what the court says someone else did to the case. A model that reads all of these as the court's own treatment reports things that did not happen.

Calibration

A single temperature (1.6) was fitted on held-out data drawn at natural label frequencies and is stored in decision_config.json. Averaged over the 13 tasks, expected calibration error on agreed labels is 3.3% for Lev, 4.1% for Jev and 3.5% for OpenAI. Answers Lev gives with at least 90% confidence are right 100% of the time; in the 80 to 90% band, 88%. Ranking its citator answers by confidence, Lev can decide 97% of the agreed judgments before accuracy on the accepted set drops below 98% (Jev 96%, OpenAI 84%).

Training

Lev was trained on a private corpus of Canadian court decisions with model-assisted labels; test labels were adjudicated by a second, independent model. Training decisions are dated before 1 July 2024 and test decisions on or after it.

• Low-rank adapters (rank 64) on every linear layer of the upper 16 of 32 decoder layers, plus the readout, trained through a reimplementation of the d3 forward pass, since the released runtime is inference-only.
• 127,888 examples of Lev's own questions, 78,300 examples from public legal classification datasets (LegalBench tasks that are in none of the evaluation suites, ContractNLI, MAUD, ECtHR, LEDGAR, CaseHOLD, Supreme Court issue areas, unfair terms of service), and 17458 replay prompts answered beforehand by the base model and used as soft targets, so that the fine-tune keeps the base model's behaviour on tasks it was not tuned for.
• Yes/no questions are drawn near their real rates and many-option questions close to balanced. The order of the options is shuffled for every training example, so the model has to read the definitions.
• Six rounds. In each round several adapters are trained in parallel on different parts of the data from the same starting point and their weight updates are averaged; the next round starts from the average. Later rounds add examples drawn from training decisions not used before, with more weight on area of law, and keep each area-of-law example with a probability equal to how often two annotators agree on its label. The arms that make up the released model saw 287,032 examples in all.
• Training took about 20 A100 GPU-hours.
• Adapters are merged into the bfloat16 weights.
• Several of the held-out public tasks reported above come from source datasets (ContractNLI, CUAD, MAUD, OPP-115, terms of service) from which other examples were used in training. The tasks were held out and exact texts were removed; the source documents are not guaranteed unseen.

The adapters and the code are in jfloydian/lev-lora, and the technical report has the details, including the experiments that did not help.

Limitations

• Both annotators of the test set are models. "Agreed" means two independent model annotators gave the same answer. No person reviewed the labels. The public tasks above have human labels.
• The test set over-samples rare labels, and it is thin in the very rarest: it has no citation in a dissent and few reported negative histories. The scores describe how each system handles each kind of case, not how often it will be right on a random citation.
• The model was tuned against held-out data more than once. Two earlier evaluation sets were used to find and fix weaknesses, and a new set was drawn each time for the next measurement. The numbers above are from the last set, scored once.
• Canadian decisions in English with numbered paragraphs. Other jurisdictions, French-language decisions and differently built excerpts are outside what was tested, apart from the public tasks above.
• Excerpts hide things. Lev sees the paragraphs around a citation, not the decision. Positive treatment in particular is often only visible from how the rest of the decision uses the case.
• It classifies an excerpt; it is not a citator. It does not find citations, resolve them to cases, or check that the excerpt is complete. Treat its output as a lead to check against the decision. It is not legal advice.
• Images and video are supported by the d3 runtime it inherits, but Lev was trained and tested on text only.
• Hosted baselines change. Jev 1.13 and the OpenAI Decisions API (in public beta) were called over two days.

Files

• model-0000{1,2,3,4}-of-00004.safetensors, readout.safetensors, decision_config.json: the model in d3 package format.
• d3_*.py, modeling_d3.py, pipeline_d3.py: the d3 runtime, unchanged from d3-flash.
• questions.py, excerpts.py: the trained questions and the excerpt builders.
• lev-technical-report.pdf: the technical report.
• The LoRA adapters and the training and evaluation code are in jfloydian/lev-lora.

License and attribution

Apache-2.0. Lev is built on d3-flash by the vLLM Semantic Router team, which is built on Qwen3.5-9B; their licence and notice files are included. If you use Lev, please credit Jonathan Kleiman.

@misc{kleiman2026lev,
  title  = {Lev: a 9B open decision model for citator and case-law classification},
  author = {Kleiman, Jonathan},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/jfloydian/lev}}
}