How to use from the
Use from the
Laya library
# No code snippets available yet for this library.

# To use this model, check the repository files and the library's documentation.

# Want to help? PRs adding snippets are welcome at:
# https://github.com/huggingface/huggingface.js

laya-ledgar

A fine-tune of Laya that names which of 100 kinds of contract provision a paragraph is: governing law, indemnification, confidentiality, termination and 96 more. It reads provisions from real contracts filed with the US Securities and Exchange Commission, from the LEDGAR task of LexGLUE. It gives a probability for every type in one forward pass, and its confidence can be trusted. So it can tag the provisions it is sure about by itself and send the rest to a reviewer.

This is a sorting aid, not legal advice. It says what kind of provision a paragraph is, not whether it is fair, valid or standard.

Results

All 10,000 provisions of LEDGAR's test set, one fine-tune (seed 0), one RTX 4070 SUPER, 2026-11-18 to 19.

Untuned Laya This model
Right (micro-F1) 3,912 of 10,000 (39.1%) 8,700 of 10,000 (87.0%)
95% confidence interval 38.2 to 40.1% 86.3 to 87.6%
Macro-F1 (each type weighted the same) 0.312 0.814
Mean stated confidence 0.807 0.907
Calibration error (ECE, lower is better) 0.416 0.042

It runs a little overconfident: it states 0.907 on average and is right 87.0% of the time. Provisions it scores 0.95 to 0.98 are right 93.8% of the time, and those at 0.98 or more 98.4%.

Tagging only when sure. A threshold was fitted on 2,000 training provisions the model never trained on, with Laya's own rule, aiming at 99% right among the answers above it. (Laya's fine-tune fits it on the 400 provisions it holds back, but its rule adds one error as a margin, so at 99% it needs about 100 sure answers in a row with no mistake, and 400 provisions with LEDGAR's near-twin types did not give that: train.json records its result, 1.0, which answers nothing.) The threshold fitted on the 2,000 is 0.9806, and it is the one in the model's config. Applied once to the test:

  • The model tags 3,639 of the 10,000 provisions (36.4%) by itself, and 3,582 of those are right (98.4%).
  • The other 6,361 go to a reviewer, each with the model's best guess.

With the cut chosen on the test itself, which describes this test and is optimistic for new provisions: 95% right while tagging 76.0% of provisions, 98% right at 44.2%, 99% right at 21.7%.

Provisions seen in training. Contracts repeat boilerplate: 282 of the 10,000 test provisions also appear word for word in the training data. They are left in, as in the official split. On the other 9,718 the model is right 86.7% of the time, and on the 282, 97.9%.

Long provisions. The 100 options take 286 of the model's 768 tokens, which leaves 481 for the provision. 253 test provisions (2.5%) are longer, and the model reads only their start. It is right on 88.1% of those (223 of 253) and 87.0% of the rest.

Speed. One provision alone: median 22 ms, 95th percentile 26 ms. In batches of 32: about 19 ms a provision. These were measured again on an idle GPU with the scorer's own timing code; eval.json records 47 ms for one provision alone, timed while the GPU was busy with something else.

Against published results. LexGLUE (Chalkidis et al. 2022, Table 3, test set) reports micro-F1 / macro-F1 on LEDGAR of 87.0 / 81.4 for a TF-IDF SVM, 87.6 / 81.8 for BERT, 88.2 / 83.0 for Legal-BERT and 88.3 / 83.0 for CaseLaw-BERT; its Table 8 adds 88.6 / 83.6 for RoBERTa-large. Most of those models read up to 512 tokens of a provision (Longformer and BigBird up to 4,096), and each number is the best of five seeds, picked on the validation set. This model is one seed, not picked. Laya has 421 million parameters; the models in Table 3 have 110 to 149 million (LexGLUE Table 2). This model ties the lowest of them, the TF-IDF SVM, on both numbers (87.0 / 81.4) and is behind every BERT-style model: 0.6 to 1.6 points on micro-F1 and 0.4 to 2.2 on macro-F1. Its 95% interval, 86.3 to 87.6%, takes in the SVM's and BERT's micro-F1 but not Legal-BERT's, CaseLaw-BERT's or RoBERTa-large's. Its number also comes from a test read twice (see Training). What it adds is a calibrated confidence and the threshold, which LexGLUE does not score.

Where it is weakest. Near-twin types and catch-all ones. Applicable laws is right only 10 of 53 times (18.9%), most often taken for governing laws (35); indemnity 14 of 29 (48.3%), every miss taken for indemnifications; venues 8 of 20 (40%). The catch-alls are weak too: general 54.2% of 203, agreements 54.2%, miscellaneous 53.2%. The most common mix-ups are definitions taken for defined terms (37), applicable laws for governing laws (35), tax withholdings for withholdings (20), and waivers and no waivers for each other (20 and 19).

Use

pip install laya==0.4.31
import json

import laya
from huggingface_hub import hf_hub_download

REPO = "VibeHash/laya-ledgar"
model = laya.load(REPO)
with open(hf_hub_download(REPO, "questions.json")) as f:
    questions = json.load(f)

text = "This Agreement shall be governed by and construed in accordance with the laws of the State of New York."
result = model.predict(text, questions, min_confidence=0.9806)
answer = result["answers"]["provision"]
if answer["abstention"] == "abstained":
    print("send to a reviewer; best guess:", answer["choice"])
else:
    print(answer["choice"], answer["answer_confidence"])

What it returned on the CPU, for five provisions written for this card:

Provision Answer Confidence At 0.9806
This Agreement shall be governed by and construed in accordance with the laws of the State of New York. governing laws 0.8032 to a reviewer
If any provision of this Agreement is held invalid or unenforceable, the remaining provisions shall continue in full force and effect. severability 0.9785 to a reviewer
This Agreement may be executed in counterparts, each of which shall be deemed an original, and all of which together shall constitute one instrument. counterparts 0.9684 to a reviewer
All notices under this Agreement shall be in writing and delivered by hand, by courier or by certified mail to the addresses set out above. notices 0.9883 tagged
The Company shall pay the Executive a base salary of $250,000 per year, payable in accordance with the Company's normal payroll practices. base salary 0.9863 tagged

All five best guesses are the right type; three fall under the threshold and would go to a reviewer.

The results above were measured on a GPU, and this table on the CPU. On the first 100 test provisions the two gave the same answers; their confidences differed by 0.0005 at the median, by more than 0.01 for 9 provisions and by up to 0.08. A provision close to the threshold can fall on either side of it depending on the hardware.

  • Ask the question in questions.json exactly as written: the same instruction and the same 100 options in the same order. labels.json maps each option back to LEDGAR's label name.
  • The model reads at most 768 tokens, options included. Pass max_len=768 if you call it with your own settings; laya.load reads it from the saved config.
  • min_confidence is the threshold fitted in training. Leave it out to get an answer for every provision, or pick your own cut from the trade-off above.
  • predict_batch(texts, questions, batch_size=32) scores many provisions in shared forward passes.
  • To make sure you load exactly these weights, pass expected_sha256={"model.safetensors": "<hash from SHA256SUMS>"} to laya.load. SHA256SUMS lists every file.

Limits

  • English provisions from US public-company contracts filed with the SEC. Private contracts, other countries' law and other languages were not tested.
  • Each provision is labelled with one type, the way LEDGAR labels it. A provision that does two jobs gets one name.
  • It always picks one of the 100 types. Text that is not a contract provision was not tested.
  • One fine-tune. How much a second seed would differ was not measured.
  • The threshold was fitted on the 2,000 dev provisions, where 730 of the 736 answers above it were right (99.2%). On the test it gave 98.4% (3,582 of 3,639), under the 99% it aimed at.

Training

Laya's own laya.train.finetune at its default settings (4 epochs, effective batch 64, its learning rates), with these changes:

  • head_max_len 512 instead of 192, so the 100 options keep all their words (286 tokens).
  • max_len 768 instead of 512. At 512 only 225 tokens are left for the provision and 16.4% of test provisions are cut short; at 768, 481 are left and 2.5% are.
  • target_error 0.01 instead of 0.10, so the threshold aims at 99% right.
  • Plain cross-entropy (soft-ce) instead of Laya's default RLCD loss, and 2,000 training provisions kept out as a dev set. These two came after a first model.

This is the second model, and the test was read twice. The first used RLCD for 2 epochs and was scored on the test once: 79.90% right, macro-F1 0.680. It got only 81.8% of its own training provisions right, so it had not learned them. Short runs on 8,000 training provisions, scored on 2,000 others with the test unread, then compared losses: RLCD 73.2%, plain cross-entropy 78.3%, cross-entropy at twice the learning rates 78.4%. This model is the cross-entropy recipe at Laya's own rates and epochs, judged on its dev set (88.45% right) before the test was read again. Nothing was chosen on the test, but a second reading is a second chance, and the number above is not as clean as a single one.

The options are LEDGAR's label names in lower case. It trained on 57,534 provisions. The 2,000 dev provisions, and 66 more with the same text as one of them, were kept out; Laya held back another 400 to fit its temperature. Training took 433 minutes on one RTX 4070 SUPER, measured once, and the GPU may not have been free the whole time. LEDGAR's own validation split was not used. The test was read only by the scorer, through laya.load(...).predict_batch, the same call a user makes.

Reproduce

The code that built and scored this model is in code/. Data, weights and runs go under ~/.laya-tasks/.

hf download VibeHash/laya-ledgar --include "code/*" --local-dir laya-ledgar
cd laya-ledgar/code
pip install torch --index-url https://download.pytorch.org/whl/cu126   # or the build for your GPU
pip install -r requirements.txt
python -m layatasks.data ledgar              # LEDGAR at a pinned revision; writes and locks the test
python -m layatasks.fit ledgar               # how many provisions are cut short, on the CPU
python -m layatasks.train ledgar --seed 0    # about 433 minutes on an RTX 4070 SUPER
python -m layatasks.evaluate ledgar --model ~/.laya-tasks/ledgar/runs/seed0/model --name seed0 --dev --fit-gate
python -m layatasks.evaluate ledgar --model ~/.laya-tasks/ledgar/runs/seed0/model --name seed0

Every source is pinned by revision and SHA-256, and the scorer refuses a test file whose SHA-256 changed. eval.json holds the full scores behind this card, train.json the training record and fit.json the token budget.

Licence and credits

Apache 2.0. A fine-tune of Laya by Convai Innovations (Apache 2.0); please cite their preprint, doi:10.20944/preprints202611.0781.v1. Trained on LEDGAR as distributed in LexGLUE (CC BY 4.0, per the LexGLUE dataset card): Tuggener et al., LEDGAR: A Large-Scale Multi-label Corpus for Text Classification of Legal Provisions in Contracts, LREC 2020; Chalkidis et al., LexGLUE: A Benchmark Dataset for Legal Language Understanding in English, ACL 2022. See NOTICE.

Downloads last month
16
Safetensors
Model size
0.4B params
Tensor type
F16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for VibeHash/laya-ledgar

Finetuned
(258)
this model

Dataset used to train VibeHash/laya-ledgar

Evaluation results