Instructions to use VibeHash/laya-ledgar with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Laya
How to use VibeHash/laya-ledgar with Laya:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Download README.md from VibeHash/laya-ledgar: direct link, hf CLI and curl.
- Browser
- Download file 12.5 kB
-
https://huggingface.co/VibeHash/laya-ledgar/resolve/main/README.md
- Command line
-
hf download hf://VibeHash/laya-ledgar/README.md
-
curl -L -o README.md https://huggingface.co/VibeHash/laya-ledgar/resolve/main/README.md
license: apache-2.0
base_model: convaiinnovations/laya
datasets:
- coastalcph/lex_glue
language:
- en
pipeline_tag: text-classification
tags:
- laya
- ledgar
- lexglue
- legal
- contracts
- calibrated
- selective-prediction
model-index:
- name: laya-ledgar
results:
- task:
type: text-classification
name: Contract provision classification
dataset:
name: LEDGAR (LexGLUE)
type: coastalcph/lex_glue
config: ledgar
split: test
revision: c23fdff1a6bf74e0e1a71cb86f1e781d37da888c
metrics:
- type: f1
value: 0.87
name: Micro-F1 (equal to accuracy)
- type: f1
value: 0.8141
name: Macro-F1
- type: ece
value: 0.0418
name: Expected calibration error (15 bins)
laya-ledgar
A fine-tune of Laya that names which of 100 kinds of contract provision a paragraph is: governing law, indemnification, confidentiality, termination and 96 more. It reads provisions from real contracts filed with the US Securities and Exchange Commission, from the LEDGAR task of LexGLUE. It gives a probability for every type in one forward pass, and its confidence can be trusted. So it can tag the provisions it is sure about by itself and send the rest to a reviewer.
This is a sorting aid, not legal advice. It says what kind of provision a paragraph is, not whether it is fair, valid or standard.
Results
All 10,000 provisions of LEDGAR's test set, one fine-tune (seed 0), one RTX 4070 SUPER, 2026-11-18 to 19.
| Untuned Laya | This model | |
|---|---|---|
| Right (micro-F1) | 3,912 of 10,000 (39.1%) | 8,700 of 10,000 (87.0%) |
| 95% confidence interval | 38.2 to 40.1% | 86.3 to 87.6% |
| Macro-F1 (each type weighted the same) | 0.312 | 0.814 |
| Mean stated confidence | 0.807 | 0.907 |
| Calibration error (ECE, lower is better) | 0.416 | 0.042 |
It runs a little overconfident: it states 0.907 on average and is right 87.0% of the time. Provisions it scores 0.95 to 0.98 are right 93.8% of the time, and those at 0.98 or more 98.4%.
Tagging only when sure. A threshold was fitted on 2,000 training provisions the model never trained on, with
Laya's own rule, aiming at 99% right among the answers above it. (Laya's fine-tune fits it on the 400 provisions it
holds back, but its rule adds one error as a margin, so at 99% it needs about 100 sure answers in a row with no
mistake, and 400 provisions with LEDGAR's near-twin types did not give that: train.json records its result, 1.0,
which answers nothing.) The threshold fitted on the 2,000 is 0.9806, and it is the one in the model's config.
Applied once to the test:
- The model tags 3,639 of the 10,000 provisions (36.4%) by itself, and 3,582 of those are right (98.4%).
- The other 6,361 go to a reviewer, each with the model's best guess.
With the cut chosen on the test itself, which describes this test and is optimistic for new provisions: 95% right while tagging 76.0% of provisions, 98% right at 44.2%, 99% right at 21.7%.
Provisions seen in training. Contracts repeat boilerplate: 282 of the 10,000 test provisions also appear word for word in the training data. They are left in, as in the official split. On the other 9,718 the model is right 86.7% of the time, and on the 282, 97.9%.
Long provisions. The 100 options take 286 of the model's 768 tokens, which leaves 481 for the provision. 253 test provisions (2.5%) are longer, and the model reads only their start. It is right on 88.1% of those (223 of 253) and 87.0% of the rest.
Speed. One provision alone: median 22 ms, 95th percentile 26 ms. In batches of 32: about 19 ms a provision.
These were measured again on an idle GPU with the scorer's own timing code; eval.json records 47 ms for one
provision alone, timed while the GPU was busy with something else.
Against published results. LexGLUE (Chalkidis et al. 2022, Table 3, test set) reports micro-F1 / macro-F1 on LEDGAR of 87.0 / 81.4 for a TF-IDF SVM, 87.6 / 81.8 for BERT, 88.2 / 83.0 for Legal-BERT and 88.3 / 83.0 for CaseLaw-BERT; its Table 8 adds 88.6 / 83.6 for RoBERTa-large. Most of those models read up to 512 tokens of a provision (Longformer and BigBird up to 4,096), and each number is the best of five seeds, picked on the validation set. This model is one seed, not picked. Laya has 421 million parameters; the models in Table 3 have 110 to 149 million (LexGLUE Table 2). This model ties the lowest of them, the TF-IDF SVM, on both numbers (87.0 / 81.4) and is behind every BERT-style model: 0.6 to 1.6 points on micro-F1 and 0.4 to 2.2 on macro-F1. Its 95% interval, 86.3 to 87.6%, takes in the SVM's and BERT's micro-F1 but not Legal-BERT's, CaseLaw-BERT's or RoBERTa-large's. Its number also comes from a test read twice (see Training). What it adds is a calibrated confidence and the threshold, which LexGLUE does not score.
Where it is weakest. Near-twin types and catch-all ones. Applicable laws is right only 10 of 53 times (18.9%), most often taken for governing laws (35); indemnity 14 of 29 (48.3%), every miss taken for indemnifications; venues 8 of 20 (40%). The catch-alls are weak too: general 54.2% of 203, agreements 54.2%, miscellaneous 53.2%. The most common mix-ups are definitions taken for defined terms (37), applicable laws for governing laws (35), tax withholdings for withholdings (20), and waivers and no waivers for each other (20 and 19).
Use
pip install laya==0.4.31
import json
import laya
from huggingface_hub import hf_hub_download
REPO = "VibeHash/laya-ledgar"
model = laya.load(REPO)
with open(hf_hub_download(REPO, "questions.json")) as f:
questions = json.load(f)
text = "This Agreement shall be governed by and construed in accordance with the laws of the State of New York."
result = model.predict(text, questions, min_confidence=0.9806)
answer = result["answers"]["provision"]
if answer["abstention"] == "abstained":
print("send to a reviewer; best guess:", answer["choice"])
else:
print(answer["choice"], answer["answer_confidence"])
What it returned on the CPU, for five provisions written for this card:
| Provision | Answer | Confidence | At 0.9806 |
|---|---|---|---|
| This Agreement shall be governed by and construed in accordance with the laws of the State of New York. | governing laws | 0.8032 | to a reviewer |
| If any provision of this Agreement is held invalid or unenforceable, the remaining provisions shall continue in full force and effect. | severability | 0.9785 | to a reviewer |
| This Agreement may be executed in counterparts, each of which shall be deemed an original, and all of which together shall constitute one instrument. | counterparts | 0.9684 | to a reviewer |
| All notices under this Agreement shall be in writing and delivered by hand, by courier or by certified mail to the addresses set out above. | notices | 0.9883 | tagged |
| The Company shall pay the Executive a base salary of $250,000 per year, payable in accordance with the Company's normal payroll practices. | base salary | 0.9863 | tagged |
All five best guesses are the right type; three fall under the threshold and would go to a reviewer.
The results above were measured on a GPU, and this table on the CPU. On the first 100 test provisions the two gave the same answers; their confidences differed by 0.0005 at the median, by more than 0.01 for 9 provisions and by up to 0.08. A provision close to the threshold can fall on either side of it depending on the hardware.
- Ask the question in
questions.jsonexactly as written: the same instruction and the same 100 options in the same order.labels.jsonmaps each option back to LEDGAR's label name. - The model reads at most 768 tokens, options included. Pass
max_len=768if you call it with your own settings;laya.loadreads it from the saved config. min_confidenceis the threshold fitted in training. Leave it out to get an answer for every provision, or pick your own cut from the trade-off above.predict_batch(texts, questions, batch_size=32)scores many provisions in shared forward passes.- To make sure you load exactly these weights, pass
expected_sha256={"model.safetensors": "<hash from SHA256SUMS>"}tolaya.load.SHA256SUMSlists every file.
Limits
- English provisions from US public-company contracts filed with the SEC. Private contracts, other countries' law and other languages were not tested.
- Each provision is labelled with one type, the way LEDGAR labels it. A provision that does two jobs gets one name.
- It always picks one of the 100 types. Text that is not a contract provision was not tested.
- One fine-tune. How much a second seed would differ was not measured.
- The threshold was fitted on the 2,000 dev provisions, where 730 of the 736 answers above it were right (99.2%). On the test it gave 98.4% (3,582 of 3,639), under the 99% it aimed at.
Training
Laya's own laya.train.finetune at its default settings (4 epochs, effective batch 64, its learning rates), with
these changes:
head_max_len512 instead of 192, so the 100 options keep all their words (286 tokens).max_len768 instead of 512. At 512 only 225 tokens are left for the provision and 16.4% of test provisions are cut short; at 768, 481 are left and 2.5% are.target_error0.01 instead of 0.10, so the threshold aims at 99% right.- Plain cross-entropy (
soft-ce) instead of Laya's default RLCD loss, and 2,000 training provisions kept out as a dev set. These two came after a first model.
This is the second model, and the test was read twice. The first used RLCD for 2 epochs and was scored on the test once: 79.90% right, macro-F1 0.680. It got only 81.8% of its own training provisions right, so it had not learned them. Short runs on 8,000 training provisions, scored on 2,000 others with the test unread, then compared losses: RLCD 73.2%, plain cross-entropy 78.3%, cross-entropy at twice the learning rates 78.4%. This model is the cross-entropy recipe at Laya's own rates and epochs, judged on its dev set (88.45% right) before the test was read again. Nothing was chosen on the test, but a second reading is a second chance, and the number above is not as clean as a single one.
The options are LEDGAR's label names in lower case. It trained on 57,534 provisions. The 2,000 dev provisions, and
66 more with the same text as one of them, were kept out; Laya held back another 400 to fit its temperature. Training
took 433 minutes on one RTX 4070 SUPER, measured once, and the GPU may not have been free the whole time. LEDGAR's own validation split was not used. The test was read
only by the scorer, through laya.load(...).predict_batch, the same call a user makes.
Reproduce
The code that built and scored this model is in code/. Data, weights and runs go under ~/.laya-tasks/.
hf download VibeHash/laya-ledgar --include "code/*" --local-dir laya-ledgar
cd laya-ledgar/code
pip install torch --index-url https://download.pytorch.org/whl/cu126 # or the build for your GPU
pip install -r requirements.txt
python -m layatasks.data ledgar # LEDGAR at a pinned revision; writes and locks the test
python -m layatasks.fit ledgar # how many provisions are cut short, on the CPU
python -m layatasks.train ledgar --seed 0 # about 433 minutes on an RTX 4070 SUPER
python -m layatasks.evaluate ledgar --model ~/.laya-tasks/ledgar/runs/seed0/model --name seed0 --dev --fit-gate
python -m layatasks.evaluate ledgar --model ~/.laya-tasks/ledgar/runs/seed0/model --name seed0
Every source is pinned by revision and SHA-256, and the scorer refuses a test file whose SHA-256 changed.
eval.json holds the full scores behind this card, train.json the training record and fit.json the token
budget.
Licence and credits
Apache 2.0. A fine-tune of Laya by Convai Innovations (Apache 2.0); please cite their preprint,
doi:10.20944/preprints202611.0781.v1. Trained on LEDGAR as
distributed in LexGLUE (CC BY 4.0, per the LexGLUE dataset card): Tuggener et al., LEDGAR: A Large-Scale
Multi-label Corpus for Text Classification of Legal Provisions in Contracts, LREC 2020; Chalkidis et al., LexGLUE:
A Benchmark Dataset for Legal Language Understanding in English, ACL 2022. See NOTICE.