laya-ledgar / README.md
VibeHash's picture
Card: CPU confidences differ slightly from the GPU scores
394981a verified
|
Raw History Blame Contribute Delete
12.5 kB
---
license: apache-2.0
base_model: convaiinnovations/laya
datasets:
- coastalcph/lex_glue
language:
- en
pipeline_tag: text-classification
tags:
- laya
- ledgar
- lexglue
- legal
- contracts
- calibrated
- selective-prediction
model-index:
- name: laya-ledgar
results:
- task:
type: text-classification
name: Contract provision classification
dataset:
name: LEDGAR (LexGLUE)
type: coastalcph/lex_glue
config: ledgar
split: test
revision: c23fdff1a6bf74e0e1a71cb86f1e781d37da888c
metrics:
- type: f1
value: 0.8700
name: Micro-F1 (equal to accuracy)
- type: f1
value: 0.8141
name: Macro-F1
- type: ece
value: 0.0418
name: Expected calibration error (15 bins)
---
# laya-ledgar
A fine-tune of [Laya](https://huggingface.co/convaiinnovations/laya) that names which of 100 kinds of contract
provision a paragraph is: governing law, indemnification, confidentiality, termination and 96 more. It reads
provisions from real contracts filed with the US Securities and Exchange Commission, from the LEDGAR task of
[LexGLUE](https://huggingface.co/datasets/coastalcph/lex_glue). It gives a probability for every type in one
forward pass, and its confidence can be trusted. So it can tag the provisions it is sure about by itself and send the
rest to a reviewer.
This is a sorting aid, not legal advice. It says what kind of provision a paragraph is, not whether it is fair,
valid or standard.
## Results
All 10,000 provisions of LEDGAR's test set, one fine-tune (seed 0), one RTX 4070 SUPER, 2026-11-18 to 19.
| | Untuned Laya | This model |
|---|---|---|
| Right (micro-F1) | 3,912 of 10,000 (39.1%) | 8,700 of 10,000 (87.0%) |
| 95% confidence interval | 38.2 to 40.1% | 86.3 to 87.6% |
| Macro-F1 (each type weighted the same) | 0.312 | 0.814 |
| Mean stated confidence | 0.807 | 0.907 |
| Calibration error (ECE, lower is better) | 0.416 | 0.042 |
It runs a little overconfident: it states 0.907 on average and is right 87.0% of the time. Provisions it scores 0.95
to 0.98 are right 93.8% of the time, and those at 0.98 or more 98.4%.
**Tagging only when sure.** A threshold was fitted on 2,000 training provisions the model never trained on, with
Laya's own rule, aiming at 99% right among the answers above it. (Laya's fine-tune fits it on the 400 provisions it
holds back, but its rule adds one error as a margin, so at 99% it needs about 100 sure answers in a row with no
mistake, and 400 provisions with LEDGAR's near-twin types did not give that: `train.json` records its result, 1.0,
which answers nothing.) The threshold fitted on the 2,000 is **0.9806**, and it is the one in the model's config.
Applied once to the test:
- The model tags 3,639 of the 10,000 provisions (36.4%) by itself, and 3,582 of those are right (98.4%).
- The other 6,361 go to a reviewer, each with the model's best guess.
With the cut chosen on the test itself, which describes this test and is optimistic for new provisions: 95% right
while tagging 76.0% of provisions, 98% right at 44.2%, 99% right at 21.7%.
**Provisions seen in training.** Contracts repeat boilerplate: 282 of the 10,000 test provisions also appear word
for word in the training data. They are left in, as in the official split. On the other 9,718 the model is right
86.7% of the time, and on the 282, 97.9%.
**Long provisions.** The 100 options take 286 of the model's 768 tokens, which leaves 481 for the provision. 253
test provisions (2.5%) are longer, and the model reads only their start. It is right on 88.1% of those (223 of
253) and 87.0% of the rest.
**Speed.** One provision alone: median 22 ms, 95th percentile 26 ms. In batches of 32: about 19 ms a provision.
These were measured again on an idle GPU with the scorer's own timing code; `eval.json` records 47 ms for one
provision alone, timed while the GPU was busy with something else.
**Against published results.** LexGLUE (Chalkidis et al. 2022, Table 3, test set) reports micro-F1 / macro-F1 on
LEDGAR of 87.0 / 81.4 for a TF-IDF SVM, 87.6 / 81.8 for BERT, 88.2 / 83.0 for Legal-BERT and 88.3 / 83.0 for
CaseLaw-BERT; its Table 8 adds 88.6 / 83.6 for RoBERTa-large. Most of those models read up to 512 tokens of a
provision (Longformer and BigBird up to 4,096), and each number is the best of five seeds, picked on the validation
set. This model is one seed, not picked. Laya has 421 million parameters; the models in Table 3 have 110 to 149
million (LexGLUE Table 2). This model ties the lowest of them, the TF-IDF SVM, on both numbers (87.0 / 81.4) and
is behind every BERT-style model: 0.6 to 1.6 points on micro-F1 and 0.4 to 2.2 on macro-F1. Its 95% interval,
86.3 to 87.6%, takes in the SVM's and BERT's micro-F1 but not Legal-BERT's, CaseLaw-BERT's or RoBERTa-large's. Its
number also comes from a test read twice (see Training). What it adds is a calibrated confidence and the threshold,
which LexGLUE does not score.
**Where it is weakest.** Near-twin types and catch-all ones. Applicable laws is right only 10 of 53 times (18.9%),
most often taken for governing laws (35); indemnity 14 of 29 (48.3%), every miss taken for indemnifications; venues
8 of 20 (40%). The catch-alls are weak too: general 54.2% of 203, agreements 54.2%, miscellaneous 53.2%. The most
common mix-ups are definitions taken for defined terms (37), applicable laws for governing laws (35), tax
withholdings for withholdings (20), and waivers and no waivers for each other (20 and 19).
## Use
```bash
pip install laya==0.4.31
```
```python
import json
import laya
from huggingface_hub import hf_hub_download
REPO = "VibeHash/laya-ledgar"
model = laya.load(REPO)
with open(hf_hub_download(REPO, "questions.json")) as f:
questions = json.load(f)
text = "This Agreement shall be governed by and construed in accordance with the laws of the State of New York."
result = model.predict(text, questions, min_confidence=0.9806)
answer = result["answers"]["provision"]
if answer["abstention"] == "abstained":
print("send to a reviewer; best guess:", answer["choice"])
else:
print(answer["choice"], answer["answer_confidence"])
```
What it returned on the CPU, for five provisions written for this card:
| Provision | Answer | Confidence | At 0.9806 |
|---|---|---|---|
| This Agreement shall be governed by and construed in accordance with the laws of the State of New York. | governing laws | 0.8032 | to a reviewer |
| If any provision of this Agreement is held invalid or unenforceable, the remaining provisions shall continue in full force and effect. | severability | 0.9785 | to a reviewer |
| This Agreement may be executed in counterparts, each of which shall be deemed an original, and all of which together shall constitute one instrument. | counterparts | 0.9684 | to a reviewer |
| All notices under this Agreement shall be in writing and delivered by hand, by courier or by certified mail to the addresses set out above. | notices | 0.9883 | tagged |
| The Company shall pay the Executive a base salary of $250,000 per year, payable in accordance with the Company's normal payroll practices. | base salary | 0.9863 | tagged |
All five best guesses are the right type; three fall under the threshold and would go to a reviewer.
The results above were measured on a GPU, and this table on the CPU. On the first 100 test provisions the two gave
the same answers; their confidences differed by 0.0005 at the median, by more than 0.01 for 9 provisions and by up to
0.08. A provision close to the threshold can fall on either side of it depending on the hardware.
- Ask the question in `questions.json` exactly as written: the same instruction and the same 100 options in the same
order. `labels.json` maps each option back to LEDGAR's label name.
- The model reads at most 768 tokens, options included. Pass `max_len=768` if you call it with your own settings;
`laya.load` reads it from the saved config.
- `min_confidence` is the threshold fitted in training. Leave it out to get an answer for every provision, or pick
your own cut from the trade-off above.
- `predict_batch(texts, questions, batch_size=32)` scores many provisions in shared forward passes.
- To make sure you load exactly these weights, pass
`expected_sha256={"model.safetensors": "<hash from SHA256SUMS>"}` to `laya.load`. `SHA256SUMS` lists every file.
## Limits
- English provisions from US public-company contracts filed with the SEC. Private contracts, other countries' law
and other languages were not tested.
- Each provision is labelled with one type, the way LEDGAR labels it. A provision that does two jobs gets one name.
- It always picks one of the 100 types. Text that is not a contract provision was not tested.
- One fine-tune. How much a second seed would differ was not measured.
- The threshold was fitted on the 2,000 dev provisions, where 730 of the 736 answers above it were right (99.2%).
On the test it gave 98.4% (3,582 of 3,639), under the 99% it aimed at.
## Training
Laya's own `laya.train.finetune` at its default settings (4 epochs, effective batch 64, its learning rates), with
these changes:
- `head_max_len` 512 instead of 192, so the 100 options keep all their words (286 tokens).
- `max_len` 768 instead of 512. At 512 only 225 tokens are left for the provision and 16.4% of test provisions are
cut short; at 768, 481 are left and 2.5% are.
- `target_error` 0.01 instead of 0.10, so the threshold aims at 99% right.
- Plain cross-entropy (`soft-ce`) instead of Laya's default RLCD loss, and 2,000 training provisions kept out as a
dev set. These two came after a first model.
**This is the second model, and the test was read twice.** The first used RLCD for 2 epochs and was scored on the
test once: 79.90% right, macro-F1 0.680. It got only 81.8% of its own training provisions right, so it had not
learned them. Short runs on 8,000 training provisions, scored on 2,000 others with the test unread, then compared
losses: RLCD 73.2%, plain cross-entropy 78.3%, cross-entropy at twice the learning rates 78.4%. This model is the
cross-entropy recipe at Laya's own rates and epochs, judged on its dev set (88.45% right) before the test was read
again. Nothing was chosen on the test, but a second reading is a second chance, and the number above is not as clean
as a single one.
The options are LEDGAR's label names in lower case. It trained on 57,534 provisions. The 2,000 dev provisions, and
66 more with the same text as one of them, were kept out; Laya held back another 400 to fit its temperature. Training
took 433 minutes on one RTX 4070 SUPER, measured once, and the GPU may not have been free the whole time. LEDGAR's own validation split was not used. The test was read
only by the scorer, through `laya.load(...).predict_batch`, the same call a user makes.
## Reproduce
The code that built and scored this model is in `code/`. Data, weights and runs go under `~/.laya-tasks/`.
```bash
hf download VibeHash/laya-ledgar --include "code/*" --local-dir laya-ledgar
cd laya-ledgar/code
pip install torch --index-url https://download.pytorch.org/whl/cu126 # or the build for your GPU
pip install -r requirements.txt
python -m layatasks.data ledgar # LEDGAR at a pinned revision; writes and locks the test
python -m layatasks.fit ledgar # how many provisions are cut short, on the CPU
python -m layatasks.train ledgar --seed 0 # about 433 minutes on an RTX 4070 SUPER
python -m layatasks.evaluate ledgar --model ~/.laya-tasks/ledgar/runs/seed0/model --name seed0 --dev --fit-gate
python -m layatasks.evaluate ledgar --model ~/.laya-tasks/ledgar/runs/seed0/model --name seed0
```
Every source is pinned by revision and SHA-256, and the scorer refuses a test file whose SHA-256 changed.
`eval.json` holds the full scores behind this card, `train.json` the training record and `fit.json` the token
budget.
## Licence and credits
Apache 2.0. A fine-tune of Laya by Convai Innovations (Apache 2.0); please cite their preprint,
[doi:10.20944/preprints202611.0781.v1](https://doi.org/10.20944/preprints202611.0781.v1). Trained on LEDGAR as
distributed in LexGLUE (CC BY 4.0, per the LexGLUE dataset card): Tuggener et al., *LEDGAR: A Large-Scale
Multi-label Corpus for Text Classification of Legal Provisions in Contracts*, LREC 2020; Chalkidis et al., *LexGLUE:
A Benchmark Dataset for Legal Language Understanding in English*, ACL 2022. See `NOTICE`.