Kwokou/BeeNara-Big

BeeNara Big 🐝 v0.4

Suggest the right folder in a whole folder tree, or say that none fits.

BeeNara Big reads a document (its text and, if available, the page image) together with the complete folder tree of a user (from 3 to several hundred folders) and returns a calibrated probability for every folder and for "none fits". It is the large sibling of BeeNara, which decides between at most 9 folders on a CPU.

Status: sorts on its own in small trees, suggests in large ones. For trees of 3–9 folders v0.4 is certified to sort automatically (at most 1 % wrong among automatic decisions, see Results). For larger trees the calibration found no such threshold, so v0.4 suggests folders there and the user confirms.

What is new in v0.4

• Automatic sorting for the first time. v0.3 was trained with label smoothing, and its calibration found a safe threshold for no list size. v0.4 is trained without label smoothing. It sorts 57 % of the held-out tasks with 3–9 folders on its own, and 0.9 % of those were wrong.
• Confidence that separates right from wrong: AUROC 0.89 (v0.3: 0.69–0.79 for trees up to 49 folders).
• English as good as German: 84.4 % in both languages (v0.3: 82 % vs. 87 %).
• More, and more varied, training data: 32 document types and documents from two teacher models that check each other's output.
• Calibration on 16,158 held-out tasks, about 3,000 per list size (v0.3: 7,115 in total, with very few large trees).

v0.3 was evaluated on a different held-out world, so its accuracy table is not directly comparable with the one below.

Quickstart

pip install torch "transformers>=5.17" peft safetensors tokenizers huggingface_hub pillow
hf download Kwokou/BeeNara-Big beenara_big.py --local-dir .

from beenara_big import BeeNaraBig

bb = BeeNaraBig.from_pretrained("Kwokou/BeeNara-Big")   # adapter 164 MB + Qwen3.5-2B-Base 4.6 GB, GPU if available

s = bb.suggest(
    "Einkommensteuerbescheid 2025\nFinanzamt Musterstadt\nFestgesetzte Einkommensteuer: 1.234,00 EUR",
    ["Finanzen/Steuer/2024", "Finanzen/Steuer/2025", "Finanzen/Bank", "Wohnung/Miete", "Arbeit/Verträge"],
)
s.suggestions     # [('Finanzen/Steuer/2025', ...), ('Finanzen/Steuer/2024', ...), ...]
s.none            # p("none of these folders fits")
s.category        # the folder, when the risk threshold allows automatic sorting; else None
s.ask             # True: show the suggestions and let the user decide

• Folder names are free text; paths like Finanzen/Steuer/2025 work best. A description can follow the name: "Wohnung/Miete — Mietvertrag, Nebenkosten".
• images=[...] adds page images (file paths or PIL images); language="en" switches the internal prompt to English.
• On a GPU the base model loads in bf16, on a CPU in fp32.

The previous version stays available under the tag v0.3.

Key points

• Whole folder trees. One forward pass over the complete tree, with folder paths such as Finanzen/Steuer/2025. In our tests a tree of 607 folders still fit into the context; one of more than 750 folders did not.
• Best up to about 50 folders: 89–94 % accuracy on held-out trees with 3–49 folders.
• Reads the page. The page image (letterhead, logo, layout) is used together with the text in half of the training tasks; it also works with text only.
• Calibrated. Expected calibration error 0.018.
• Stable under reordering. Shuffling the folder list changes the decision in 1.3 % of the tested tasks.

Results

Held-out "world": folder trees, folder names and documents that were never used for training. 2,400 tasks, 400 per list size, including tasks where no folder fits, where only a similar neighbour folder is present, and where the parent folder is the right answer. Calibrated, PyTorch bf16. v0.4 was trained twice with different seeds; this repository ships seed 2, chosen on the separate selection set, not on this benchmark.

Folders in the tree	Tasks	Accuracy, seed 2 (shipped)	Accuracy, seed 1	Sorted automatically, seed 2 (wrong)
3–9	400	92.3 %	90.0 %	57.0 % (0.9 %)
10–24	400	94.0 %	91.3 %	–
25–49	400	89.0 %	89.8 %	–
50–99	400	83.5 %	83.3 %	–
100–199	400	81.5 %	80.8 %	–
200 and more	400	66.3 %	65.0 %	–
All	2,400	84.4 %	83.3 %

"–": the calibration found no threshold for this list size; the model suggests, the user confirms. Seed 1 has an additional threshold for 10–24 folders (33 % automatic, 1.5 % of those wrong on the benchmark); it is not shipped.

Metric	Seed 2	Seed 1
"None fits" recall	74.2 %	73.2 %
"None fits" precision	81.8 %	80.2 %
Wrongly said "none fits"	5.2 %	5.7 %
Calibration error (ECE)	0.018	0.025
AUROC, right vs. wrong answer	0.89	0.86
Decision changes when the folder order is shuffled (300 tasks)	1.3 %	3.3 %

Speed and memory.

Laptop GPU (RTX PRO 1000, 8 GB), bf16, per document in a PollySort demo run	0.12–0.19 s
GPU memory, 8-bit weights, 300 folders	4.3 GB
CPU, 4 threads, 7 folders	2.5 s
CPU, 4 threads, 175 folders	87 s
CPU, 4 threads, 607 folders	187 s

On a CPU it is too slow for interactive use; plan for a GPU. CPU times depend strongly on the page image and the length of the folder names. The laptop GPU time was measured in PollySort with the v0.3 weights; v0.4 has the same architecture and size.

How it works

• Base: Qwen3.5-2B-Base (revision b1485b2f), a hybrid model with linear and full attention and a vision encoder.
• Input (variant "L2"): [task] [page image] [document text] [folder list] [folder list with one marker per folder] [none marker]. The folder list appears twice so that every folder marker sees the document and the complete list. Markers are existing special tokens; no new vocabulary.
• Head: a small decision head reads the hidden states at the markers and gives one score per folder and one for "none fits"; together they form one probability distribution.
• Limits: context up to 16,384 tokens; document text up to 2,048 tokens (start and end are kept); up to 32 tokens per folder name; page images up to about 1024 × 1448 pixels.
• Calibration: temperature scaling and risk thresholds per list size (Learn-then-Test, ≤ 1 % wrong among automatic decisions, δ = 0.1) on 16,158 tasks from the held-out world, about 3,000 per list size (1,165 for trees of 200 folders and more). Separate calibration files for bf16 and for 8-bit weights (kalibrierung.json, kalibrierung_8bit.json), each bound to its weights by SHA-256. The list-size buckets are stored in the calibration file.

Training

• Data. Only synthetic documents, no user data:
  • 30,503 documents in the training world after all checks, written by two teacher models or re-used from v0.3 and placed in new folder trees, plus 12,000 template documents assigned to trees by their metadata; every teacher document was checked against the facts it was built from, and the other teacher checked its document type;
  • 3,000 synthetic folder trees of 13–499 folders (German, English and mixed; private households, freelancers, small firms, construction offices, associations, law offices);
  • page images rendered for all documents; part of them degraded (scan, phone photo, skew) and read with Tesseract for OCR text.
• Episodes. Every step draws a task per document: the whole tree or a part of it (3–300 folders, 5 % up to 500), with folder descriptions in 30 % and spelling variants in 20 % of the tasks; in 25 % the right folder is missing, half of those with a similar neighbour present; in 15 % only the parent of the right folder is in the list, and the parent is the answer.
• Kept apart by tests. Separate names and trees for training and for the held-out world; normalized and substring overlap checks; 116 training documents that came too close to held-out ones were removed. No leakage remained.
• Setup: LoRA (r = 32, α = 64, dropout 0.05) on attention, linear attention and MLP of the language model; vision encoder frozen. Learning rate 1e-4 for LoRA and 3e-4 for the head, 3 % warmup, 2 epochs = 4,716 steps of 16 tasks, no label smoothing, gradient checkpointing, bf16. Best checkpoint by validation NLL (step 2,750).
• Hardware: one NVIDIA RTX A6000 (48 GB), about 27 hours.

Limitations

• Automatic sorting only for 3–9 folders. For larger trees the model is sure of some wrong answers, so no threshold keeps automatic decisions at ≤ 1 % wrong. Use it for suggestions there.
• Large trees are harder. Accuracy falls from about 90 % (up to 49 folders) to about 82 % (50–199) and 66 % for trees of 200 folders and more. Very large trees (in our tests, more than 750 folders) do not fit into the context.
• "None fits" is less reliable than in the small model (74 % recall, small model 92 %), especially when the missing folder's whole branch is absent (71 %).
• Weaker document types: insurance policies, bank statements, health insurance, rental contracts and telecom bills (72–77 %), and OCR text from degraded scans (75 %).
• Slow on a CPU (seconds to minutes per document), heavy with a page image.
• Synthetic training data. Real archives are messier. Expect a gap until the model is checked on real, consented documents.
• German and English only.

Files

File	Content	Size
lora.safetensors	LoRA weights for the language model	125 MB
kopf.safetensors	decision head	19 MB
big_config.json	base model and revision, LoRA settings, input limits, variant	< 1 KB
kalibrierung.json	calibration for bf16 weights	8 KB
kalibrierung_8bit.json	calibration for 8-bit weights	8 KB
tokenizer/	Qwen3.5 tokenizer	20 MB
bildprozessor/	image preprocessing settings	< 1 KB
beenara_big.py	loader and inference: input format, head, calibration	15 KB

The base model weights are not included; they come from Qwen/Qwen3.5-2B-Base at the revision in big_config.json.

Credits

Source	What BeeNara Big takes from it
Qwen3.5-2B-Base, Qwen team, Apache-2.0	the base model
LoRA, Hu et al. 2021	low-rank fine-tuning
laya, convaiinnovations, Apache-2.0	the idea of one marker per option, scored in one pass
Guo et al. 2017	temperature scaling
Angelopoulos et al. 2021, Learn then Test	the risk calibration
Xia et al., Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It	the effect that made us drop label smoothing
Qwen3.8-27B, Qwen team, Apache-2.0	wrote training and held-out documents and drafted folder trees; checked the other teacher's names, trees and document types; run locally
gpt-oss-120b, OpenAI, Apache-2.0	wrote training and held-out documents, all folder names, and drafted folder trees; checked the other teacher's trees and document types; run locally
Tesseract OCR, Apache-2.0	the OCR text in training

License

Apache-2.0, see LICENSE. The adapter derives from Qwen3.5-2B-Base, which is Apache-2.0 licensed; see NOTICE. Developed by Akinara as part of PollySort.