docs/batch-annotation-experiment.md

Batch Annotation Experiment

Background

The evaluation script (scripts/evaluate_llm.py) originally called the LLM once per <bibl> record — one round-trip per citation. For a gold standard of 162 records this means 162 sequential API calls, creating considerable latency.

The question: does sending multiple records in a single LLM call degrade annotation quality, and does the answer differ by model?

Research (BatchPrompt, ICLR 2024) suggests batch prompting can maintain competitive accuracy while reducing token overhead by 18–30 %. The primary known risk is the lost in the middle effect: models tend to perform less reliably on items that appear in the middle of a long context window.

Implementation

A --batch-size N flag was added to evaluate_llm.py (default 1, fully backward-compatible). A --timeout SECONDS flag controls the HTTP read timeout per API call (default 120).

When batch_size > 1 the evaluator:

1. Collects the plain text of N consecutive <bibl> elements.

2. Joins them with a distinctive ASCII sentinel:

   \n---RECORD|||SEP|||BOUNDARY---\n
   

3. Passes the combined text to the existing annotate() pipeline as a single call.

4. Splits the annotated XML output on the same sentinel to recover N per-record fragments.

5. Evaluates each fragment against its gold standard independently.

6. Falls back to empty predictions for the entire batch if the separator count mismatches.

The separator is guaranteed to survive the annotation pass unchanged because inject_xml() never modifies text characters — it only inserts XML tags. However, very complex prompts can cause a model to omit or duplicate the separator, triggering the fallback.

Experiment Setup

• Gold standard: tests/fixtures/blbl-examples.tei.xml (162 <bibl> records)
• Match mode: text (normalised whitespace)
• GLiNER: disabled

Two models were tested:

Model	Provider	Inference speed
Gemini 2.0 Flash	Google	Fast (~3–6 s/record)
llama-3.3-70b-instruct	KISSKI / AcademicCloud	Slow (~14–20 s/record)

Note on concurrency: Runs against the same KISSKI API key must be sequential — concurrent jobs cause GPU resource contention, inflating response times and triggering timeouts.

---

Results summary

Model	Batch size	Records	Timeout	Micro F1	Macro F1	API calls
Gemini 2.0 Flash	1	30	120 s	0.825	0.754	30
Gemini 2.0 Flash	10	30	120 s	0.847	0.779	3
Gemini 2.0 Flash	1	162	120 s	—	—	—
Gemini 2.0 Flash	10	162	120 s	0.744	0.627	17
KISSKI Llama 3.3 70b	1	162	120 s	0.867	0.636	162
KISSKI Llama 3.3 70b	10	162	120 s	timed out	—	—
KISSKI Llama 3.3 70b	10	162	600 s	0.854	0.619	17

A full Gemini batch-1 run on 162 records has not been executed.

---

Detailed results: Gemini 2.0 Flash

30-record pilot (batch sizes 1 and 10)

Metric	Batch-1	Batch-10	Δ
Micro F1	0.825	0.847	+0.022
Micro Precision	0.807	0.825	+0.018
Micro Recall	0.843	0.870	+0.027
Macro F1	0.754	0.779	+0.025
API calls	30	3	−90 %
Wall time	~2:14	~2:21	+7 s

On the 30-record pilot, batch-10 was slightly better across every metric.

Full 162-record run (batch-10)

Metric	30-rec pilot (batch-10)	162-rec full (batch-10)	Δ
Micro F1	0.847	0.744	−0.103
Micro Precision	0.825	0.799	−0.026
Micro Recall	0.870	0.696	−0.174
Macro F1	0.779	0.627	−0.152

Quality degraded significantly on the full dataset. Inspection of the worst records reveals that five consecutive entries (#141–145, all within the same batch) scored F1=0.000 with every gold span missed and nothing predicted. This is the batch split failure pattern: the model omitted the separator in that prompt, the mismatch fallback triggered, and all 10 records in the batch received empty predictions.

Per-element breakdown (Gemini, 162 records, batch-10)

Element	Batch-10 F1	TP	FP	FN
surname	0.878	258	6	66
label	0.842	8	0	3
date	0.832	131	16	37
forename	0.820	242	27	79
pubPlace	0.794	54	15	13
ptr	0.750	3	0	2
biblScope	0.758	111	12	59
publisher	0.750	45	10	20
title	0.634	149	55	117
author	0.568	96	88	58
idno	0.667	1	0	1
note	0.359	7	10	15
orgName	0.300	6	23	5
editor	0.448	15	21	16

---

Detailed results: KISSKI llama-3.3-70b-instruct

Full 162-record run, all batch sizes

Batch size	Timeout	Records completed	Micro F1	Macro F1	Notes
1	120 s	162 / 162	0.867	0.636	Full, reliable
10	120 s	22 / 162	0.843*	0.672*	14/16 batches timed out
50	120 s	0 / 162	—	—	All timed out
10	600 s	162 / 162	0.854	0.619	Full run with raised timeout

* Biased sample — only 3 of 16 batches completed; not representative.

Batch-1 vs batch-10 (600 s) — full 162 records

Metric	Batch-1	Batch-10 (600 s)	Δ
Micro F1	0.867	0.854	−0.013
Micro Precision	0.876	0.869	−0.007
Micro Recall	0.858	0.839	−0.019
Macro F1	0.636	0.619	−0.017
API calls	162	17	−90 %
Wall time	~43 min	~57 min	+14 min

A small but consistent quality drop across the board. No complete batch failures (no batch scored all-zero), confirming Llama respects the separator more reliably than Gemini on this dataset.

Per-element comparison (KISSKI, 162 records)

Element	Batch-1 F1	Batch-10 F1	Δ
forename	0.954	0.945	−0.009
surname	0.983	0.972	−0.011
date	0.949	0.913	−0.036
editor	0.793	0.828	+0.035
biblScope	0.847	0.804	−0.043
pubPlace	0.853	0.855	+0.002
publisher	0.825	0.794	−0.031
author	0.778	0.778	0
title	0.743	0.733	−0.010
note	0.485	0.485	0
orgName	0.421	0.457	+0.036
label	0.235	0.222	−0.013

Most regressions are small (< 5 pp). editor and orgName actually improved slightly in batch mode.

---

Conclusions

Batch size is viable but model-dependent

Model	Batch-10 outcome	Recommendation
Gemini 2.0 Flash	−10 pp F1 at scale; risk of complete batch failure	Use batch-1 for full runs; batch-10 acceptable for quick pilots
KISSKI Llama 3.3 70b	−1.3 pp F1; no batch failures; requires 600 s timeout	Use --batch-size 10 --timeout 600

Key findings

1. Small batches on small datasets look deceptively good. Gemini batch-10 on 30 records was better than batch-1 (+2 pp). On 162 records it was significantly worse (−10 pp). The 30-record pilot was not representative because it happened to avoid the complex, multi-author batches near the end of the corpus.

2. Batch failure is catastrophic, not graceful. When Gemini loses the separator in a complex batch, all 10 records score 0.000 — a hard floor that drags down corpus-level metrics far more than occasional per-record errors.

3. Llama respects the separator more reliably. With adequate timeout (600 s), all 162 batches completed and there were no separator-loss events. The quality drop is small (−1.3 pp) and the latency gain is real (162 → 17 API calls).

4. Wall time does not improve with batching for slow models. Llama batch-10 took longer than batch-1 (~57 min vs ~43 min) because each batch generates ~10× the tokens in a single call. The benefit is fewer API calls, which matters for rate-limit management, not raw speed.

5. The --timeout flag is essential for slow models. Without --timeout 600, Llama batch-10 timed out on 87 % of batches.

Recommendations

Scenario	Command
Gemini, quality-critical full run	--provider gemini --batch-size 1
Gemini, quick pilot (≤ 30 records)	--provider gemini --batch-size 10
KISSKI Llama, full run (balanced)	--provider kisski --batch-size 10 --timeout 600
KISSKI Llama, full run (highest quality)	--provider kisski --batch-size 1