Add detailed explanation

.gitignore

@@ -8,4 +8,5 @@ wheels/
 
 # Virtual environments
 .venv
-.env*
+.env*
+.DS_Store

README.md

@@ -12,6 +12,73 @@ Works with any inference endpoint through an injected `call_fn: (str) -> str`
 
 ---
 
+## Pipeline diagram
+
+```text
+  Input text (may contain XML markup)
+               │
+               ▼
+  ┌────────────────────────────────────┐
+  │  Strip existing XML tags           │  pipeline.py
+  └──────────────┬─────────────────────┘
+                 │
+                 ▼  (optional)
+  ┌────────────────────────────────────┐
+  │  GLiNER pre-detection              │  detection/
+  └──────────────┬─────────────────────┘
+                 │
+                 ▼
+  ┌────────────────────────────────────┐
+  │  Chunk text                        │  chunking/
+  └──────────────┬─────────────────────┘
+                 │
+          ╔══════╧══════╗
+          ║  per chunk  ║
+          ╚══════╤══════╝
+                 │
+                 ▼
+  ┌────────────────────────────────────┐
+  │  Build LLM prompt                  │  prompting/
+  └──────────────┬─────────────────────┘
+                 │
+                 ▼
+  ┌────────────────────────────────────┐
+  │  LLM inference                     │  inference/
+  └──────────────┬─────────────────────┘
+                 │
+                 ▼
+  ┌────────────────────────────────────┐
+  │  Parse JSON response               │  postprocessing/
+  └──────────────┬─────────────────────┘
+                 │
+        ╔════════╧════════╗
+        ║ merge all chunks ║
+        ╚════════╤════════╝
+                 │
+                 ▼
+  ┌────────────────────────────────────┐
+  │  Resolve spans → char offsets      │  postprocessing/
+  ├────────────────────────────────────┤
+  │  Validate against schema           │  postprocessing/
+  ├────────────────────────────────────┤
+  │  Inject XML tags                   │  postprocessing/
+  └──────────────┬─────────────────────┘
+                 │
+                 ▼
+  Annotated XML output
+```
+
+Detailed documentation for each stage:
+[Data models](tei_annotator/models/README.md) ·
+[GLiNER detection](tei_annotator/detection/README.md) ·
+[Chunking](tei_annotator/chunking/README.md) ·
+[Prompt building](tei_annotator/prompting/README.md) ·
+[Inference configuration](tei_annotator/inference/README.md) ·
+[Post-processing](tei_annotator/postprocessing/README.md) ·
+[Evaluation](tei_annotator/evaluation/README.md)
+
+---
+
 ## Installation
 
 Requires Python ≥ 3.12 and [uv](https://docs.astral.sh/uv/).

tei_annotator/chunking/README.md

@@ -0,0 +1,60 @@
+# Chunking
+
+Long texts exceed the context window (or the practical attention span) of most LLMs. The chunker splits the source text into overlapping windows so that each LLM call sees a manageable piece of text while entity boundaries between windows are never lost.
+
+---
+
+## API
+
+```python
+from tei_annotator.chunking.chunker import chunk_text, Chunk
+
+chunks: list[Chunk] = chunk_text(text, chunk_size=1500, chunk_overlap=200)
+```
+
+Each `Chunk` is a dataclass:
+
+```python
+@dataclass
+class Chunk:
+    text: str          # content of this window
+    start_offset: int  # character position of chunk[0] in the original text
+```
+
+`start_offset` is used by the resolver to convert chunk-local character positions back to positions in the full source text.
+
+---
+
+## Splitting algorithm
+
+1. Walk the text left-to-right, accumulating characters.
+2. When the accumulated length reaches `chunk_size`, scan backwards for the last whitespace character at or before that position and cut there. The boundary is always between words, never mid-token.
+3. The **next chunk begins** `chunk_overlap` characters before the cut point, so consecutive chunks share a strip of text.
+4. Repeat until all text is consumed.
+
+The final chunk is whatever remains after the last cut, even if it is shorter than `chunk_size`.
+
+---
+
+## XML safety
+
+If the source text contains existing XML markup, the splitter is tag-aware: it never places a cut inside an XML tag. Tags are treated as **zero-width** for length accounting — only visible text characters count toward `chunk_size`. This ensures that pre-existing markup is always preserved intact within whichever chunk it falls in.
+
+---
+
+## Why overlap matters
+
+Without overlap, an entity that straddles the boundary between two chunks could be missed by both LLM calls — the first call sees the entity's opening but not its close, and vice versa. With `chunk_overlap` characters of shared context, the entity appears complete in at least one chunk with enough surrounding text for the model to recognise and annotate it.
+
+Because the overlap causes the same entity to appear in two consecutive chunks, the pipeline deduplicates resolved spans after collecting results from all chunks: identical `(element, start, end)` triples are merged and only one instance is kept.
+
+---
+
+## Tuning
+
+| Parameter | Default | Effect |
+|-----------|---------|--------|
+| `chunk_size` | `1500` | Larger values send more text per LLM call (cheaper) but risk exceeding context limits. Reduce for models with small windows. |
+| `chunk_overlap` | `200` | Larger values make boundary entities safer but increase the number of duplicate detections that must be deduplicated. |
+
+A `chunk_size` of 1500 characters corresponds to roughly 375–500 tokens for Latin-script text, leaving ample room for the prompt preamble and JSON response within a 4 k-token limit.

tei_annotator/detection/README.md

@@ -0,0 +1,67 @@
+# GLiNER pre-detection (optional)
+
+[GLiNER](https://github.com/urchade/GLiNER) is a zero-shot named-entity recognition model that runs locally on CPU. The pre-detection pass uses GLiNER to quickly surface entity candidates before the (slower, more expensive) LLM annotation pass.
+
+---
+
+## Why a pre-detection pass?
+
+Pre-detection serves two purposes:
+
+1. **Improved recall** — GLiNER is fast and broad. It catches many candidate spans that the LLM can then verify, correct, and extend. This is especially useful in longer texts where lower-salience entities might otherwise be missed.
+
+2. **Cost reduction** — By surfacing strong candidates in the prompt, the LLM can spend less effort on exhaustive detection and more on attribute filling and disambiguation.
+
+The step is entirely optional: passing `gliner_model=None` to `annotate()` skips it and the LLM works from the raw text alone.
+
+---
+
+## How it works
+
+`detect_spans(text, schema, model_name)` in `gliner_detector.py`:
+
+1. **Label mapping** — Each `TEIElement.description` becomes a GLiNER label string. For example, `TEIElement(tag="persName", description="a person's name")` produces the label `"a person's name"`. GLiNER's zero-shot design means no fine-tuning is needed — labels are matched by semantic similarity at inference time.
+
+2. **Inference** — The GLiNER model runs span prediction on the full source text using those labels. It returns a list of `(text, label, score)` triples.
+
+3. **Context windowing** — For each detected span, a context window of ±60 characters is extracted from the source text to populate `SpanDescriptor.context`. This context string is what the resolver later uses to locate the span in the source text when converting it to a character offset.
+
+4. **Output** — Each detection becomes a `SpanDescriptor(element=<tag>, text=<matched text>, context=<window>, score=<confidence>)`.
+
+---
+
+## Candidates in the LLM prompt
+
+The `SpanDescriptor` list is passed to the prompt builder, which renders the candidates as a JSON block inside the LLM prompt. The LLM is instructed to treat them as suggestions, not ground truth: it may confirm, correct, split, merge, or discard them based on its own reading of the text.
+
+---
+
+## Installation
+
+The GLiNER integration is an optional dependency group:
+
+```bash
+uv sync --extra gliner
+```
+
+This installs `gliner`, PyTorch, and Hugging Face Transformers. Without this extra, importing `detect_spans` raises `ImportError` with a clear message. The `annotate()` function imports GLiNER lazily, so the base package always loads without it.
+
+---
+
+## Choosing a model
+
+Any GLiNER-compatible Hugging Face model can be used. Recommended starting points:
+
+| Model | Download size | Notes |
+|-------|--------------|-------|
+| `numind/NuNER_Zero` | ~200 MB | Good general-purpose zero-shot NER; default in the README examples |
+| `urchade/gliner_medium-v2.1` | ~400 MB | Broader entity type coverage |
+| `urchade/gliner_large-v2.1` | ~800 MB | Higher accuracy on ambiguous entities |
+
+Models are downloaded automatically from Hugging Face Hub on first use and cached in the default HF cache directory (`~/.cache/huggingface/`).
+
+---
+
+## Performance characteristics
+
+GLiNER inference is CPU-based and typically completes in milliseconds to low seconds per text chunk, depending on text length and hardware. It is orders of magnitude faster than a remote LLM call. For batch evaluation workloads the GLiNER pass adds negligible overhead.

tei_annotator/evaluation/README.md

@@ -0,0 +1,143 @@
+# Evaluation
+
+The `evaluation` module measures how accurately the annotator reproduces a hand-annotated (gold-standard) TEI XML file. It computes **precision**, **recall**, and **F1 score** per element type and in aggregate.
+
+---
+
+## Evaluation flow
+
+For each gold element (e.g. a `<biblStruct>` record in the gold file):
+
+```
+Gold XML element
+      │
+      ▼
+ 1. extract_spans()   →  gold EvaluationSpans + plain text   (extractor.py)
+      │
+      ▼
+ 2. annotate()        →  predicted XML string                (pipeline.py)
+      │
+      ▼
+ 3. extract_spans()   →  predicted EvaluationSpans           (extractor.py)
+      │
+      ▼
+ 4. compute_metrics() →  EvaluationResult                    (metrics.py)
+```
+
+Because the annotator receives *exactly the same plain text* that the gold spans are anchored to, character offsets align without any additional normalisation.
+
+---
+
+## Span extraction (`extractor.py`)
+
+### `EvaluationSpan`
+
+```python
+@dataclass
+class EvaluationSpan:
+    tag: str    # element name, e.g. "author"
+    start: int  # start offset in the element's plain text (inclusive)
+    end: int    # end offset (exclusive)
+    text: str   # raw text content
+```
+
+`normalized_text` is a computed property that collapses runs of internal whitespace to a single space and strips leading/trailing whitespace. It is used by the `TEXT` match mode to compare spans independent of formatting differences.
+
+### `extract_spans(element)`
+
+Walks the lxml element tree depth-first. At each descendant element it records the cumulative plain-text length seen so far as `start`, then adds the element's own text content to get `end`. Tail text (text following a closing tag but before the next sibling's opening tag) is accumulated but not itself attributed to a span.
+
+Returns `(plain_text: str, spans: list[EvaluationSpan])` — the plain text is what `annotate()` will receive; the spans are what it should ideally produce.
+
+---
+
+## Match modes (`metrics.py`)
+
+Three strategies determine when a predicted span "matches" a gold span:
+
+| Mode | Match condition |
+|------|-----------------|
+| `EXACT` | Same element tag **and** identical `(start, end)` offsets |
+| `TEXT` *(default)* | Same element tag **and** `normalized_text` is equal |
+| `OVERLAP` | Same element tag **and** intersection-over-union of offset ranges ≥ threshold (default 0.5) |
+
+`TEXT` mode is the most useful in practice: it is invariant to small character-position differences (which can arise from whitespace normalisation or fuzzy span matching) while still requiring the annotator to have found the right text.
+
+`EXACT` is strictest and useful for regression testing. `OVERLAP` is the most lenient and is appropriate when partial matches should count as correct.
+
+### Greedy matching algorithm
+
+`match_spans(gold, predicted, mode)`:
+
+1. Enumerate all `(gold_span, predicted_span)` candidate pairs that share the same element tag.
+2. Score each pair:
+   - `EXACT` / `TEXT` → 1.0 if matching, 0.0 otherwise.
+   - `OVERLAP` → IoU value in [0, 1].
+3. Sort by score descending, then greedily assign: take the highest-scoring unmatched pair, mark both spans as matched.
+4. Return `SpanMatch` objects for matched pairs, plus lists of unmatched gold and unmatched predicted spans.
+
+---
+
+## Metrics (`metrics.py`)
+
+### `ElementMetrics`
+
+Per-element-type counts (`tp`, `fp`, `fn`) and derived rates:
+
+- **Precision** = TP / (TP + FP)
+- **Recall** = TP / (TP + FN)
+- **F1** = 2 · P · R / (P + R)
+
+### `EvaluationResult`
+
+Aggregates `SpanMatch` objects and per-element `ElementMetrics`. Exposes two averaging strategies:
+
+| Strategy | How computed | When to prefer |
+|----------|-------------|----------------|
+| **Micro** | Sum TP / FP / FN across all element types, then compute P / R / F1 | Imbalanced element distributions — frequent types dominate the score, which reflects real-world impact |
+| **Macro** | Compute P / R / F1 per element type, then average | All element types weighted equally regardless of frequency — highlights performance on rare types |
+
+`result.report()` prints a formatted table with both averages and a per-element breakdown.
+
+### `aggregate(results)`
+
+Merges a list of `EvaluationResult` objects (one per document record) into a single corpus-level result by summing TP / FP / FN counts across all records before computing rates.
+
+---
+
+## High-level API (`evaluator.py`)
+
+### `evaluate_bibl(element, schema, endpoint, match_mode)`
+
+Evaluates annotation of a single lxml `_Element`. Handles:
+
+- XML parsing errors in the annotated output (caught and counted as zero predictions).
+- Literal `<` / `>` characters in the annotated text that are not valid XML tags: `_escape_nonschema_brackets()` escapes any angle bracket that is not part of a known schema element tag, so lxml can parse the result without error.
+
+### `evaluate_file(gold_xml_path, schema, endpoint, match_mode, max_items)`
+
+Evaluates an entire TEI XML file:
+
+1. Parses the XML file with lxml.
+2. Finds all first-level child elements of the root.
+3. Calls `evaluate_bibl()` on each, up to `max_items`.
+4. Returns `(list[EvaluationResult], EvaluationResult)` — individual results per record and the corpus-level aggregate.
+
+---
+
+## Example output
+
+```
+=== Evaluation Results ===
+Micro  P=0.821  R=0.754  F1=0.786  (TP=83  FP=18  FN=27)
+Macro  P=0.834  R=0.762  F1=0.791
+
+Per-element breakdown:
+  author               P=0.923  R=0.960  F1=0.941  (TP=24  FP=2  FN=1)
+  biblScope            P=0.750  R=0.600  F1=0.667  (TP=12  FP=4  FN=8)
+  date                 P=0.867  R=0.867  F1=0.867  (TP=13  FP=2  FN=2)
+  title                P=0.900  R=0.818  F1=0.857  (TP=18  FP=2  FN=4)
+  ...
+```
+
+The `matched`, `unmatched_gold`, and `unmatched_pred` lists on each `EvaluationResult` are available for detailed error analysis beyond the summary table.

tei_annotator/inference/README.md

@@ -0,0 +1,143 @@
+# Inference configuration
+
+The annotator is endpoint-agnostic: it talks to any language model (or extraction model) through a single `call_fn: (str) -> str` callable. `EndpointConfig` wires together a capability declaration and that callable.
+
+---
+
+## `EndpointCapability`
+
+```python
+from tei_annotator import EndpointCapability
+```
+
+| Value | When to use |
+|-------|-------------|
+| `TEXT_GENERATION` | Standard chat/completion LLM. JSON is requested via the prompt. If the response cannot be parsed, the pipeline sends a self-correction follow-up and retries once. |
+| `JSON_ENFORCED` | Constrained-decoding endpoint that guarantees syntactically valid JSON output (e.g. a vLLM server with `--guided-decoding-backend`). The correction retry is skipped because output is always parseable. |
+| `EXTRACTION` | Native extraction model (GLiNER2 / NuExtract-style). The raw source text is passed directly; no Jinja2 prompt is built. Used internally when `gliner_model=` is set on `annotate()`; do not wrap these models in `EndpointConfig`. |
+
+---
+
+## `EndpointConfig`
+
+```python
+from tei_annotator import EndpointConfig, EndpointCapability
+
+endpoint = EndpointConfig(
+    capability=EndpointCapability.TEXT_GENERATION,
+    call_fn=my_call_fn,
+)
+```
+
+`call_fn` receives the complete prompt string and must return the model's raw response string. Any implementation is valid — an `openai.Client`, an `anthropic.Anthropic` client, a local `requests.post` to Ollama, or a function that reads from a file for testing.
+
+---
+
+## Examples
+
+### Anthropic (Claude)
+
+```python
+import anthropic
+
+client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from environment
+
+def call_fn(prompt: str) -> str:
+    msg = client.messages.create(
+        model="claude-opus-4-6",
+        max_tokens=4096,
+        messages=[{"role": "user", "content": prompt}],
+    )
+    return msg.content[0].text
+
+endpoint = EndpointConfig(
+    capability=EndpointCapability.TEXT_GENERATION,
+    call_fn=call_fn,
+)
+```
+
+### OpenAI
+
+```python
+from openai import OpenAI
+
+client = OpenAI()  # reads OPENAI_API_KEY from environment
+
+def call_fn(prompt: str) -> str:
+    resp = client.chat.completions.create(
+        model="gpt-4o-mini",
+        messages=[{"role": "user", "content": prompt}],
+    )
+    return resp.choices[0].message.content
+
+endpoint = EndpointConfig(
+    capability=EndpointCapability.TEXT_GENERATION,
+    call_fn=call_fn,
+)
+```
+
+### Google Gemini
+
+```python
+import google.generativeai as genai
+
+genai.configure(api_key=os.environ["GEMINI_API_KEY"])
+model = genai.GenerativeModel("gemini-2.0-flash")
+
+def call_fn(prompt: str) -> str:
+    return model.generate_content(prompt).text
+
+endpoint = EndpointConfig(
+    capability=EndpointCapability.TEXT_GENERATION,
+    call_fn=call_fn,
+)
+```
+
+### Ollama (local)
+
+```python
+import requests
+
+def call_fn(prompt: str) -> str:
+    resp = requests.post(
+        "http://localhost:11434/api/generate",
+        json={"model": "llama3.1", "prompt": prompt, "stream": False},
+    )
+    return resp.json()["response"]
+
+endpoint = EndpointConfig(
+    capability=EndpointCapability.TEXT_GENERATION,
+    call_fn=call_fn,
+)
+```
+
+### vLLM with constrained JSON decoding
+
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
+
+def call_fn(prompt: str) -> str:
+    resp = client.chat.completions.create(
+        model="meta-llama/Llama-3.3-70B-Instruct",
+        messages=[{"role": "user", "content": prompt}],
+        extra_body={"guided_json": True},
+    )
+    return resp.choices[0].message.content
+
+endpoint = EndpointConfig(
+    capability=EndpointCapability.JSON_ENFORCED,  # skip correction retry
+    call_fn=call_fn,
+)
+```
+
+---
+
+## How the capability affects pipeline behaviour
+
+| Capability | Prompt template | Retry on parse failure |
+|------------|-----------------|------------------------|
+| `TEXT_GENERATION` | `text_gen.jinja2` (verbose, with instructions) | Yes — one self-correction attempt |
+| `JSON_ENFORCED` | `json_enforced.jinja2` (compact) | No |
+| `EXTRACTION` | None — raw text passed directly | N/A |

tei_annotator/models/README.md

@@ -0,0 +1,93 @@
+# Data models
+
+Two groups of data classes power the pipeline: **schema models** (what the annotator is allowed to produce) and **span models** (the annotations as they flow through each stage).
+
+---
+
+## Schema models (`schema.py`)
+
+### `TEIAttribute`
+
+Describes a single XML attribute that may appear on a `TEIElement`.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `name` | `str` | Attribute name, e.g. `"ref"` |
+| `description` | `str` | Human-readable explanation included in LLM prompts |
+| `required` | `bool` | Whether the attribute must be present (informational only — not enforced by the pipeline) |
+| `allowed_values` | `list[str] \| None` | If set, the validator rejects any value not in this list |
+
+### `TEIElement`
+
+Describes one XML element the annotator may produce.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `tag` | `str` | XML tag name, e.g. `"persName"` |
+| `description` | `str` | Human-readable explanation included in LLM prompts |
+| `children` | `list[TEIElement]` | Allowed child elements (enables nested annotation) |
+| `attributes` | `list[TEIAttribute]` | Allowed attributes |
+
+Elements may nest: a `biblStruct` can contain `author`, which can contain `persName`. The injector uses the children hierarchy to build valid nesting trees.
+
+### `TEISchema`
+
+A flat container of `TEIElement` objects with a `get(tag) -> TEIElement | None` method for O(1) lookup by tag name.
+
+**Building a schema programmatically:**
+
+```python
+from tei_annotator import TEISchema, TEIElement, TEIAttribute
+
+schema = TEISchema(elements=[
+    TEIElement(
+        tag="persName",
+        description="a person's name",
+        attributes=[TEIAttribute(name="ref", description="authority URI")],
+    ),
+    TEIElement(tag="placeName", description="a geographical place name"),
+])
+```
+
+**Building from a RELAX NG file** (see [`tei.py`](../tei.py)):
+
+```python
+from tei_annotator import create_schema
+
+schema = create_schema("schema/tei-bib.rng", element="biblStruct", depth=1)
+```
+
+`create_schema` walks the RNG content model breadth-first to `depth` levels, collecting allowed child elements and their attribute definitions automatically.
+
+---
+
+## Span models (`spans.py`)
+
+### `SpanDescriptor`
+
+Produced by the LLM (via the parser) or by the GLiNER detector. A `SpanDescriptor` is **context-anchored**: instead of character offsets (which LLMs count unreliably), it stores the surrounding text around the entity. The resolver later searches the source text for `context` to determine where `text` lives.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `element` | `str` | TEI tag to apply, e.g. `"persName"` |
+| `text` | `str` | Verbatim text of the entity |
+| `context` | `str` | Surrounding text used to locate `text` in the source |
+| `attributes` | `dict[str, str]` | Attribute key/value pairs |
+| `score` | `float \| None` | Confidence score (populated by GLiNER; LLM-produced spans use `None`) |
+
+All `SpanDescriptor` objects are **flat** — they carry no children. Nesting is inferred geometrically by the injector once offsets are known.
+
+### `ResolvedSpan`
+
+The output of the resolver and the input to the injector. Offsets are absolute positions in the plain (tag-stripped) source text.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `element` | `str` | TEI tag |
+| `start` | `int` | Start character offset (inclusive) |
+| `end` | `int` | End character offset (exclusive) |
+| `attributes` | `dict[str, str]` | Attribute key/value pairs |
+| `children` | `list[ResolvedSpan]` | Nested child spans (populated by the injector via offset containment) |
+| `fuzzy_match` | `bool` | `True` if the context was located by fuzzy rather than exact matching |
+
+Span A is a child of span B if `B.start <= A.start` and `A.end <= B.end`. The injector constructs this nesting tree from the flat list of `ResolvedSpan` objects.

tei_annotator/postprocessing/README.md

@@ -0,0 +1,112 @@
+# Post-processing
+
+Post-processing converts the LLM's raw text output into XML tags inserted at the correct positions in the source text. It is split into four focused modules that execute in sequence:
+
+```
+LLM response (string)
+       │
+       ▼
+ 1. parse      →  list[SpanDescriptor]         (parser.py)
+       │
+       ▼
+ 2. resolve    →  list[ResolvedSpan]            (resolver.py)
+       │
+       ▼
+ 3. validate   →  list[ResolvedSpan] (filtered) (validator.py)
+       │
+       ▼
+ 4. inject     →  annotated XML string          (injector.py)
+```
+
+---
+
+## 1. Parse (`parser.py`)
+
+**Input:** raw string returned by `call_fn`
+**Output:** `list[SpanDescriptor]`
+
+Handles the messiness of real LLM output:
+
+- **Fence stripping** — models often wrap JSON in markdown code fences (`` ```json … ``` ``). These are detected and removed before parsing, regardless of the language tag used.
+- **JSON extraction** — the cleaned string is parsed with `json.loads`. Only a JSON array is accepted; any other top-level type raises `ValueError`.
+- **Dict-to-span conversion** — each dict is validated for required keys (`element`, `text`, `context`) and coerced to a `SpanDescriptor`. Dicts with missing or non-string values are silently dropped.
+- **Retry on failure** — for `TEXT_GENERATION` endpoints, if the initial parse fails the pipeline calls `make_correction_prompt()` and retries exactly once with the same `call_fn`. If the second parse also fails, the chunk is skipped with a warning.
+
+---
+
+## 2. Resolve (`resolver.py`)
+
+**Input:** `list[SpanDescriptor]`, plain source text
+**Output:** `list[ResolvedSpan]`
+
+Converts context-anchored descriptors to absolute character offsets.
+
+### Why context anchoring?
+
+LLMs cannot reliably count characters in long strings, so asking them to return numeric offsets produces frequent errors. Instead, each span is described with its surrounding text (`SpanDescriptor.context`). The resolver searches the source text for that context string to determine where the entity lives. This approach trades a small risk of positional ambiguity (two identical context strings in different positions) for a large gain in robustness. In practice, entity text is distinctive enough that collisions are rare.
+
+### Resolution algorithm
+
+For each `SpanDescriptor`:
+
+1. **Locate context** — search for `context` in the source text:
+   - First, try exact substring search (`str.find`).
+   - If not found, fall back to **fuzzy matching** via [rapidfuzz](https://github.com/rapidfuzz/RapidFuzz) with a default similarity threshold of 0.92. Fuzzy-matched spans receive `ResolvedSpan.fuzzy_match = True` and appear in `AnnotationResult.fuzzy_spans` for human review.
+   - If neither match succeeds, the span is silently discarded.
+
+2. **Locate text within context** — once the context window is pinned to a position in the source text, do a substring search for `SpanDescriptor.text` within that window.
+   - If not found, the span is discarded.
+
+3. **Compute offsets** — the context window's start offset plus the text's offset within the window gives `(start, end)` in the original source text.
+
+### Fuzzy matching details
+
+rapidfuzz's `extractOne` function scores candidate substrings using the `partial_ratio` scorer, which finds the best-matching substring of the target for the query string. The 0.92 threshold was chosen empirically to accept minor OCR errors, whitespace normalisation differences, and Unicode equivalences while rejecting clearly wrong matches.
+
+---
+
+## 3. Validate (`validator.py`)
+
+**Input:** `list[ResolvedSpan]`, `TEISchema`
+**Output:** `list[ResolvedSpan]` (filtered)
+
+Three checks are applied. Spans failing any check are silently dropped and logged at `WARNING` level:
+
+1. **Element exists** — the span's `element` must appear in the schema's element list.
+2. **Attribute names** — every key in `ResolvedSpan.attributes` must be declared in the element's `TEIAttribute` list.
+3. **Attribute values** — if `TEIAttribute.allowed_values` is set, the value must be one of those strings.
+
+Additionally, spans with invalid bounds (`start < 0`, `end > len(text)`, or `start >= end`) are rejected.
+
+Validation is a safety net against hallucinations: the LLM occasionally invents element tags or attribute names not in the schema. Dropping those silently keeps the output valid.
+
+---
+
+## 4. Inject (`injector.py`)
+
+**Input:** plain source text, `list[ResolvedSpan]`
+**Output:** annotated XML string
+
+### Building the nesting tree
+
+Spans are flat at this point (no children). The injector infers nesting geometrically: span A is a child of span B if `B.start <= A.start` and `A.end <= B.end`.
+
+`_build_nesting_tree` implements a greedy algorithm:
+
+1. Sort spans by start offset, then by length descending (so parents — which are longer — sort before their children).
+2. For each span, find its tightest enclosing parent by scanning already-placed spans.
+3. Attach it as a child of that parent, or as a root-level span if no enclosing parent exists.
+
+**Overlapping spans** — where two spans partially overlap without one containing the other — are detected and logged as warnings. The offending span (the one with the later start position) is skipped, since overlapping XML tags are not well-formed.
+
+### Recursive tag injection
+
+`_inject_recursive` walks the nesting tree depth-first. At each node it emits:
+
+1. Any source text between the previous sibling's end and this span's start.
+2. The opening tag with any attributes, e.g. `<persName ref="...">`.
+3. Recursively, the content of all child spans.
+4. Any source text after the last child and before this span's end.
+5. The closing tag, e.g. `</persName>`.
+
+Because the injector works from pre-computed offsets on the **original source text**, the text content is never altered — only tags are inserted. This preserves the exact wording, whitespace, and punctuation of the input.

tei_annotator/prompting/README.md

@@ -0,0 +1,69 @@
+# Prompt building
+
+The prompt builder (`builder.py`) constructs the full instruction string sent to the LLM for each text chunk. It uses [Jinja2](https://jinja.palletsprojects.com/) templates so that the prompt structure can be read and modified independently of the Python code.
+
+---
+
+## Templates
+
+Two templates live in `templates/`, one per `EndpointCapability`:
+
+### `text_gen.jinja2` — for `TEXT_GENERATION` endpoints
+
+Used with standard chat/completion LLMs. The template includes:
+
+1. **Role preamble** — declares the model's role as a TEI XML annotation assistant.
+2. **Schema description** — for each `TEIElement`: tag name, description, allowed attributes (with descriptions and allowed values if constrained), and allowed child elements.
+3. **Pre-detected candidates** (if any) — rendered as a JSON array so the LLM sees what GLiNER found and can decide whether to confirm, correct, or discard each candidate.
+4. **Source text** — the chunk to annotate, presented verbatim.
+5. **Output instructions** — instructs the model to return a JSON array of objects with no prose, markdown, or explanation:
+
+```json
+[
+  {
+    "element": "persName",
+    "text": "Marie Curie",
+    "context": "scientist Marie Curie was born in",
+    "attrs": {"ref": "https://viaf.org/viaf/36924049/"}
+  }
+]
+```
+
+### `json_enforced.jinja2` — for `JSON_ENFORCED` endpoints
+
+A compact variant for constrained-decoding endpoints (e.g. vLLM with guided JSON). Verbose explanations are omitted because the endpoint guarantees syntactically valid JSON output; less hand-holding is needed. The expected JSON structure is identical.
+
+---
+
+## The `context` field: why it exists
+
+LLMs cannot reliably count characters in long strings, so asking them to return numeric offsets produces errors. Instead, each span is described by its surrounding text (`context`). The resolver later anchors each span to the source text by searching for `context` as a substring (exact match first, then fuzzy). This makes the pipeline robust to small model errors in character arithmetic.
+
+The prompt instructs the model to copy a ~20-40 character window around the entity verbatim from the source text — enough to uniquely identify the span in most real-world texts.
+
+---
+
+## Self-correction / retry
+
+When a `TEXT_GENERATION` response cannot be parsed as JSON, `make_correction_prompt()` constructs a follow-up prompt that:
+
+1. Quotes the malformed response.
+2. States the parse error.
+3. Asks the model to return only a corrected JSON array, nothing else.
+
+The pipeline sends this correction prompt to the same `call_fn`. If the corrected response also fails to parse, the chunk is skipped with a warning — one retry is the limit.
+
+---
+
+## Jinja2 environment
+
+`_get_env()` initialises a `jinja2.Environment` with a custom `tojson` filter (a thin wrapper around `json.dumps`). This is necessary because `tojson` is provided automatically only in Flask applications; without it, serialising Python objects to JSON inside templates would fail.
+
+---
+
+## Prompt size considerations
+
+Each prompt is built per chunk. With `chunk_size=1500` characters the schema preamble typically adds 200–500 characters depending on the number of elements and attributes. For very large schemas (many elements, many attributes with long descriptions), consider:
+
+- Reducing `chunk_size` to keep total prompt length within model limits.
+- Splitting the schema into focused subsets and running multiple `annotate()` passes, one per element group.