from pathlib import Path
import matplotlib.pyplot as plt
-images = list(Path("assets/vllm-structured-generation/indexes").rglob("*.JPG"))
-images
-
-# display a grid of images using matplotlib (len of images)
-number_of_images = len(images)
-cols = 3
-rows = (number_of_images + cols - 1) // cols
-fig, axs = plt.subplots(rows, cols, figsize=(15, 5 * rows))
-for i, img_path in enumerate(images):
- img = plt.imread(img_path)
- ax = axs[i // cols, i % cols] if rows > 1 else axs[i % cols]
- ax.imshow(img)
- ax.axis('off')
- ax.set_title(img_path.stem)
-plt.tight_layout()
-plt.show()-
@@ -279,11 +277,10 @@ pre > code.sourceCode > span > a:first-child::before { text-decoration: underlin
- Evaluating extraction quality - different strategies for assessing accuracy
- Handling edge cases and failures -
- Surname: Family name
- Forenames: Given names @@ -296,30 +293,39 @@ pre > code.sourceCode > span > a:first-child::before { text-decoration: underlin
- Simple over complex: Only 6 core fields + 2 metadata fields -
- String types: Manuscript numbers stay as strings (can be complex like “33.1.7”) -
- Optional fields: Forenames, epithet, and notes can be missing -
- Quality tracking:
failed_to_parseflag andnotesfield for edge cases
- - Preserve original: Don’t over-normalize - keep dates/descriptions as written -
- It seems that in these examples the
notesfield isn’t really adding much value and potentially it just adds noise.
+ - While the
failed_to_parseflag sounds useful, we may want to rely on other approaches to identify failures since the model may not always set this flag correctly (and in this case we probably have some other ways to identify failures like looking for missing critical fields).
+ - Overall, we should prioritize extracting the most relevant information and avoid including fields that do not contribute to the understanding of the index card content. The simpler the schema the less for us to have to check and the fewer tokens the model has to generate. When we’re testing with small batches it doesn’t seem so important but when scaling to thousands of cards it can make a bigger difference. +
- Start with manual ground truth on a small sample (~50-100 cards) to establish baseline accuracy @@ -867,14 +894,14 @@ pre > code.sourceCode > span > a:first-child::before { text-decoration: underlin
This represents a real consulting engagement with specific requirements from library curators.
4.2 The Task: Advisor Index Cards
-The National Library of Scotland has a collection of historical index cards documenting manuscripts and correspondence. Each card follows a consistent format:
+The National Library of Scotland has a collection of historical index cards documenting manuscripts and correspondence. Each card follows a fairly consistent format:
4.2.1 Example Cards
Let’s look at a few sample cards from the collection:
-
+
+images = list(Path("../../assets/vllm-structured-generation/indexes/").rglob("*.JPG"))
+images
+
+
+
+[PosixPath('../../assets/vllm-structured-generation/indexes/DSC00172.JPG'),
+ PosixPath('../../assets/vllm-structured-generation/indexes/DSC00173.JPG'),
+ PosixPath('../../assets/vllm-structured-generation/indexes/DSC00171.JPG'),
+ PosixPath('../../assets/vllm-structured-generation/indexes/DSC00170.JPG'),
+ PosixPath('../../assets/vllm-structured-generation/indexes/DSC00169.JPG'),
+ PosixPath('../../assets/vllm-structured-generation/indexes/DSC00168.JPG')]
+
-
@@ -329,77 +335,105 @@ pre > code.sourceCode > span > a:first-child::before { text-decoration: underlin
# display a grid of images using matplotlib (len of images)
+number_of_images = len(images)
+cols = 3
+rows = (number_of_images + cols - 1) // cols
+fig, axs = plt.subplots(rows, cols, figsize=(15, 5 * rows))
+for i, img_path in enumerate(images):
+ img = plt.imread(img_path)
+ ax = axs[i // cols, i % cols] if rows > 1 else axs[i % cols]
+ ax.imshow(img)
+ ax.axis('off')
+ ax.set_title(img_path.stem)
+plt.tight_layout()
+plt.show()
4.3 Schema Design
Working with the library curators, we designed a schema that matches their cataloging requirements. The schema is intentionally simple - complex schemas are harder for VLMs to extract reliably.
-
-
-
from pydantic import BaseModel, Field
-from typing import Optional
-
-class IndexCardEntry(BaseModel):
- """Schema for index card extraction matching curator specification"""
-
- surname: str = Field(..., description="Family name as written on card")
- forenames: Optional[str] = Field(None, description="Given names")
- epithet: Optional[str] = Field(None, description="Title, occupation, or role")
- ms_no: str = Field(..., description="Manuscript number")
- description: str = Field(..., description="Document description with date")
- folios: str = Field(..., description="Folio reference")
-
- failed_to_parse: bool = Field(
- False,
- description="Set to True if the card cannot be reliably extracted (illegible, damaged, etc.)"
- )
- notes: Optional[str] = Field(
- None,
- description="Optional notes about the card: handwritten annotations, ambiguities, "
- "corrections, or reasons for failed parsing."
- )
-
-# Display the schema
-print(IndexCardEntry.model_json_schema())
-
-
{'description': 'Schema for index card extraction matching curator specification', 'properties': {'surname': {'description': 'Family name as written on card', 'title': 'Surname', 'type': 'string'}, 'forenames': {'anyOf': [{'type': 'string'}, {'type': 'null'}], 'default': None, 'description': 'Given names', 'title': 'Forenames'}, 'epithet': {'anyOf': [{'type': 'string'}, {'type': 'null'}], 'default': None, 'description': 'Title, occupation, or role', 'title': 'Epithet'}, 'ms_no': {'description': 'Manuscript number', 'title': 'Ms No', 'type': 'string'}, 'description': {'description': 'Document description with date', 'title': 'Description', 'type': 'string'}, 'folios': {'description': 'Folio reference', 'title': 'Folios', 'type': 'string'}, 'failed_to_parse': {'default': False, 'description': 'Set to True if the card cannot be reliably extracted (illegible, damaged, etc.)', 'title': 'Failed To Parse', 'type': 'boolean'}, 'notes': {'anyOf': [{'type': 'string'}, {'type': 'null'}], 'default': None, 'description': 'Optional notes about the card: handwritten annotations, ambiguities, corrections, or reasons for failed parsing.', 'title': 'Notes'}}, 'required': ['surname', 'ms_no', 'description', 'folios'], 'title': 'IndexCardEntry', 'type': 'object'}This schema is something we can iterate on later based on extraction quality but gives us a solid starting point.
+
+
+
+from pydantic import BaseModel, Field
+from typing import Optional
+
+class IndexCardEntry(BaseModel):
+ """Schema for index card extraction matching curator specification"""
+
+ surname: str = Field(..., description="Family name as written on card")
+ forenames: Optional[str] = Field(None, description="Given names")
+ epithet: Optional[str] = Field(None, description="Title, occupation, or role")
+ ms_no: str = Field(..., description="Manuscript number")
+ description: str = Field(..., description="Document description with date")
+ folios: str = Field(..., description="Folio reference")
+
+ failed_to_parse: bool = Field(
+ False,
+ description="Set to True if the card cannot be reliably extracted (illegible, damaged, etc.)"
+ )
+ notes: Optional[str] = Field(
+ None,
+ description="Optional notes about the card: handwritten annotations, ambiguities, "
+ "corrections, or reasons for failed parsing."
+ )
+Let’s take a look at the schema definition we’ll use for extraction:
+
+
+
-# Display the schema
+from rich import print
+print(IndexCardEntry.model_json_schema())
+
{ + 'description': 'Schema for index card extraction matching curator specification', + 'properties': { + 'surname': {'description': 'Family name as written on card', 'title': 'Surname', 'type': 'string'}, + 'forenames': { + 'anyOf': [{'type': 'string'}, {'type': 'null'}], + 'default': None, + 'description': 'Given names', + 'title': 'Forenames' + }, + 'epithet': { + 'anyOf': [{'type': 'string'}, {'type': 'null'}], + 'default': None, + 'description': 'Title, occupation, or role', + 'title': 'Epithet' + }, + 'ms_no': {'description': 'Manuscript number', 'title': 'Ms No', 'type': 'string'}, + 'description': { + 'description': 'Document description with date', + 'title': 'Description', + 'type': 'string' + }, + 'folios': {'description': 'Folio reference', 'title': 'Folios', 'type': 'string'}, + 'failed_to_parse': { + 'default': False, + 'description': 'Set to True if the card cannot be reliably extracted (illegible, damaged, etc.)', + 'title': 'Failed To Parse', + 'type': 'boolean' + }, + 'notes': { + 'anyOf': [{'type': 'string'}, {'type': 'null'}], + 'default': None, + 'description': 'Optional notes about the card: handwritten annotations, ambiguities, corrections, or +reasons for failed parsing.', + 'title': 'Notes' + } + }, + 'required': ['surname', 'ms_no', 'description', 'folios'], + 'title': 'IndexCardEntry', + 'type': 'object' +} +
4.3.1 Design Decisions
-Key choices in this schema:
--
-
4.4 Setup
We’ll reuse the VLM setup from the previous chapter. If you haven’t already, make sure LM Studio is running with a VLM loaded.
-
-
-
-from openai import OpenAI
-import base64
-from io import BytesIO
-from PIL import Image as PILImage
-
-
-
-client = OpenAI(
- base_url="http://localhost:1234/v1",
- api_key="lm-studio"
-)
-
-
client.models.list()
-
+SyncPage[Model](data=[Model(id='qwen/qwen3-vl-4b', created=None, object='model', owned_by='organization_owner'), Model(id='qwen3-vl-2b-instruct-mlx', created=None, object='model', owned_by='organization_owner'), Model(id='qwen/qwen3-vl-8b', created=None, object='model', owned_by='organization_owner'), Model(id='text-embedding-nomic-embed-text-v1.5', created=None, object='model', owned_by='organization_owner'), Model(id='qwen3-vl-30b-a3b-instruct', created=None, object='model', owned_by='organization_owner'), Model(id='qwen3-vl-30b-a3b-thinking@4bit', created=None, object='model', owned_by='organization_owner'), Model(id='qwen3-vl-30b-a3b-thinking@3bit', created=None, object='model', owned_by='organization_owner'), Model(id='qwen/qwen3-4b-thinking-2507', created=None, object='model', owned_by='organization_owner'), Model(id='google/gemma-3-12b', created=None, object='model', owned_by='organization_owner'), Model(id='google/gemma-3-4b', created=None, object='model', owned_by='organization_owner'), Model(id='qwen2-0.5b-instruct-fingreylit', created=None, object='model', owned_by='organization_owner'), Model(id='google/gemma-3n-e4b', created=None, object='model', owned_by='organization_owner'), Model(id='granite-vision-3.3-2b', created=None, object='model', owned_by='organization_owner'), Model(id='ibm/granite-4-h-tiny', created=None, object='model', owned_by='organization_owner'), Model(id='iconclass-vlm', created=None, object='model', owned_by='organization_owner'), Model(id='mlx-community/qwen2.5-vl-3b-instruct', created=None, object='model', owned_by='organization_owner'), Model(id='lmstudio-community/qwen2.5-vl-3b-instruct', created=None, object='model', owned_by='organization_owner'), Model(id='lfm2-vl-1.6b', created=None, object='model', owned_by='organization_owner'), Model(id='mimo-vl-7b-rl-2508@q4_k_s', created=None, object='model', owned_by='organization_owner'), Model(id='mimo-vl-7b-rl-2508@q8_0', created=None, object='model', owned_by='organization_owner'), Model(id='qwen3-30b-a3b-instruct-2507', created=None, object='model', owned_by='organization_owner'), Model(id='qwen3-4b-instruct-2507-mlx', created=None, object='model', owned_by='organization_owner'), Model(id='openai/gpt-oss-20b', created=None, object='model', owned_by='organization_owner'), Model(id='qwen/qwen2.5-vl-7b', created=None, object='model', owned_by='organization_owner'), Model(id='mistralai/mistral-small-3.2', created=None, object='model', owned_by='organization_owner'), Model(id='qwen3-30b-a3b-instruct-2507-mlx', created=None, object='model', owned_by='organization_owner'), Model(id='liquid/lfm2-1.2b', created=None, object='model', owned_by='organization_owner'), Model(id='smollm3-3b-mlx', created=None, object='model', owned_by='organization_owner'), Model(id='unsloth/smollm3-3b', created=None, object='model', owned_by='organization_owner'), Model(id='ggml-org/smollm3-3b', created=None, object='model', owned_by='organization_owner'), Model(id='mlx-community/smollm3-3b', created=None, object='model', owned_by='organization_owner')], object='list')
+
-from openai import OpenAI
+import base64
+from io import BytesIO
+from PIL import Image as PILImage
+
+
+client = OpenAI(
+ base_url="http://localhost:1234/v1",
+ api_key="lm-studio"
+)
-
-
-
pydantic._internal._model_construction.ModelMetaclass
+
+
-client.models.list()
+
SyncPage[Model](data=[Model(id='qwen3-vl-2b-instruct-mlx', created=None, object='model', owned_by='organization_owner'), Model(id='qwen/qwen3-vl-8b', created=None, object='model', owned_by='organization_owner'), Model(id='qwen/qwen3-vl-4b', created=None, object='model', owned_by='organization_owner'), Model(id='text-embedding-nomic-embed-text-v1.5', created=None, object='model', owned_by='organization_owner'), Model(id='qwen3-vl-30b-a3b-instruct', created=None, object='model', owned_by='organization_owner'), Model(id='qwen3-vl-30b-a3b-thinking@4bit', created=None, object='model', owned_by='organization_owner'), Model(id='qwen3-vl-30b-a3b-thinking@3bit', created=None, object='model', owned_by='organization_owner'), Model(id='qwen/qwen3-4b-thinking-2507', created=None, object='model', owned_by='organization_owner'), Model(id='google/gemma-3-12b', created=None, object='model', owned_by='organization_owner'), Model(id='google/gemma-3-4b', created=None, object='model', owned_by='organization_owner'), Model(id='qwen2-0.5b-instruct-fingreylit', created=None, object='model', owned_by='organization_owner'), Model(id='google/gemma-3n-e4b', created=None, object='model', owned_by='organization_owner'), Model(id='granite-vision-3.3-2b', created=None, object='model', owned_by='organization_owner'), Model(id='ibm/granite-4-h-tiny', created=None, object='model', owned_by='organization_owner'), Model(id='iconclass-vlm', created=None, object='model', owned_by='organization_owner'), Model(id='mlx-community/qwen2.5-vl-3b-instruct', created=None, object='model', owned_by='organization_owner'), Model(id='lmstudio-community/qwen2.5-vl-3b-instruct', created=None, object='model', owned_by='organization_owner'), Model(id='lfm2-vl-1.6b', created=None, object='model', owned_by='organization_owner'), Model(id='mimo-vl-7b-rl-2508@q4_k_s', created=None, object='model', owned_by='organization_owner'), Model(id='mimo-vl-7b-rl-2508@q8_0', created=None, object='model', owned_by='organization_owner'), Model(id='qwen3-30b-a3b-instruct-2507', created=None, object='model', owned_by='organization_owner'), Model(id='qwen3-4b-instruct-2507-mlx', created=None, object='model', owned_by='organization_owner'), Model(id='openai/gpt-oss-20b', created=None, object='model', owned_by='organization_owner'), Model(id='qwen/qwen2.5-vl-7b', created=None, object='model', owned_by='organization_owner'), Model(id='mistralai/mistral-small-3.2', created=None, object='model', owned_by='organization_owner'), Model(id='qwen3-30b-a3b-instruct-2507-mlx', created=None, object='model', owned_by='organization_owner'), Model(id='liquid/lfm2-1.2b', created=None, object='model', owned_by='organization_owner'), Model(id='smollm3-3b-mlx', created=None, object='model', owned_by='organization_owner'), Model(id='unsloth/smollm3-3b', created=None, object='model', owned_by='organization_owner'), Model(id='ggml-org/smollm3-3b', created=None, object='model', owned_by='organization_owner'), Model(id='mlx-community/smollm3-3b', created=None, object='model', owned_by='organization_owner')], object='list')
+
from typing import Union
def query_image_structured(image: Union[PILImage.Image, str], prompt: str, schema: BaseModel, model='qwen/qwen3-vl-4b'):
"""
@@ -445,7 +479,7 @@ pre > code.sourceCode > span > a:first-child::before { text-decoration: underlin
4.5 Extraction Examples
Let’s run extraction on several sample cards to see how the model performs.
-
+
prompt = """Extract structured information from this historical library index card and return it as JSON.
This is an index card from the National Library of Scotland's Advocate's Library collection. Each card documents a person and associated manuscript references.
@@ -470,23 +504,23 @@ pre > code.sourceCode > span > a:first-child::before { text-decoration: underlin
- Only set "failed_to_parse" to true if you genuinely cannot extract the required fields
- Use null for optional fields (forenames, epithet, notes) if they are not present or marked with a line"""
-
+
image = PILImage.open(images[0])
image
-
+
-
+
-
+
from rich import print
result = query_image_structured(image, prompt, IndexCardEntry, model='qwen/qwen3-vl-4b')
-
+
print(result)
IndexCardEntry(
@@ -497,12 +531,9 @@ pre > code.sourceCode > span > a:first-child::before { text-decoration: underlin
description='letter of (1878)',
folios='f. 38',
failed_to_parse=False,
- notes="Handwritten correction: 'Cairo' appears to be a miswrite of 'ABBAATE (Itala)' and 'Cairo' is likely a
-misread of 'ABBATE (Itala)'. Also, 'Physician' is underlined and possibly misread as 'Physician' in the original
-text. The handwritten note 'Auchio' appears to be a scribble or correction, possibly 'Auchio' is a misread of
-'Physician' or a separate annotation. The text 'Cairo' is likely a misread of 'ABBATE (Itala)' and 'Cairo' is not a
-valid surname in this context. The text 'Daughter of the Physician' is underlined, suggesting it may be a
-correction or a clarification."
+ notes="Handwritten corrections and annotations present: 'Cairo' (instead of 'ABBAATE'), 'Cairo' (instead of
+'ABBAATE'), 'Physician' (instead of 'Physician'), '2633' (instead of '2633'), 'f. 38' (instead of 'f. 38'). Also,
+'Cairo' appears to be a scribbled correction or miswriting of 'ABBAATE'."
)
@@ -510,7 +541,7 @@ pre > code.sourceCode > span > a:first-child::before { text-decoration: underlin
4.5.1 Comparing Extraction to Ground Truth
Let’s compare a few extractions to the actual card content:
-
+
from tqdm.auto import tqdm
results = []
@@ -520,7 +551,7 @@ pre > code.sourceCode > span > a:first-child::before { text-decoration: underlin
results.append((img_path.stem, result))
@@ -529,105 +560,107 @@ pre > code.sourceCode > span > a:first-child::before { text-decoration: underlin
4.6 Evaluation Strategies
How do we know if the extraction is working well? There are several approaches to evaluation, each with different tradeoffs.
-Checking against the images
-
-# Display images with extracted data side-by-side
-# Two columns: left = image, right = extracted text
-
-for i, (img_stem, result) in enumerate(results):
- fig, (ax_img, ax_text) = plt.subplots(1, 2, figsize=(16, 6),
- gridspec_kw={'width_ratios': [1, 1]})
-
- # Left: Display image
- img = plt.imread(images[i])
- ax_img.imshow(img)
- ax_img.axis('off')
- ax_img.set_title(f"Card {i+1}: {img_stem}", fontsize=14, fontweight='bold')
-
- # Right: Display extracted data as formatted text
- ax_text.axis('off')
-
- # Format the extracted data nicely
- text_lines = [
- "Extracted Data:",
- "",
- f"Surname: {result.surname}",
- f"Forenames: {result.forenames or 'N/A'}",
- f"Epithet: {result.epithet or 'N/A'}",
- f"MS No: {result.ms_no}",
- f"Description: {result.description}",
- f"Folios: {result.folios}",
- "",
- f"Failed to Parse: {result.failed_to_parse}",
- ]
-
- # Add notes if present
- if result.notes:
- text_lines.append("")
- text_lines.append("Notes:")
- # Wrap long notes
- import textwrap
- wrapped_notes = textwrap.fill(result.notes, width=60)
- text_lines.append(wrapped_notes)
-
- # Join and display
- formatted_text = "\n".join(text_lines)
- ax_text.text(0.05, 0.95, formatted_text,
- transform=ax_text.transAxes,
- fontsize=11,
- verticalalignment='top',
- fontfamily='monospace',
- bbox=dict(boxstyle='round', facecolor='wheat', alpha=0.3))
-
- plt.tight_layout()
- plt.show()
+
+4.6.1 Looking at lots of samples
+It sounds simple, but looking at a large number of random samples can give a good sense of overall quality. You can spot common errors and get a feel for how reliable the extraction is. You can quickly build intuition about what might be going wrong and where to focus improvement efforts. Realistically you will usually spend some time iterating on the prompt and schema at this stage. Looking at more than one example is important to avoid overfitting to a single case but you don’t immediately need to look at hundreds of examples or set up complex metrics or evaluations. This can come later.
+
+for i, (img_stem, result) in enumerate(results):
+ fig, (ax_img, ax_text) = plt.subplots(1, 2, figsize=(16, 6),
+ gridspec_kw={'width_ratios': [1, 1]})
+
+ # Left: Display image
+ img = plt.imread(images[i])
+ ax_img.imshow(img)
+ ax_img.axis('off')
+ ax_img.set_title(f"Card {i+1}: {img_stem}", fontsize=14, fontweight='bold')
+
+ # Right: Display extracted data as formatted text
+ ax_text.axis('off')
+
+ # Format the extracted data nicely
+ text_lines = [
+ "Extracted Data:",
+ "",
+ f"Surname: {result.surname}",
+ f"Forenames: {result.forenames or 'N/A'}",
+ f"Epithet: {result.epithet or 'N/A'}",
+ f"MS No: {result.ms_no}",
+ f"Description: {result.description}",
+ f"Folios: {result.folios}",
+ "",
+ f"Failed to Parse: {result.failed_to_parse}",
+ ]
+
+ # Add notes if present
+ if result.notes:
+ text_lines.extend(("", "Notes:"))
+ # Wrap long notes
+ import textwrap
+ wrapped_notes = textwrap.fill(result.notes, width=60)
+ text_lines.append(wrapped_notes)
+
+ # Join and display
+ formatted_text = "\n".join(text_lines)
+ ax_text.text(0.05, 0.95, formatted_text,
+ transform=ax_text.transAxes,
+ fontsize=11,
+ verticalalignment='top',
+ fontfamily='monospace',
+ bbox=dict(boxstyle='round', facecolor='wheat', alpha=0.3))
+
+ plt.tight_layout()
+ plt.show()
-
+
-
+
-
+
-
+
-
+
-
+
-notes
-Notes don’t seem that useful and waste tokens. Let’s remove.
-We should also test that the “failed to parse” flag works correctly. let’s try an image that is blank or has no text.
-
+
+4.6.1.1 What we learned from these samples
+
+
+
from pydantic import BaseModel, Field
from typing import Optional
@@ -655,27 +688,26 @@ pre > code.sourceCode > span > a:first-child::before { text-decoration: underlin
"ms_no": "Manuscript number exactly as written (e.g., '5538', '5529')",
"description": "Document description with date (e.g., 'letter of (1783)', 'copy of petition of (ca. 1783)')",
"folios": "Folio reference exactly as written (e.g., 'f.11', 'f.169')",
- "notes": "Optional notes about handwritten corrections, ambiguities, or parsing issues"
- }
-
- Guidelines:
- - Extract text exactly as it appears - do not correct spelling or expand abbreviations
- - Preserve original punctuation and formatting
- - Use null for optional fields (forenames, epithet, notes) if they are not present or marked with a line"""
+ }
+
+ Guidelines:
+ - Extract text exactly as it appears - do not correct spelling or expand abbreviations
+ - Preserve original punctuation and formatting
+ - Use null for optional fields (forenames, epithet, notes) if they are not present or marked with a line"""
-
+
results = []
for img_path in tqdm(images):
image = PILImage.open(img_path)
- result = query_image_structured(image, prompt, IndexCardEntry, model='qwen/qwen3-vl-4b')
+ result = query_image_structured(image, prompt, IndexCardEntry, model='qwen/qwen3-vl-8b')
results.append((img_path.stem, result))
-
+
# Display images with extracted data side-by-side
# Two columns: left = image, right = extracted text
@@ -703,75 +735,70 @@ pre > code.sourceCode > span > a:first-child::before { text-decoration: underlin
f"Description: {result.description}",
f"Folios: {result.folios}",
"",
- f"Failed to Parse: {result.failed_to_parse}",
- ]
-
- # Add notes if present
- if result.notes:
- text_lines.append("")
- text_lines.append("Notes:")
- # Wrap long notes
- import textwrap
- wrapped_notes = textwrap.fill(result.notes, width=60)
- text_lines.append(wrapped_notes)
-
- # Join and display
- formatted_text = "\n".join(text_lines)
- ax_text.text(0.05, 0.95, formatted_text,
- transform=ax_text.transAxes,
- fontsize=11,
- verticalalignment='top',
- fontfamily='monospace',
- bbox=dict(boxstyle='round', facecolor='wheat', alpha=0.3))
-
- plt.tight_layout()
- plt.show()
-
-
----------------------------------------------------------------------------
-AttributeError Traceback (most recent call last)
-Cell In[69], line 28
- 15 ax_text.axis('off')
- 17 # Format the extracted data nicely
- 18 text_lines = [
- 19 "Extracted Data:",
- 20 "",
- 21 f"Surname: {result.surname}",
- 22 f"Forenames: {result.forenames or 'N/A'}",
- 23 f"Epithet: {result.epithet or 'N/A'}",
- 24 f"MS No: {result.ms_no}",
- 25 f"Description: {result.description}",
- 26 f"Folios: {result.folios}",
- 27 "",
----> 28 f"Failed to Parse: {result.failed_to_parse}",
- 29 ]
- 31 # Add notes if present
- 32 if result.notes:
-
-File ~/Documents/nls-work/ai-patterns-for-glam/.venv/lib/python3.13/site-packages/pydantic/main.py:991, in BaseModel.__getattr__(self, item)
- 988 return super().__getattribute__(item) # Raises AttributeError if appropriate
- 989 else:
- 990 # this is the current error
---> 991 raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}')
-
-AttributeError: 'IndexCardEntry' object has no attribute 'failed_to_parse'
+ ]
+ # Join and display
+ formatted_text = "\n".join(text_lines)
+ ax_text.text(0.05, 0.95, formatted_text,
+ transform=ax_text.transAxes,
+ fontsize=11,
+ verticalalignment='top',
+ fontfamily='monospace',
+ bbox=dict(boxstyle='round', facecolor='wheat', alpha=0.3))
+
+ plt.tight_layout()
+ plt.show()
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
-
+
-
-4.6.1 1. Manual Ground Truth Evaluation
+
+
+
+4.6.2 1. Manual Ground Truth Evaluation
The Gold Standard: Manually annotate a sample of cards and compare.
Pros: - Most accurate measure of performance - Catches all types of errors - Builds training data for future improvements
Cons: - Time consuming - Requires expert annotators - Limited sample size
Best for: Final validation, establishing baselines, understanding failure modes
-
+
# TODO: Load manually annotated ground truth
# Compare predictions to ground truth
# Calculate field-level accuracy
@@ -782,13 +809,13 @@ pre > code.sourceCode > span > a:first-child::before { text-decoration: underlin
# - Common error patterns
-
-4.6.2 2. Cross-Model Evaluation (Model-as-Judge)
+
+4.6.3 2. Cross-Model Evaluation (Model-as-Judge)
The Pragmatic Approach: Use a stronger/different model to evaluate outputs.
Pros: - Much faster than manual annotation - Can evaluate full dataset - Good for catching obvious errors
Cons: - Requires access to multiple models - May miss subtle errors - Judge model can be wrong too
Best for: Large-scale quality monitoring, automated testing, identifying problem areas for manual review
-
+
# TODO: Implement model-as-judge evaluation
# - Extract with Model A (e.g., local Qwen)
# - Show image + extraction to Model B (e.g., Claude/GPT-4)
@@ -808,14 +835,14 @@ pre > code.sourceCode > span > a:first-child::before { text-decoration: underlin
# """
-
-4.6.3 3. Internal Consistency Checks
+
+4.6.4 3. Internal Consistency Checks
The Automated Approach: Use business rules and patterns to identify suspicious outputs.
Examples: - Manuscript numbers should follow known patterns - Dates should be within expected ranges - Folio references have consistent formats - Certain fields should always be present
Pros: - Completely automated - Fast - can run on full dataset - No additional model costs
Cons: - Only catches specific error types - Requires domain knowledge to design rules - Can miss errors that follow valid patterns
Best for: Flagging outliers for review, automated quality gates, monitoring production systems
-
+
# TODO: Implement consistency checks
# def validate_extraction(entry: IndexCardEntry) -> list[str]:
@@ -839,20 +866,20 @@ pre > code.sourceCode > span > a:first-child::before { text-decoration: underlin
# return warnings
-
-4.6.4 4. Confidence Scoring
+
+4.6.5 4. Confidence Scoring
Many VLM APIs return confidence scores or logprobs. We can use these to identify uncertain extractions.
Pros: - No additional cost or models needed - Can prioritize review efforts - Helps establish quality thresholds
Cons: - Not all models/APIs provide confidence scores - High confidence doesn’t guarantee correctness - Requires calibration
Best for: Prioritizing manual review, quality-based routing, understanding model uncertainty
-
+
# TODO: If available, extract and analyze confidence scores
# Plot distribution of confidence scores
# Correlate confidence with manual evaluation results
-
-4.6.5 Combining Evaluation Approaches
+
+4.6.6 Combining Evaluation Approaches
In practice, a robust evaluation strategy uses multiple approaches:
4.7 Batch Processing
Now let’s process a larger batch of cards and analyze the results.
-
+
# TODO: Process all available cards
# Track timing, failures, warnings
# Save results to file
4.7.1 Results Analysis
-
+
# TODO: Analyze batch results
# - Success rate
# - Failed to parse rate
@@ -886,7 +913,7 @@ pre > code.sourceCode > span > a:first-child::before { text-decoration: underlin
4.8 Edge Cases and Failure Modes
What kinds of cards are hard for the model to process?
-
+
# TODO: Examine failed/problematic extractions
# Common patterns:
# - Handwritten corrections/additions
@@ -898,7 +925,7 @@ pre > code.sourceCode > span > a:first-child::before { text-decoration: underlin
4.9 Export for Cataloging
Convert the extracted data to formats suitable for library systems.
-
+
# TODO: Export to CSV/JSON/XML
# Consider catalog system requirements (MARC, Dublin Core, etc.)