init

.agents/skills/gradio/SKILL.md

@@ -0,0 +1,84 @@
+---
+name: gradio
+description: Build Gradio web UIs and demos in Python. Use when creating, modifying, debugging, or answering questions about Gradio and its capabilties, components, event listeners, or layouts.
+---
+
+# Gradio
+
+Gradio is a Python library for building interactive web UIs and ML demos. This skill covers the core API, patterns, and examples.
+
+## References
+- `references/examples.md` -  Illustrative examples showcasing the core Gradio API.
+- `references/api-signatures.md` - API signatures of commonly used components.
+- `references/event-listeners.md` - API signatures of events supported by each component.
+
+## Guides
+
+Detailed guides on specific topics (read these when relevant):
+
+- [Quickstart](https://www.gradio.app/guides/quickstart)
+- [The Interface Class](https://www.gradio.app/guides/the-interface-class)
+- [Blocks and Event Listeners](https://www.gradio.app/guides/blocks-and-event-listeners)
+- [Controlling Layout](https://www.gradio.app/guides/controlling-layout)
+- [More Blocks Features](https://www.gradio.app/guides/more-blocks-features)
+- [Custom CSS and JS](https://www.gradio.app/guides/custom-CSS-and-JS)
+- [Streaming Outputs](https://www.gradio.app/guides/streaming-outputs)
+- [Streaming Inputs](https://www.gradio.app/guides/streaming-inputs)
+- [Sharing Your App](https://www.gradio.app/guides/sharing-your-app)
+- [Custom HTML Components](https://www.gradio.app/guides/custom-HTML-components)
+- [Getting Started with the Python Client](https://www.gradio.app/guides/getting-started-with-the-python-client)
+- [Getting Started with the JS Client](https://www.gradio.app/guides/getting-started-with-the-js-client)
+
+## Core Patterns
+
+**Interface** (high-level): wraps a function with input/output components.
+
+```python
+import gradio as gr
+
+def greet(name):
+    return f"Hello {name}!"
+
+gr.Interface(fn=greet, inputs="text", outputs="text").launch()
+```
+
+**Blocks** (low-level): flexible layout with explicit event wiring.
+
+```python
+import gradio as gr
+
+with gr.Blocks() as demo:
+    name = gr.Textbox(label="Name")
+    output = gr.Textbox(label="Greeting")
+    btn = gr.Button("Greet")
+    btn.click(fn=lambda n: f"Hello {n}!", inputs=name, outputs=output)
+
+demo.launch()
+```
+
+**ChatInterface**: high-level wrapper for chatbot UIs.
+
+```python
+import gradio as gr
+
+def respond(message, history):
+    return f"You said: {message}"
+
+gr.ChatInterface(fn=respond).launch()
+```
+
+## Custom HTML Components
+
+If a task requires significant customization of an existing component or a component that doesn't exist in Gradio, you can create one with `gr.HTML`. It supports `html_template` (with `${}` JS expressions and `{{}}` Handlebars syntax), `css_template` for scoped styles, and `js_on_load` for interactivity — where `props.value` updates the component value and `trigger('event_name')` fires Gradio events. For reuse, subclass `gr.HTML` and define `api_info()` for API/MCP support. 
+
+See the [full guide](https://www.gradio.app/guides/custom-HTML-components) as well as example in `references/examples.md`
+
+## Server Mode
+
+Use `gr.Server` instead of gr.Blocks when the users requests any of the following:
+- Completely custom UI (your own HTML, React, Svelte, etc.) powered by Gradio's backend.
+- Full control of FastAPI server (custom GET/POST routes, middleware, dependency injection) alongside Gradio API endpoints
+
+See the [full guide](https://www.gradio.app/guides/server-mode) and example in `references/examples.md`.
+
+If the user's use case can be handled by Gradio's built-in components or customizable HTML components, prefer not to use `gr.Server`.

.agents/skills/gradio/references/api-signatures.md

@@ -0,0 +1,115 @@
+# Component API Signatures
+
+Quick reference for common Gradio component constructors.
+
+## `Textbox`
+
+```python
+Textbox(value: str | I18nData | Callable | None = None, type: Literal['text', 'password', 'email'] = "text", lines: int = 1, max_lines: int | None = None, placeholder: str | I18nData | None = None, label: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, autofocus: bool = False, autoscroll: bool = True, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", text_align: Literal['left', 'right'] | None = None, rtl: bool = False, buttons: list[Literal['copy'] | Button] | None = None, max_length: int | None = None, submit_btn: str | bool | None = False, stop_btn: str | bool | None = False, html_attributes: InputHTMLAttributes | None = None)
+```
+
+Creates a textarea for user to enter string input or display string output.
+
+## `Number`
+
+```python
+Number(value: float | Callable | None = None, label: str | I18nData | None = None, placeholder: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", buttons: list[Button] | None = None, precision: int | None = None, minimum: float | None = None, maximum: float | None = None, step: float = 1)
+```
+
+Creates a numeric field for user to enter numbers as input or display numeric output.
+
+## `Slider`
+
+```python
+Slider(minimum: float = 0, maximum: float = 100, value: float | Callable | None = None, step: float | None = None, precision: int | None = None, label: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", randomize: bool = False, buttons: list[Literal['reset']] | None = None)
+```
+
+Creates a slider that ranges from {minimum} to {maximum} with a step size of {step}.
+
+## `Checkbox`
+
+```python
+Checkbox(value: bool | Callable = False, label: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", buttons: list[Button] | None = None)
+```
+
+Creates a checkbox that can be set to `True` or `False`. Can be used as an input to pass a boolean value to a function or as an output to display a boolean value.
+
+## `Dropdown`
+
+```python
+Dropdown(choices: Sequence[str | int | float | tuple[str | I18nData, str | int | float]] | None = None, value: str | int | float | Sequence[str | int | float] | Callable | DefaultValue | None = DefaultValue(), type: Literal['value', 'index'] = "value", multiselect: bool | None = None, allow_custom_value: bool = False, max_choices: int | None = None, filterable: bool = True, label: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", buttons: list[Button] | None = None)
+```
+
+Creates a dropdown of choices from which a single entry or multiple entries can be selected (as an input component) or displayed (as an output component).
+
+## `Radio`
+
+```python
+Radio(choices: Sequence[str | int | float | tuple[str | I18nData, str | int | float]] | None = None, value: str | int | float | Callable | None = None, type: Literal['value', 'index'] = "value", label: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", rtl: bool = False, buttons: list[Button] | None = None)
+```
+
+Creates a set of (string or numeric type) radio buttons of which only one can be selected.
+
+## `Image`
+
+```python
+Image(value: str | PIL.Image.Image | np.ndarray | Callable | None = None, format: str = "webp", height: int | str | None = None, width: int | str | None = None, image_mode: Literal['1', 'L', 'P', 'RGB', 'RGBA', 'CMYK', 'YCbCr', 'LAB', 'HSV', 'I', 'F'] | None = "RGB", sources: list[Literal['upload', 'webcam', 'clipboard']] | Literal['upload', 'webcam', 'clipboard'] | None = None, type: Literal['numpy', 'pil', 'filepath'] = "numpy", label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, buttons: list[Literal['download', 'share', 'fullscreen'] | Button] | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, streaming: bool = False, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", webcam_options: WebcamOptions | None = None, placeholder: str | None = None, watermark: WatermarkOptions | None = None)
+```
+
+Creates an image component that can be used to upload images (as an input) or display images (as an output).
+
+## `Audio`
+
+```python
+Audio(value: str | Path | tuple[int, np.ndarray] | Callable | None = None, sources: list[Literal['upload', 'microphone']] | Literal['upload', 'microphone'] | None = None, type: Literal['numpy', 'filepath'] = "numpy", label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, streaming: bool = False, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", format: Literal['wav', 'mp3'] | None = None, autoplay: bool = False, editable: bool = True, buttons: list[Literal['download', 'share'] | Button] | None = None, waveform_options: WaveformOptions | dict | None = None, loop: bool = False, recording: bool = False, subtitles: str | Path | list[dict[str, Any]] | None = None, playback_position: float = 0)
+```
+
+Creates an audio component that can be used to upload/record audio (as an input) or display audio (as an output).
+
+## `Video`
+
+```python
+Video(value: str | Path | Callable | None = None, format: str | None = None, sources: list[Literal['upload', 'webcam']] | Literal['upload', 'webcam'] | None = None, height: int | str | None = None, width: int | str | None = None, label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", webcam_options: WebcamOptions | None = None, include_audio: bool | None = None, autoplay: bool = False, buttons: list[Literal['download', 'share'] | Button] | None = None, loop: bool = False, streaming: bool = False, watermark: WatermarkOptions | None = None, subtitles: str | Path | list[dict[str, Any]] | None = None, playback_position: float = 0)
+```
+
+Creates a video component that can be used to upload/record videos (as an input) or display videos (as an output). For the video to be playable in the browser it must have a compatible container and codec combination. Allowed combinations are .mp4 with h264 codec, .ogg with theora codec, and .webm with vp9 codec. If the component detects that the output video would not be playable in the browser it will attempt to convert it to a playable mp4 video. If the conversion fails, the original video is returned.
+
+## `File`
+
+```python
+File(value: str | list[str] | Callable | None = None, file_count: Literal['single', 'multiple', 'directory'] = "single", file_types: list[str] | None = None, type: Literal['filepath', 'binary'] = "filepath", label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, height: int | str | float | None = None, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", allow_reordering: bool = False, buttons: list[Button] | None = None)
+```
+
+Creates a file component that allows uploading one or more generic files (when used as an input) or displaying generic files or URLs for download (as output).       Demo: zip_files, zip_to_json
+
+## `Chatbot`
+
+```python
+Chatbot(value: list[MessageDict | Message] | Callable | None = None, label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, autoscroll: bool = True, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", height: int | str | None = 400, resizable: bool = False, max_height: int | str | None = None, min_height: int | str | None = None, editable: Literal['user', 'all'] | None = None, latex_delimiters: list[dict[str, str | bool]] | None = None, rtl: bool = False, buttons: list[Literal['share', 'copy', 'copy_all'] | Button] | None = None, watermark: str | None = None, avatar_images: tuple[str | Path | None, str | Path | None] | None = None, sanitize_html: bool = True, render_markdown: bool = True, feedback_options: list[str] | tuple[str, ...] | None = ('Like', 'Dislike'), feedback_value: Sequence[str | None] | None = None, line_breaks: bool = True, layout: Literal['panel', 'bubble'] | None = None, placeholder: str | None = None, examples: list[ExampleMessage] | None = None, allow_file_downloads: bool = True, group_consecutive_messages: bool = True, allow_tags: list[str] | bool = True, reasoning_tags: list[tuple[str, str]] | None = None, like_user_message: bool = False)
+```
+
+Creates a chatbot that displays user-submitted messages and responses. Supports a subset of Markdown including bold, italics, code, tables. Also supports audio/video/image files, which are displayed in the Chatbot, and other kinds of files which are displayed as links. This component is usually used as an output component.
+
+## `Button`
+
+```python
+Button(value: str | I18nData | Callable = "Run", every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, variant: Literal['primary', 'secondary', 'stop', 'huggingface'] = "secondary", size: Literal['sm', 'md', 'lg'] = "lg", icon: str | Path | None = None, link: str | None = None, link_target: Literal['_self', '_blank', '_parent', '_top'] = "_self", visible: bool | Literal['hidden'] = True, interactive: bool = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", scale: int | None = None, min_width: int | None = None)
+```
+
+Creates a button that can be assigned arbitrary .click() events. The value (label) of the button can be used as an input to the function (rarely used) or set via the output of a function.
+
+## `Markdown`
+
+```python
+Markdown(value: str | I18nData | Callable | None = None, label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, scale: int | None = None, min_width: int | None = None, rtl: bool = False, latex_delimiters: list[dict[str, str | bool]] | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", sanitize_html: bool = True, line_breaks: bool = False, header_links: bool = False, height: int | str | None = None, max_height: int | str | None = None, min_height: int | str | None = None, buttons: list[Literal['copy']] | None = None, container: bool = False, padding: bool = False)
+```
+
+Used to render arbitrary Markdown output. Can also render latex enclosed by dollar signs as well as code blocks with syntax highlighting. Supported languages are bash, c, cpp, go, java, javascript, json, php, python, rust, sql, and yaml. As this component does not accept user input, it is rarely used as an input component.
+
+## `HTML`
+
+```python
+HTML(value: Any | Callable | None = None, label: str | I18nData | None = None, html_template: str = "${value}", css_template: str = "", js_on_load: str | None = "element.addEventListener('click', function() { trigger('click') });", apply_default_css: bool = True, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool = False, scale: int | None = None, min_width: int | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", min_height: int | None = None, max_height: int | None = None, container: bool = False, padding: bool = False, autoscroll: bool = False, buttons: list[Button] | None = None, head: str | None = None, server_functions: list[Callable] | None = None, props: Any)
+```
+
+Creates a component with arbitrary HTML. Can include CSS and JavaScript to create highly customized and interactive components.

.agents/skills/gradio/references/event-listeners.md

@@ -0,0 +1,132 @@
+# Event Listeners
+
+Events supported by each component.
+
+
+## Event Listener Signature
+
+```python
+component.event_name(
+    fn: Callable | None | Literal["decorator"] = "decorator",
+    inputs: Component | Sequence[Component] | set[Component] | None = None,
+    outputs: Component | Sequence[Component] | set[Component] | None = None,
+    api_name: str | None = None,
+    api_description: str | None | Literal[False] = None,
+    scroll_to_output: bool = False,
+    show_progress: Literal["full", "minimal", "hidden"] = "full",
+    show_progress_on: Component | Sequence[Component] | None = None,
+    queue: bool = True,
+    batch: bool = False,
+    max_batch_size: int = 4,
+    preprocess: bool = True,
+    postprocess: bool = True,
+    cancels: dict[str, Any] | list[dict[str, Any]] | None = None,
+    trigger_mode: Literal["once", "multiple", "always_last"] | None = None,
+    js: str | Literal[True] | None = None,
+    concurrency_limit: int | None | Literal["default"] = "default",
+    concurrency_id: str | None = None,
+    api_visibility: Literal["public", "private", "undocumented"] = "public",
+    time_limit: int | None = None,
+    stream_every: float = 0.5,
+    key: int | str | tuple[int | str, ...] | None = None,
+    validator: Callable | None = None,
+) -> Dependency
+```
+
+## Supported Events by Component
+
+- **AnnotatedImage**: change, select
+
+- **Audio**: stream, change, clear, play, pause, stop, pause, start_recording, pause_recording, stop_recording, upload, input
+
+- **BarPlot**: change, select, double_click
+
+- **BrowserState**: change
+
+- **Button**: change, click
+
+- **Chatbot**: change, select, like, retry, undo, example_select, option_select, clear, copy, edit
+
+- **Checkbox**: change, input, select
+
+- **CheckboxGroup**: change, input, select
+
+- **ClearButton**: change, click
+
+- **Code**: change, input, focus, blur
+
+- **ColorPicker**: change, input, release, submit, focus, blur
+
+- **Dataframe**: change, input, select, edit
+
+- **Dataset**: change, click, select
+
+- **DateTime**: change, submit
+
+- **DeepLinkButton**: change, click
+
+- **Dialogue**: change, input, submit
+
+- **DownloadButton**: change, click
+
+- **Dropdown**: change, input, select, focus, blur, key_up
+
+- **DuplicateButton**: change, click
+
+- **File**: change, select, clear, upload, delete, download
+
+- **FileExplorer**: change, input, select
+
+- **Gallery**: select, upload, change, delete, preview_close, preview_open
+
+- **HTML**: change, input, click, double_click, submit, stop, edit, clear, play, pause, end, start_recording, pause_recording, stop_recording, focus, blur, upload, release, select, stream, like, example_select, option_select, load, key_up, apply, delete, tick, undo, retry, expand, collapse, download, copy
+
+- **HighlightedText**: change, select
+
+- **Image**: clear, change, stream, select, upload, input
+
+- **ImageEditor**: clear, change, input, select, upload, apply
+
+- **ImageSlider**: clear, change, stream, select, upload, input
+
+- **JSON**: change
+
+- **Label**: change, select
+
+- **LinePlot**: change, select, double_click
+
+- **LoginButton**: change, click
+
+- **Markdown**: change, copy
+
+- **Model3D**: change, upload, edit, clear
+
+- **MultimodalTextbox**: change, input, select, submit, focus, blur, stop
+
+- **Navbar**: change
+
+- **Number**: change, input, submit, focus, blur
+
+- **ParamViewer**: change, upload
+
+- **Plot**: change
+
+- **Radio**: select, change, input
+
+- **ScatterPlot**: change, select, double_click
+
+- **SimpleImage**: clear, change, upload
+
+- **Slider**: change, input, release
+
+- **State**: change
+
+- **Textbox**: change, input, select, submit, focus, blur, stop, copy
+
+- **Timer**: change, tick
+
+- **UploadButton**: change, click, upload
+
+- **Video**: change, clear, start_recording, stop_recording, stop, play, pause, end, upload, input
+
+- **WorkflowCanvas**: change

.agents/skills/gradio/references/examples.md

@@ -0,0 +1,645 @@
+# Gradio End-to-End Examples
+
+Complete working Gradio apps for reference.
+
+## Blocks Essay Simple
+
+```python
+import gradio as gr
+
+def change_textbox(choice):
+    if choice == "short":
+        return gr.Textbox(lines=2, visible=True)
+    elif choice == "long":
+        return gr.Textbox(lines=8, visible=True, value="Lorem ipsum dolor sit amet")
+    else:
+        return gr.Textbox(visible=False)
+
+with gr.Blocks() as demo:
+    radio = gr.Radio(
+        ["short", "long", "none"], label="What kind of essay would you like to write?"
+    )
+    text = gr.Textbox(lines=2, interactive=True, buttons=["copy"])
+    radio.change(fn=change_textbox, inputs=radio, outputs=text)
+
+demo.launch()
+```
+
+## Blocks Flipper
+
+```python
+import numpy as np
+import gradio as gr
+
+def flip_text(x):
+    return x[::-1]
+
+def flip_image(x):
+    return np.fliplr(x)
+
+with gr.Blocks() as demo:
+    gr.Markdown("Flip text or image files using this demo.")
+    with gr.Tab("Flip Text"):
+        text_input = gr.Textbox()
+        text_output = gr.Textbox()
+        text_button = gr.Button("Flip")
+    with gr.Tab("Flip Image"):
+        with gr.Row():
+            image_input = gr.Image()
+            image_output = gr.Image()
+        image_button = gr.Button("Flip")
+
+    with gr.Accordion("Open for More!", open=False):
+        gr.Markdown("Look at me...")
+        temp_slider = gr.Slider(
+            0, 1,
+            value=0.1,
+            step=0.1,
+            interactive=True,
+            label="Slide me",
+        )
+
+    text_button.click(flip_text, inputs=text_input, outputs=text_output)
+    image_button.click(flip_image, inputs=image_input, outputs=image_output)
+
+demo.launch()
+```
+
+## Blocks Form
+
+```python
+import gradio as gr
+
+with gr.Blocks() as demo:
+    name_box = gr.Textbox(label="Name")
+    age_box = gr.Number(label="Age", minimum=0, maximum=100)
+    symptoms_box = gr.CheckboxGroup(["Cough", "Fever", "Runny Nose"])
+    submit_btn = gr.Button("Submit")
+
+    with gr.Column(visible=False) as output_col:
+        diagnosis_box = gr.Textbox(label="Diagnosis")
+        patient_summary_box = gr.Textbox(label="Patient Summary")
+
+    def submit(name, age, symptoms):
+        return {
+            submit_btn: gr.Button(visible=False),
+            output_col: gr.Column(visible=True),
+            diagnosis_box: "covid" if "Cough" in symptoms else "flu",
+            patient_summary_box: f"{name}, {age} y/o",
+        }
+
+    submit_btn.click(
+        submit,
+        [name_box, age_box, symptoms_box],
+        [submit_btn, diagnosis_box, patient_summary_box, output_col],
+    )
+
+demo.launch()
+```
+
+## Blocks Hello
+
+```python
+import gradio as gr
+
+def welcome(name):
+    return f"Welcome to Gradio, {name}!"
+
+with gr.Blocks() as demo:
+    gr.Markdown(
+    """
+    # Hello World!
+    Start typing below to see the output.
+    """)
+    inp = gr.Textbox(placeholder="What is your name?")
+    out = gr.Textbox()
+    inp.change(welcome, inp, out)
+
+demo.launch()
+```
+
+## Blocks Layout
+
+```python
+import gradio as gr
+
+demo = gr.Blocks()
+
+with demo:
+    with gr.Row():
+        gr.Image(interactive=True, scale=2)
+        gr.Image()
+    with gr.Row():
+        gr.Textbox(label="Text")
+        gr.Number(label="Count", scale=2)
+        gr.Radio(choices=["One", "Two"])
+    with gr.Row():
+        gr.Button("500", scale=0, min_width=500)
+        gr.Button("A", scale=0)
+        gr.Button("grow")
+    with gr.Row():
+        gr.Textbox()
+        gr.Textbox()
+        gr.Button()
+    with gr.Row():
+        with gr.Row():
+            with gr.Column():
+                gr.Textbox(label="Text")
+                gr.Number(label="Count")
+                gr.Radio(choices=["One", "Two"])
+            gr.Image()
+            with gr.Column():
+                gr.Image(interactive=True)
+                gr.Image()
+    gr.Image()
+    gr.Textbox(label="Text")
+    gr.Number(label="Count")
+    gr.Radio(choices=["One", "Two"])
+
+demo.launch()
+```
+
+## Calculator
+
+```python
+import gradio as gr
+
+def calculator(num1, operation, num2):
+    if operation == "add":
+        return num1 + num2
+    elif operation == "subtract":
+        return num1 - num2
+    elif operation == "multiply":
+        return num1 * num2
+    elif operation == "divide":
+        if num2 == 0:
+            raise gr.Error("Cannot divide by zero!")
+        return num1 / num2
+
+demo = gr.Interface(
+    calculator,
+    [
+        "number",
+        gr.Radio(["add", "subtract", "multiply", "divide"]),
+        "number"
+    ],
+    "number",
+    examples=[
+        [45, "add", 3],
+        [3.14, "divide", 2],
+        [144, "multiply", 2.5],
+        [0, "subtract", 1.2],
+    ],
+    title="Toy Calculator",
+    description="Here's a sample toy calculator.",
+    api_name="predict"
+)
+
+demo.launch()
+```
+
+## Chatbot Simple
+
+```python
+import gradio as gr
+import random
+import time
+
+with gr.Blocks() as demo:
+    chatbot = gr.Chatbot()
+    msg = gr.Textbox()
+    clear = gr.ClearButton([msg, chatbot])
+
+    def respond(message, chat_history):
+        bot_message = random.choice(["How are you?", "Today is a great day", "I'm very hungry"])
+        chat_history.append({"role": "user", "content": message})
+        chat_history.append({"role": "assistant", "content": bot_message})
+        time.sleep(2)
+        return "", chat_history
+
+    msg.submit(respond, [msg, chatbot], [msg, chatbot])
+
+demo.launch()
+```
+
+## Chatbot Streaming
+
+```python
+import gradio as gr
+import random
+import time
+
+with gr.Blocks() as demo:
+    chatbot = gr.Chatbot()
+    msg = gr.Textbox()
+    clear = gr.Button("Clear")
+
+    def user(user_message, history: list):
+        return "", history + [{"role": "user", "content": user_message}]
+
+    def bot(history: list):
+        bot_message = random.choice(["How are you?", "I love you", "I'm very hungry"])
+        history.append({"role": "assistant", "content": ""})
+        for character in bot_message:
+            history[-1]['content'] += character
+            time.sleep(0.05)
+            yield history
+
+    msg.submit(user, [msg, chatbot], [msg, chatbot], queue=False).then(
+        bot, chatbot, chatbot
+    )
+    clear.click(lambda: None, None, chatbot, queue=False)
+
+demo.launch()
+```
+
+## Custom Css
+
+```python
+import gradio as gr
+
+with gr.Blocks() as demo:
+    with gr.Column(elem_classes="cool-col"):
+        gr.Markdown("### Gradio Demo with Custom CSS", elem_classes="darktest")
+        gr.Markdown(
+            elem_classes="markdown",
+            value="Resize the browser window to see the CSS media query in action.",
+        )
+
+if __name__ == "__main__":
+    demo.launch(css_paths=["demo/custom_css/custom_css.css"])
+```
+
+## Fake Diffusion
+
+```python
+import gradio as gr
+import numpy as np
+import time
+
+def fake_diffusion(steps):
+    rng = np.random.default_rng()
+    for i in range(steps):
+        time.sleep(1)
+        image = rng.random(size=(600, 600, 3))
+        yield image
+    image = np.ones((1000,1000,3), np.uint8)
+    image[:] = [255, 124, 0]
+    yield image
+
+demo = gr.Interface(fake_diffusion,
+                    inputs=gr.Slider(1, 10, 3, step=1),
+                    outputs="image",
+                    api_name="predict")
+
+demo.launch()
+```
+
+## Hello World
+
+```python
+import gradio as gr
+
+
+def greet(name):
+    return "Hello " + name + "!"
+
+
+demo = gr.Interface(fn=greet, inputs="textbox", outputs="textbox", api_name="predict")
+
+demo.launch()
+```
+
+## Image Editor
+
+```python
+import gradio as gr
+import time
+
+
+def sleep(im):
+    time.sleep(5)
+    return [im["background"], im["layers"][0], im["layers"][1], im["composite"]]
+
+
+def predict(im):
+    return im["composite"]
+
+
+with gr.Blocks() as demo:
+    with gr.Row():
+        im = gr.ImageEditor(
+            type="numpy",
+        )
+        im_preview = gr.Image()
+    n_upload = gr.Number(0, label="Number of upload events", step=1)
+    n_change = gr.Number(0, label="Number of change events", step=1)
+    n_input = gr.Number(0, label="Number of input events", step=1)
+
+    im.upload(lambda x: x + 1, outputs=n_upload, inputs=n_upload)
+    im.change(lambda x: x + 1, outputs=n_change, inputs=n_change)
+    im.input(lambda x: x + 1, outputs=n_input, inputs=n_input)
+    im.change(predict, outputs=im_preview, inputs=im, show_progress="hidden")
+
+demo.launch()
+```
+
+## On Listener Decorator
+
+```python
+import gradio as gr
+
+with gr.Blocks() as demo:
+    name = gr.Textbox(label="Name")
+    output = gr.Textbox(label="Output Box")
+    greet_btn = gr.Button("Greet")
+
+    @gr.on(triggers=[name.submit, greet_btn.click], inputs=name, outputs=output)
+    def greet(name):
+        return "Hello " + name + "!"
+
+demo.launch()
+```
+
+## Render Merge
+
+```python
+import gradio as gr
+import time
+
+
+JS_ON_LOAD = """
+element.querySelectorAll('button').forEach((button) => {
+    button.addEventListener('click', () => {
+        trigger('click', {clicked: button.innerText});
+    });
+});
+"""
+
+
+with gr.Blocks() as demo:
+    text_count = gr.Slider(1, 5, value=1, step=1, label="Textbox Count")
+
+    @gr.render(inputs=text_count)
+    def render_count(count):
+        boxes = []
+        for i in range(count):
+            box = gr.Textbox(label=f"Box {i}")
+            boxes.append(box)
+
+        def merge(*args):
+            time.sleep(0.2)  # simulate a delay
+            return " ".join(args)
+
+        merge_btn.click(merge, boxes, output)
+
+        def clear():
+            time.sleep(0.2)  # simulate a delay
+            return [" "] * count
+
+        clear_btn.click(clear, None, boxes)
+
+        def countup():
+            time.sleep(0.2)  # simulate a delay
+            return list(range(count))
+
+        count_btn.click(countup, None, boxes, queue=False)
+
+    with gr.Row():
+        merge_btn = gr.Button("Merge")
+        clear_btn = gr.Button("Clear")
+        count_btn = gr.Button("Count")
+
+    output = gr.Textbox()
+
+    html_button_count = gr.State(2)
+
+    @gr.render(inputs=html_button_count)
+    def render_html_buttons(count):
+        buttons = gr.HTML(
+            "".join(f"<button>HTML button {i}</button>" for i in range(count)),
+            js_on_load=JS_ON_LOAD,
+        )
+        clicked = gr.Textbox(label="Clicked HTML button")
+
+        def select_button(evt: gr.EventData):
+            return evt.clicked
+
+        buttons.click(select_button, outputs=clicked)
+
+    add_html_button = gr.Button("Add HTML button")
+    add_html_button.click(
+        lambda count: count + 1,
+        inputs=html_button_count,
+        outputs=html_button_count,
+    )
+
+demo.launch()
+```
+
+## Reverse Audio 2
+
+```python
+import gradio as gr
+import numpy as np
+
+def reverse_audio(audio):
+    sr, data = audio
+    return (sr, np.flipud(data))
+
+demo = gr.Interface(fn=reverse_audio,
+                    inputs="microphone",
+                    outputs="audio", api_name="predict")
+
+demo.launch()
+```
+
+## Sepia Filter
+
+```python
+import numpy as np
+import gradio as gr
+
+def sepia(input_img):
+    sepia_filter = np.array([
+        [0.393, 0.769, 0.189],
+        [0.349, 0.686, 0.168],
+        [0.272, 0.534, 0.131]
+    ])
+    sepia_img = input_img.dot(sepia_filter.T)
+    sepia_img /= sepia_img.max()
+    return sepia_img
+
+demo = gr.Interface(sepia, gr.Image(), "image", api_name="predict")
+demo.launch()
+```
+
+## Sort Records
+
+```python
+import gradio as gr
+
+def sort_records(records):
+    return records.sort("Quantity")
+
+demo = gr.Interface(
+    sort_records,
+    gr.Dataframe(
+        headers=["Item", "Quantity"],
+        datatype=["str", "number"],  
+        row_count=3,
+        column_count=2,
+        column_limits=(2, 2),
+        type="polars"
+    ),
+    "dataframe",
+    description="Sort by Quantity"
+)
+
+demo.launch()
+```
+
+## Streaming Simple
+
+```python
+import gradio as gr
+
+with gr.Blocks() as demo:
+    with gr.Row():
+        with gr.Column():
+            input_img = gr.Image(label="Input", sources="webcam")
+        with gr.Column():
+            output_img = gr.Image(label="Output")
+        input_img.stream(lambda s: s, input_img, output_img, time_limit=15, stream_every=0.1, concurrency_limit=30)
+
+if __name__ == "__main__":
+
+    demo.launch()
+```
+
+## Tabbed Interface Lite
+
+```python
+import gradio as gr
+
+hello_world = gr.Interface(lambda name: "Hello " + name, "text", "text", api_name="predict")
+bye_world = gr.Interface(lambda name: "Bye " + name, "text", "text", api_name="predict")
+chat = gr.ChatInterface(lambda *args: "Hello " + args[0], api_name="chat")
+
+demo = gr.TabbedInterface([hello_world, bye_world, chat], ["Hello World", "Bye World", "Chat"])
+
+demo.launch()
+```
+
+## Tax Calculator
+
+```python
+import gradio as gr
+
+def tax_calculator(income, marital_status, assets):
+    tax_brackets = [(10, 0), (25, 8), (60, 12), (120, 20), (250, 30)]
+    total_deductible = sum(cost for cost, deductible in zip(assets["Cost"], assets["Deductible"]) if deductible)
+    taxable_income = income - total_deductible
+
+    total_tax = 0
+    for bracket, rate in tax_brackets:
+        if taxable_income > bracket:
+            total_tax += (taxable_income - bracket) * rate / 100
+
+    if marital_status == "Married":
+        total_tax *= 0.75
+    elif marital_status == "Divorced":
+        total_tax *= 0.8
+
+    return round(total_tax)
+
+demo = gr.Interface(
+    tax_calculator,
+    [
+        "number",
+        gr.Radio(["Single", "Married", "Divorced"]),
+        gr.Dataframe(
+            headers=["Item", "Cost", "Deductible"],
+            datatype=["str", "number", "bool"],  
+            label="Assets Purchased this Year",
+        ),
+    ],
+    gr.Number(label="Tax due"),
+    examples=[
+        [10000, "Married", [["Suit", 5000, True], ["Laptop (for work)", 800, False], ["Car", 1800, True]]],
+        [80000, "Single", [["Suit", 800, True], ["Watch", 1800, True], ["Food", 800, True]]],
+    ],
+    live=True,
+    api_name="predict"
+)
+
+demo.launch()
+```
+
+## Timer Simple
+
+```python
+import gradio as gr
+import random
+import time
+
+with gr.Blocks() as demo:
+  timer = gr.Timer(1)
+  timestamp = gr.Number(label="Time")
+  timer.tick(lambda: round(time.time()), outputs=timestamp, api_name="timestamp")
+
+  number = gr.Number(lambda: random.randint(1, 10), every=timer, label="Random Number")
+  with gr.Row():
+    gr.Button("Start").click(lambda: gr.Timer(active=True), None, timer)
+    gr.Button("Stop").click(lambda: gr.Timer(active=False), None, timer)
+    gr.Button("Go Fast").click(lambda: 0.2, None, timer)
+
+if __name__ == "__main__":
+  demo.launch()
+```
+
+## Variable Outputs
+
+```python
+import gradio as gr
+
+max_textboxes = 10
+
+def variable_outputs(k):
+    k = int(k)
+    return [gr.Textbox(visible=True)]*k + [gr.Textbox(visible=False)]*(max_textboxes-k)
+
+with gr.Blocks() as demo:
+    s = gr.Slider(1, max_textboxes, value=max_textboxes, step=1, label="How many textboxes to show:")
+    textboxes = []
+    for i in range(max_textboxes):
+        t = gr.Textbox(f"Textbox {i}")
+        textboxes.append(t)
+
+    s.change(variable_outputs, s, textboxes)
+
+if __name__ == "__main__":
+   demo.launch()
+```
+
+## Video Identity
+
+```python
+import gradio as gr
+from gradio.media import get_video
+
+def video_identity(video):
+    return video
+
+# get_video() returns file paths to sample media included with Gradio
+demo = gr.Interface(video_identity,
+                    gr.Video(),
+                    "playable_video",
+                    examples=[
+                        get_video("world.mp4")
+                    ],
+                    cache_examples=True,
+                    api_name="predict",)
+
+demo.launch()
+```

.agents/skills/hf-gradio/SKILL.md

@@ -0,0 +1,83 @@
+---
+name: hf-gradio
+description: Use Gradio applications via API. Use when the user asks for to generate a prediction from a Gradio app on Hugging Face spaces or public URL. For example, "Generate an image using black-forest-labs/FLUX.2-dev". 
+---
+
+## hf-gradio CLI Skill
+
+The `hf-gradio` CLI `gradio` CLI includes `info` and `predict` commands for interacting with Gradio apps programmatically. 
+
+## Step 1 - Verify installation
+
+Verify that either `hf-gradio` or `gradio` are installed in the current virtual environment.
+
+If the `hf` CLI app is installed. The `hf-gradio` extension can be installed via
+
+```bash
+hf extensions install gradio-app/hf-gradio
+```
+
+### Step 2 - Use `info` to discover endpoints and payload format
+
+```bash
+gradio info <space_id_or_url>
+```
+
+```bash
+hf-gradio info <space_id_or_url>
+```
+
+```bash
+hf gradio info <space_id_or_url>
+```
+
+Returns a JSON payload describing all endpoints, their parameters (with types and defaults), and return values.
+
+```bash
+gradio info gradio/calculator
+#    {
+#   "/predict": {
+#     "parameters": [
+#       {"name": "num1", "required": true, "default": null, "type": {"type": "number"}},
+#       {"name": "operation", "required": true, "default": null, "type": {"enum": ["add", "subtract", "multiply", "divide"], "type": "string"}},
+#       {"name": "num2", "required": true, "default": null, "type": {"type": "number"}}
+#     ],
+#     "returns": [{"name": "output", "type": {"type": "number"}}],
+#     "description": ""
+#   }
+# }
+```
+
+File-type parameters show `"type": "filepath"` with instructions to include `"meta": {"_type": "gradio.FileData"}` — this signals the file will be uploaded to the remote server.
+
+## Step 3 - Use `predict` to generate the prediction
+
+```bash
+gradio predict <space_id_or_url> <endpoint> <json_payload>
+```
+
+```bash
+hf-gradio predict <space_id_or_url> <endpoint> <json_payload>
+```
+
+```bash
+hf gradio predict <space_id_or_url> <endpoint> <json_payload>
+```
+
+Returns a JSON object with named output keys.
+
+```bash
+# Simple numeric prediction
+gradio predict gradio/calculator /predict '{"num1": 5, "operation": "multiply", "num2": 3}'
+# {"output": 15}
+
+# Image generation
+gradio predict black-forest-labs/FLUX.2-dev /infer '{"prompt": "A majestic dragon"}'
+# {"Result": "/tmp/gradio/.../image.webp", "Seed": 1117868604}
+
+# File upload (must include meta key)
+gradio predict gradio/image_mod /predict '{"image": {"path": "/path/to/image.png", "meta": {"_type": "gradio.FileData"}}}'
+# {"output": "/tmp/gradio/.../output.png"}
+```
+
+Both commands accept `--token` for accessing private Spaces.

.codex/skills/gradio

@@ -0,0 +1 @@
+../../.agents/skills/gradio

.codex/skills/hf-gradio

@@ -0,0 +1 @@
+../../.agents/skills/hf-gradio

.gitattributes

@@ -0,0 +1,2 @@
+*.png filter=lfs diff=lfs merge=lfs -text
+*.webp filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,10 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+
+# Virtual environments
+.venv

app/config.py

@@ -0,0 +1,2 @@
+import os
+TRANSFORMERS_MODEL = os.environ.get("TRANSFORMERS_MODEL", "mradermacher/NuMarkdown-8B-Thinking-i1-GGUF")

app/nuextract.py

@@ -0,0 +1,7 @@
+from transformers import AutoModel
+from app.config import TRANSFORMERS_MODEL
+
+
+
+model = AutoModel.from_pretrained(TRANSFORMERS_MODEL, dtype="auto")
+

mise.toml

@@ -0,0 +1,7 @@
+[tools]
+powershell = "7"
+python = "3.12"
+uv = "latest"
+
+[env]
+UV_LINK_MODE="copy"

notebooks/nuextract-2.0_inference.ipynb

@@ -0,0 +1,876 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "cbe8b1f6-ec66-4fbc-a9f8-774acd8a255a",
+   "metadata": {},
+   "source": [
+    "# NuExtract 2.0 Inference\n",
+    "\n",
+    "In this notebook we will provide examples of how to use the NuExtract 2.0 models for inference.\n",
+    "\n",
+    "First, let's load a model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a561e38e-58e5-4f7e-80ee-eabc2d05e49a",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "s:\\Spaces\\Data-Extraction\\NuMarkApp\\.venv\\Lib\\site-packages\\tqdm\\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
+      "  from .autonotebook import tqdm as notebook_tqdm\n"
+     ]
+    }
+   ],
+   "source": [
+    "import torch\n",
+    "from transformers import AutoModelForImageTextToText, AutoProcessor\n",
+    "model_name = \"numind/NuExtract-2.0-2B\"\n",
+    "# model_name = \"numind/NuExtract-2.0-8B\"\n",
+    "\n",
+    "processor = AutoProcessor.from_pretrained(model_name,\n",
+    "                                          trust_remote_code=True,\n",
+    "                                          padding_side='left',\n",
+    "                                          use_fast=True)\n",
+    "model = AutoModelForImageTextToText.from_pretrained(model_name,\n",
+    "                                               trust_remote_code=True,\n",
+    "                                               dtype=torch.bfloat16,\n",
+    "                                               attn_implementation=\"flash_attention_2\",\n",
+    "                                               device_map=\"auto\")\n",
+    "\n",
+    "# we choose greedy sampling here, which works well for most information extraction tasks\n",
+    "generation_config = {\n",
+    "    \"do_sample\": False,\n",
+    "    \"num_beams\": 1,\n",
+    "    \"max_new_tokens\": 2048,\n",
+    "    \"temperature\": None,\n",
+    "    \"top_p\": None,\n",
+    "    \"top_k\": None\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d739f582-16ad-4e52-b012-429757945a84",
+   "metadata": {},
+   "source": [
+    "## Preparing Model Inputs\n",
+    "\n",
+    "Before using the model, we also need to make sure our prompts are properly formatted to work with NuExtract. NuExtract expects all input information to come as a single user chat prompt, formatted as follows:\n",
+    "\n",
+    "```python\n",
+    "f\"\"\"\n",
+    "# Template:\n",
+    "{template}\n",
+    "# Context:\n",
+    "{text}\n",
+    "\"\"\"\n",
+    "```\n",
+    "and if in-context examples are provided:\n",
+    "```python\n",
+    "f\"\"\"\n",
+    "# Template:\n",
+    "{template}\n",
+    "# Examples\n",
+    "## Input:\n",
+    "{input1}\n",
+    "## Output:\n",
+    "{output1}\n",
+    "## Input:\n",
+    "{input2}\n",
+    "## Output:\n",
+    "{output2}\n",
+    "# Context:\n",
+    "{text}\n",
+    "\"\"\"\n",
+    "```\n",
+    "\n",
+    "If you are working with image inputs, you should use image placeholders for `text`, `input1`, etc. Later, we will inject tokens representing the actual image content in the location of these placeholders.\n",
+    "\n",
+    "The following function can make this formatting more convenient for us (the model can actually do this for us internally, but we will use this function initially to illustrate how things work). "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "58b017a4-4802-4afe-b2b4-8f034f572c24",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "def construct_messages(document, template, examples=None, image_placeholder=\"<|vision_start|><|image_pad|><|vision_end|>\"):\n",
+    "    \"\"\"\n",
+    "    Construct the individual NuExtract message texts, prior to chat template formatting.\n",
+    "    \"\"\"\n",
+    "    images = []\n",
+    "    # add few-shot examples if needed\n",
+    "    if examples is not None and len(examples) > 0:\n",
+    "        icl = \"# Examples:\\n\"\n",
+    "        for row in examples:\n",
+    "            example_input = row['input']\n",
+    "\n",
+    "            if not isinstance(row['input'], str):\n",
+    "                example_input = image_placeholder\n",
+    "                images.append(row['input'])\n",
+    "\n",
+    "            icl += f\"## Input:\\n{example_input}\\n## Output:\\n{row['output']}\\n\"\n",
+    "    else:\n",
+    "        icl = \"\"\n",
+    "\n",
+    "    # if input document is an image, set text to an image placeholder\n",
+    "    text = document\n",
+    "    if not isinstance(document, str):\n",
+    "        text = image_placeholder\n",
+    "        images.append(document)\n",
+    "    text = f\"\"\"# Template:\\n{template}\\n{icl}# Context:\\n{text}\"\"\"\n",
+    "\n",
+    "    messages = [\n",
+    "        {\n",
+    "            \"role\": \"system\",\n",
+    "            \"content\": \"You are NuExtract, an information extraction tool created by NuMind.\"\n",
+    "        },\n",
+    "        {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": [{\"type\": \"text\", \"text\": text}] + images,\n",
+    "        }\n",
+    "    ]\n",
+    "    return messages"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e918d64e-bc31-4d68-8ef5-2d485b5e5745",
+   "metadata": {},
+   "source": [
+    "## Inference\n",
+    "### Basic Example\n",
+    "\n",
+    "Now we are ready to run the model!\n",
+    "\n",
+    "Let's start with a basic text-only example, where we want to extract peoples' names from a short text."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b1cad1ec-4753-4064-aca0-c89fec4b5bfd",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from qwen_vl_utils import process_vision_info\n",
+    "\n",
+    "template = \"\"\"{\"names\": [\"verbatim-string\"]}\"\"\"\n",
+    "document = \"John went to the restaurant with Mary. James went to the cinema.\"\n",
+    "\n",
+    "# prepare the user message content\n",
+    "messages = construct_messages(document, template)\n",
+    "text = processor.apply_chat_template(\n",
+    "    messages, tokenize=False, add_generation_prompt=True,\n",
+    ")\n",
+    "\n",
+    "image_inputs = process_vision_info(messages)[0]\n",
+    "inputs = processor(\n",
+    "    text=[text],\n",
+    "    images=image_inputs,\n",
+    "    padding=True,\n",
+    "    return_tensors=\"pt\",\n",
+    ").to(\"cuda\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6c0e4f92-94dd-4b37-8772-591246f9b71a",
+   "metadata": {},
+   "source": [
+    "Our NuExtract message is now formatted in standard chat template formatting; the tokenized version (`inputs`) will be given directly to the model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "42bd5add-b48a-44e4-9843-2f187e768678",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "<|im_start|>system\n",
+      "You are NuExtract, an information extraction tool created by NuMind.<|im_end|>\n",
+      "<|im_start|>user\n",
+      "# Template:\n",
+      "{\"names\": [\"verbatim-string\"]}\n",
+      "# Context:\n",
+      "John went to the restaurant with Mary. James went to the cinema.<|im_end|>\n",
+      "<|im_start|>assistant\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "df0979a7-ea46-4d25-a838-baeaeeca6ee5",
+   "metadata": {},
+   "source": [
+    "The other `image_inputs` are empty in this case because this is a text-only example."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e3493013-313f-4318-a813-d492732bb0c1",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "None\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(image_inputs)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0e300775-48a9-42fb-9c92-3e1ceb530101",
+   "metadata": {},
+   "source": [
+    "Now let's actually run the model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ed374bf3-1af1-4a3e-a9eb-9e2e7ca113b8",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "['{\"names\": [\"John\", \"Mary\", \"James\"]}']\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Inference: Generation of the output\n",
+    "generated_ids = model.generate(\n",
+    "    **inputs,\n",
+    "    **generation_config\n",
+    ")\n",
+    "generated_ids_trimmed = [\n",
+    "    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)\n",
+    "]\n",
+    "output_text = processor.batch_decode(\n",
+    "    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False\n",
+    ")\n",
+    "print(output_text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "21d37d33-85ad-421c-a014-523ddbec5a2f",
+   "metadata": {},
+   "source": [
+    "Alternatively, you can directly provide the template and in-context examples to `.apply_chat_template()`, rather then manually preparing the prompt via `construct_messages()`. This is a bit more convenient and so we will use this from now on."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "27fecb60-b0eb-46bc-ba2d-ef6a32d04f0c",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "<|im_start|>system\n",
+      "You are NuExtract, an information extraction tool created by NuMind.<|im_end|>\n",
+      "<|im_start|>user\n",
+      "# Template:\n",
+      "{\"names\": [\"verbatim-string\"]}\n",
+      "# Context:\n",
+      "John went to the restaurant with Mary. James went to the cinema.<|im_end|>\n",
+      "<|im_start|>assistant\n",
+      "\n",
+      "['{\"names\": [\"John\", \"Mary\", \"James\"]}']\n"
+     ]
+    }
+   ],
+   "source": [
+    "template = \"\"\"{\"names\": [\"verbatim-string\"]}\"\"\"\n",
+    "document = \"John went to the restaurant with Mary. James went to the cinema.\"\n",
+    "\n",
+    "# prepare the user message content\n",
+    "messages = [{\"role\": \"user\", \"content\": document}]\n",
+    "text = processor.tokenizer.apply_chat_template(\n",
+    "    messages,\n",
+    "    template=template, # template is specified here\n",
+    "    tokenize=False,\n",
+    "    add_generation_prompt=True,\n",
+    ")\n",
+    "\n",
+    "print(text)\n",
+    "\n",
+    "image_inputs = process_vision_info(messages)[0]\n",
+    "inputs = processor(\n",
+    "    text=[text],\n",
+    "    images=image_inputs,\n",
+    "    padding=True,\n",
+    "    return_tensors=\"pt\",\n",
+    ").to(\"cuda\")\n",
+    "\n",
+    "# Inference: Generation of the output\n",
+    "generated_ids = model.generate(\n",
+    "    **inputs,\n",
+    "    **generation_config\n",
+    ")\n",
+    "generated_ids_trimmed = [\n",
+    "    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)\n",
+    "]\n",
+    "output_text = processor.batch_decode(\n",
+    "    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False\n",
+    ")\n",
+    "\n",
+    "print(output_text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c18002f2-447a-45c2-90f6-0ff5a63c5dc2",
+   "metadata": {},
+   "source": [
+    "### In-Context Examples\n",
+    "\n",
+    "Sometimes the model might not perform as well as we want because our task is challenging or involves some degree of ambiguity. Alternatively, we may want the model to follow some specific formatting, or just give it a bit more help. In cases like this it can be valuable to provide \"in-context examples\" to help NuExtract better understand the task.\n",
+    "\n",
+    "To do so, we can provide a list `examples` to `apply_chat_template()` (or `construct_messages()`) which contains dictionaries of input/output pairs. In the example below, we show to the model that we want the extracted names to be in captial letters with `-` on either side (for the sake of illustration)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "146baa04-2206-4eca-86bd-66f94924e293",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "template = \"\"\"{\"names\": [\"verbatim-string\"]}\"\"\"\n",
+    "document = \"John went to the restaurant with Mary. James went to the cinema.\"\n",
+    "examples = [\n",
+    "    {\n",
+    "        \"input\": \"Stephen is the manager at Susan's store.\",\n",
+    "        \"output\": \"\"\"{\"names\": [\"-STEPHEN-\", \"-SUSAN-\"]}\"\"\"\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "messages = [{\"role\": \"user\", \"content\": document}]\n",
+    "text = processor.tokenizer.apply_chat_template(\n",
+    "    messages,\n",
+    "    template=template,\n",
+    "    examples=examples, # examples provided here\n",
+    "    tokenize=False,\n",
+    "    add_generation_prompt=True,\n",
+    ")\n",
+    "\n",
+    "image_inputs = process_vision_info(messages)[0]\n",
+    "inputs = processor(\n",
+    "    text=[text],\n",
+    "    images=image_inputs,\n",
+    "    padding=True,\n",
+    "    return_tensors=\"pt\",\n",
+    ").to(\"cuda\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1d2561cc-a662-4898-b226-41d12394fdb4",
+   "metadata": {},
+   "source": [
+    "We can see below that the in-context example has now been included in the model prompt, specifically between the template and context components."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "79b10e44-5bb4-40eb-af74-e013897948ea",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "<|im_start|>system\n",
+      "You are NuExtract, an information extraction tool created by NuMind.<|im_end|>\n",
+      "<|im_start|>user\n",
+      "# Template:\n",
+      "{\"names\": [\"verbatim-string\"]}\n",
+      "# Examples:\n",
+      "## Input:\n",
+      "Stephen is the manager at Susan's store.\n",
+      "## Output:\n",
+      "{\"names\": [\"-STEPHEN-\", \"-SUSAN-\"]}\n",
+      "# Context:\n",
+      "John went to the restaurant with Mary. James went to the cinema.<|im_end|>\n",
+      "<|im_start|>assistant\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(text)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3f1364bf-2588-4935-a478-7a85b87866fc",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "['{\"names\": [\"-JOHN-\", \"-MARY-\", \"-JAMES-\"]}']\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Inference: Generation of the output\n",
+    "generated_ids = model.generate(\n",
+    "    **inputs,\n",
+    "    **generation_config\n",
+    ")\n",
+    "generated_ids_trimmed = [\n",
+    "    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)\n",
+    "]\n",
+    "output_text = processor.batch_decode(\n",
+    "    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False\n",
+    ")\n",
+    "print(output_text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bc0c3ecf-80ee-40b7-b240-12b5a9e10042",
+   "metadata": {},
+   "source": [
+    "To get even better performance, add multiple in-context examples to your input."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "99faf117-18b7-4b5e-8cd0-b6f8941db581",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "### Image Inputs\n",
+    "\n",
+    "If we want to give image inputs to NuExtract, instead of text, we simply provide a dictionary specifying the desired image file as the message content, instead of a string. E.g. `{\"type\": \"image\", \"image\": \"file://image.jpg\"}`.\n",
+    "\n",
+    "You can also specify an image URL (e.g. `{\"type\": \"image\", \"image\": \"http://path/to/your/image.jpg\"}`) or base64 encoding (e.g. `{\"type\": \"image\", \"image\": \"data:image;base64,/9j/...\"}`).\n",
+    "\n",
+    "First, we will need a modified version of `process_vision_info()` that handles image-based in-context examples as well as primary inputs."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1aaf1912-4f50-483f-a9a0-f5f57caf66e0",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "def process_all_vision_info(messages, examples=None):\n",
+    "    \"\"\"\n",
+    "    Process vision information from both messages and in-context examples, supporting batch processing.\n",
+    "\n",
+    "    Args:\n",
+    "        messages: List of message dictionaries (single input) OR list of message lists (batch input)\n",
+    "        examples: Optional list of example dictionaries (single input) OR list of example lists (batch)\n",
+    "\n",
+    "    Returns:\n",
+    "        A flat list of all images in the correct order:\n",
+    "        - For single input: example images followed by message images\n",
+    "        - For batch input: interleaved as (item1 examples, item1 input, item2 examples, item2 input, etc.)\n",
+    "        - Returns None if no images were found\n",
+    "    \"\"\"\n",
+    "    from qwen_vl_utils import process_vision_info, fetch_image\n",
+    "\n",
+    "    # Helper function to extract images from examples\n",
+    "    def extract_example_images(example_item):\n",
+    "        if not example_item:\n",
+    "            return []\n",
+    "\n",
+    "        # Handle both list of examples and single example\n",
+    "        examples_to_process = example_item if isinstance(example_item, list) else [example_item]\n",
+    "        images = []\n",
+    "\n",
+    "        for example in examples_to_process:\n",
+    "            if isinstance(example.get('input'), dict) and example['input'].get('type') == 'image':\n",
+    "                images.append(fetch_image(example['input']))\n",
+    "\n",
+    "        return images\n",
+    "\n",
+    "    # Normalize inputs to always be batched format\n",
+    "    is_batch = messages and isinstance(messages[0], list)\n",
+    "    messages_batch = messages if is_batch else [messages]\n",
+    "    is_batch_examples = examples and isinstance(examples, list) and (isinstance(examples[0], list) or examples[0] is None)\n",
+    "    examples_batch = examples if is_batch_examples else ([examples] if examples is not None else None)\n",
+    "\n",
+    "    # Ensure examples batch matches messages batch if provided\n",
+    "    if examples and len(examples_batch) != len(messages_batch):\n",
+    "        if not is_batch and len(examples_batch) == 1:\n",
+    "            # Single example set for a single input is fine\n",
+    "            pass\n",
+    "        else:\n",
+    "            raise ValueError(\"Examples batch length must match messages batch length\")\n",
+    "\n",
+    "    # Process all inputs, maintaining correct order\n",
+    "    all_images = []\n",
+    "    for i, message_group in enumerate(messages_batch):\n",
+    "        # Get example images for this input\n",
+    "        if examples and i < len(examples_batch):\n",
+    "            input_example_images = extract_example_images(examples_batch[i])\n",
+    "            all_images.extend(input_example_images)\n",
+    "\n",
+    "        # Get message images for this input\n",
+    "        input_message_images = process_vision_info(message_group)[0] or []\n",
+    "        all_images.extend(input_message_images)\n",
+    "\n",
+    "    return all_images if all_images else None\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8db9904c-451d-4ff4-b776-5a557abc1437",
+   "metadata": {},
+   "source": [
+    "In the example below, we give an image of a receipt from Trader Joe's (`data/1.jpg`) and ask the model to extract the name of the store. We also provide an ICL example of a receipt from Walmart (`data/0.jpg`)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fd11e855-594b-4fd7-af01-7695a65b75a9",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "\n",
+    "project_root = Path.cwd().parent if Path.cwd().name == \"notebooks\" else Path.cwd()\n",
+    "samples_dir = project_root / \"data\" / \"samples\"\n",
+    "sample_images = sorted(\n",
+    "    path for path in samples_dir.iterdir()\n",
+    "    if path.suffix.lower() in {\".jpg\", \".jpeg\", \".png\", \".webp\"}\n",
+    ")\n",
+    "if not sample_images:\n",
+    "    raise FileNotFoundError(f\"No supported images found in {samples_dir}\")\n",
+    "\n",
+    "# Pick a different entry from sample_images to use another file.\n",
+    "selected_image = sample_images[0]\n",
+    "input_image_uri = selected_image.resolve().as_uri()\n",
+    "print(f\"Using input image: {selected_image}\")\n",
+    "\n",
+    "template = \"\"\"{\"store\": \"verbatim-string\"}\"\"\"\n",
+    "document = {\"type\": \"image\", \"image\": input_image_uri}\n",
+    "examples = [\n",
+    "    {\n",
+    "        \"input\": {\"type\": \"image\", \"image\": input_image_uri},\n",
+    "        \"output\": \"\"\"{\"store\": \"WALMART\"}\"\"\"\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "messages = [{\"role\": \"user\", \"content\": [document]}]\n",
+    "text = processor.tokenizer.apply_chat_template(\n",
+    "    messages,\n",
+    "    template=template,\n",
+    "    examples=examples,\n",
+    "    tokenize=False,\n",
+    "    add_generation_prompt=True,\n",
+    ")\n",
+    "\n",
+    "image_inputs = process_all_vision_info(messages, examples)\n",
+    "inputs = processor(\n",
+    "    text=[text],\n",
+    "    images=image_inputs,\n",
+    "    padding=True,\n",
+    "    return_tensors=\"pt\",\n",
+    ").to(\"cuda\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f14a8e8a-a091-410a-a964-9d275b936e75",
+   "metadata": {},
+   "source": [
+    "Just like in the text-only case above, our in-context example has been included in the prompt before the main context."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ba00c8d6-56e2-43fc-a646-61f693c6b51a",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "<|im_start|>system\n",
+      "You are NuExtract, an information extraction tool created by NuMind.<|im_end|>\n",
+      "<|im_start|>user\n",
+      "# Template:\n",
+      "{\"store\": \"verbatim-string\"}\n",
+      "# Examples:\n",
+      "## Input:\n",
+      "<|vision_start|><|image_pad|><|vision_end|>\n",
+      "## Output:\n",
+      "{\"store\": \"WALMART\"}\n",
+      "# Context:\n",
+      "<|vision_start|><|image_pad|><|vision_end|><|im_end|>\n",
+      "<|im_start|>assistant\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4820e6da-544b-431a-86fb-a6f9f0eb6bdd",
+   "metadata": {},
+   "source": [
+    "Now if we look at `image_inputs` we will see that it contains actual images. When we pass this along with `text` to `processor()` it automatically encodes the images and injects a tokenized representation into the image placeholders within `text`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1aa9fba6-eb58-4401-abc2-79b7fcd205c0",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[<PIL.Image.Image image mode=RGB size=588x896 at 0x7FDD295E5600>, <PIL.Image.Image image mode=RGB size=476x980 at 0x7FDD295E4460>]\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(image_inputs)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fbed800c-519b-4ee1-90d9-7c27e383d904",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "['{\"store\": \"TRADER JOE\\'S\"}']\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Inference: Generation of the output\n",
+    "generated_ids = model.generate(\n",
+    "    **inputs,\n",
+    "    **generation_config\n",
+    ")\n",
+    "generated_ids_trimmed = [\n",
+    "    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)\n",
+    "]\n",
+    "output_text = processor.batch_decode(\n",
+    "    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False\n",
+    ")\n",
+    "print(output_text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c6beb77f-2de5-4846-b244-2cda27f5f553",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "### Batched Inference\n",
+    "\n",
+    "Finally, we can run batched inference over a list of input examples, regardless of whether they contain text, images, and/or ICL examples."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "947b8090-236c-4821-b295-5a8fb5247943",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\"store_name\": \"WAL-MART\"}\n",
+      "{\"store_name\": \"Walmart\"}\n",
+      "{\"names\": [\"John\", \"Mary\", \"James\"]}\n",
+      "{\"names\": [\"-JOHN-\", \"-MARY-\", \"-JAMES-\"]}\n"
+     ]
+    }
+   ],
+   "source": [
+    "inputs = [\n",
+    "    # image input with no ICL examples\n",
+    "    {\n",
+    "        \"document\": {\"type\": \"image\", \"image\": input_image_uri},\n",
+    "        \"template\": \"\"\"{\"store_name\": \"verbatim-string\"}\"\"\",\n",
+    "    },\n",
+    "    # image input with 1 ICL example\n",
+    "    {\n",
+    "        \"document\": {\"type\": \"image\", \"image\": input_image_uri},\n",
+    "        \"template\": \"\"\"{\"store_name\": \"verbatim-string\"}\"\"\",\n",
+    "        \"examples\": [\n",
+    "            {\n",
+    "                \"input\": {\"type\": \"image\", \"image\": input_image_uri},\n",
+    "                \"output\": \"\"\"{\"store_name\": \"Trader Joe's\"}\"\"\",\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "    # text input with no ICL examples\n",
+    "    {\n",
+    "        \"document\": {\"type\": \"text\", \"text\": \"John went to the restaurant with Mary. James went to the cinema.\"},\n",
+    "        \"template\": \"\"\"{\"names\": [\"verbatim-string\"]}\"\"\",\n",
+    "    },\n",
+    "    # text input with ICL example\n",
+    "    {\n",
+    "        \"document\": {\"type\": \"text\", \"text\": \"John went to the restaurant with Mary. James went to the cinema.\"},\n",
+    "        \"template\": \"\"\"{\"names\": [\"verbatim-string\"]}\"\"\",\n",
+    "        \"examples\": [\n",
+    "            {\n",
+    "                \"input\": \"Stephen is the manager at Susan's store.\",\n",
+    "                \"output\": \"\"\"{\"names\": [\"-STEPHEN-\", \"-SUSAN-\"]}\"\"\"\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "# messages should be a list of lists for batch processing\n",
+    "messages = [[{\"role\": \"user\", \"content\": [x['document']]}] for x in inputs]\n",
+    "\n",
+    "# apply chat template to each example individually\n",
+    "texts = [\n",
+    "    processor.tokenizer.apply_chat_template(\n",
+    "        messages[i],  # Now this is a list containing one message\n",
+    "        template=x['template'],\n",
+    "        examples=x.get('examples', None),\n",
+    "        tokenize=False,\n",
+    "        add_generation_prompt=True)\n",
+    "    for i, x in enumerate(inputs)\n",
+    "]\n",
+    "\n",
+    "image_inputs = process_all_vision_info(messages, [x.get('examples') for x in inputs])\n",
+    "inputs = processor(\n",
+    "    text=texts,\n",
+    "    images=image_inputs,\n",
+    "    padding=True,\n",
+    "    return_tensors=\"pt\",\n",
+    ").to(\"cuda\")\n",
+    "\n",
+    "# Batch Inference\n",
+    "generated_ids = model.generate(**inputs, **generation_config)\n",
+    "generated_ids_trimmed = [\n",
+    "    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)\n",
+    "]\n",
+    "output_texts = processor.batch_decode(\n",
+    "    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False\n",
+    ")\n",
+    "for y in output_texts:\n",
+    "    print(y)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cf456dcc-52ce-495d-872f-a641c6f40f7d",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "environment": {
+   "kernel": "conda-base-py",
+   "name": "workbench-notebooks.m127",
+   "type": "gcloud",
+   "uri": "us-docker.pkg.dev/deeplearning-platform-release/gcr.io/workbench-notebooks:m127"
+  },
+  "kernelspec": {
+   "display_name": ".venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

notebooks/nuextract-2.0_sft.ipynb

@@ -0,0 +1,619 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "313eccc5-d4c7-4ee9-8082-2a1e3670d26d",
+   "metadata": {},
+   "source": [
+    "# NuExtract 2.0 Supervised Fine-tuning (SFT)\n",
+    "\n",
+    "This notebook will show a basic example of how to perform supervised fine-tuning (SFT) on top of the base NuExtract 2.0 models, with your own data.\n",
+    "\n",
+    "## Prepare Model\n",
+    "First, load the model you want to fine-tune, along with the processor."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "cd9abd77-b0d6-4fac-9964-05da8e23f1de",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import torch\n",
+    "from transformers import AutoProcessor, AutoModelForVision2Seq\n",
+    "from qwen_vl_utils import process_vision_info\n",
+    "\n",
+    "model_name = \"numind/NuExtract-2.0-2B\"\n",
+    "# model_name = \"numind/NuExtract-2.0-8B\"\n",
+    "\n",
+    "model = AutoModelForVision2Seq.from_pretrained(model_name, \n",
+    "                                               trust_remote_code=True, \n",
+    "                                               torch_dtype=torch.bfloat16,\n",
+    "                                               attn_implementation=\"flash_attention_2\",\n",
+    "                                               device_map=\"auto\",\n",
+    "                                               use_cache=False, # for training\n",
+    "                                              )\n",
+    "\n",
+    "processor = AutoProcessor.from_pretrained(model_name, \n",
+    "                                          trust_remote_code=True, \n",
+    "                                          padding_side='right', # make sure to set padding to right for training\n",
+    "                                          use_fast=True,\n",
+    "                                         )\n",
+    "processor.eos_token = processor.tokenizer.eos_token\n",
+    "processor.eos_token_id = processor.tokenizer.eos_token_id"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "861218a0-9323-495a-b82e-d9fc4bde5670",
+   "metadata": {},
+   "source": [
+    "## Prepare Data\n",
+    "\n",
+    "The `construct_messages()` function below will help us to format the messages to be fed into the model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "05ad2f4c-116b-48ec-9a6b-e70ec0a3f996",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "def construct_messages(document, template, label=None, examples=None, image_placeholder=\"<|vision_start|><|image_pad|><|vision_end|>\"):\n",
+    "    \"\"\"\n",
+    "    Construct the individual NuExtract message texts, prior to chat template formatting.\n",
+    "    \"\"\"\n",
+    "    images = []\n",
+    "    \n",
+    "    # add few-shot examples if needed\n",
+    "    icl = \"\"\n",
+    "    if examples is not None and len(examples) > 0:\n",
+    "        icl = \"# Examples:\\n\"\n",
+    "        for row in examples:\n",
+    "            example_input = row['input']\n",
+    "            \n",
+    "            if not isinstance(row['input'], str):\n",
+    "                example_input = image_placeholder\n",
+    "                images.append(row['input'])\n",
+    "                \n",
+    "            icl += f\"## Input:\\n{example_input}\\n## Output:\\n{row['output']}\\n\"\n",
+    "        \n",
+    "    # if input document is an image, set text to an image placeholder\n",
+    "    text = document\n",
+    "    if not isinstance(document, str):\n",
+    "        text = image_placeholder\n",
+    "        images.append(document)\n",
+    "    text = f\"\"\"# Template:\\n{template}\\n{icl}# Context:\\n{text}\"\"\"\n",
+    "    \n",
+    "    messages = [\n",
+    "        {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": [{\"type\": \"text\", \"text\": text}] + images,\n",
+    "        }\n",
+    "    ]\n",
+    "    if label is not None:\n",
+    "        messages.append({\n",
+    "            \"role\": \"assistant\",\n",
+    "            \"content\": [{\"type\": \"text\", \"text\": label}],\n",
+    "            \n",
+    "        })\n",
+    "    return messages"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ee16acd5-c327-4a6b-b2c9-448a766fa3c0",
+   "metadata": {},
+   "source": [
+    "For illustration purposes, we will use a small dataset of manually created examples. You should prepare your own data in a similar way before fine-tuning your own model.\n",
+    "\n",
+    "In the custom data below, we will only provide examples that return strings in full lowercase characters (unless ICL examples suggest otherwise). If we fine-tune on this, we would ideally alter the model to favour returning strings in lowercase by default.\n",
+    "\n",
+    "*Note: training on a very small dataset like this is for illustration purposes only and will almost always result in a poorly performing model in real use-cases.*"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "4443f697-b11e-4818-b679-b0de2d8eb055",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "inputs = [\n",
+    "    # image input with no ICL examples\n",
+    "    {\n",
+    "        \"document\": {\"type\": \"image\", \"image\": \"file://data/0.jpg\"},\n",
+    "        \"template\": \"\"\"{\"store_name\": \"verbatim-string\"}\"\"\",\n",
+    "        \"label\": \"\"\"{\"store_name\": \"walmart\"}\"\"\", # lowercase result\n",
+    "    },\n",
+    "    # image input with 1 ICL example\n",
+    "    {\n",
+    "        \"document\": {\"type\": \"image\", \"image\": \"file://data/1.jpg\"},\n",
+    "        \"template\": \"\"\"{\"store_name\": \"verbatim-string\"}\"\"\",\n",
+    "        \"examples\": [\n",
+    "            {\n",
+    "                \"input\": {\"type\": \"image\", \"image\": \"file://data/0.jpg\"},\n",
+    "                \"output\": \"\"\"{\"store_name\": \"Walmart\"}\"\"\",\n",
+    "            }\n",
+    "        ],\n",
+    "        \"label\": \"\"\"{\"store_name\": \"Trader Joe's\"}\"\"\",\n",
+    "    },\n",
+    "    # text input with no ICL examples\n",
+    "    {\n",
+    "        \"document\": \"John went to the restaurant with Mary. James went to the cinema.\",\n",
+    "        \"template\": \"\"\"{\"names\": [\"verbatim-string\"]}\"\"\",\n",
+    "        \"label\": \"\"\"{\"names\": [\"john\", \"mary\", \"james\"]}\"\"\", # lowercase result\n",
+    "    },\n",
+    "    # text input with ICL example\n",
+    "    {\n",
+    "        \"document\": \"John went to the restaurant with Mary. James went to the cinema.\",\n",
+    "        \"template\": \"\"\"{\"names\": [\"verbatim-string\"]}\"\"\",\n",
+    "        \"examples\": [\n",
+    "            {\n",
+    "                \"input\": \"Stephen is the manager at Susan's store.\",\n",
+    "                \"output\": \"\"\"{\"names\": [\"STEPHEN\", \"SUSAN\"]}\"\"\"\n",
+    "            }\n",
+    "        ],\n",
+    "        \"label\": \"\"\"{\"names\": [\"JOHN\", \"MARY\", \"JAMES\"]}\"\"\",\n",
+    "    },\n",
+    "] * 2 # double examples to have dataset of size 8\n",
+    "\n",
+    "messages = [\n",
+    "    construct_messages(\n",
+    "        x[\"document\"], \n",
+    "        x[\"template\"], \n",
+    "        x[\"label\"],\n",
+    "        x[\"examples\"] if \"examples\" in x else None\n",
+    "    ) for x in inputs\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "e4428274-62b1-4fe7-9296-e702b1d9aba5",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[{'role': 'user',\n",
+       "  'content': [{'type': 'text',\n",
+       "    'text': '# Template:\\n{\"store_name\": \"verbatim-string\"}\\n# Context:\\n<|vision_start|><|image_pad|><|vision_end|>'},\n",
+       "   {'type': 'image', 'image': 'file://data/0.jpg'}]},\n",
+       " {'role': 'assistant',\n",
+       "  'content': [{'type': 'text', 'text': '{\"store_name\": \"walmart\"}'}]}]"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "messages[0]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "751f3ec5-9e3e-4951-b21a-53b8b03209dc",
+   "metadata": {},
+   "source": [
+    "Let's also add a couple of validation examples that we can use to confirm our model is generalizing to unseen data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "d619853f-d76c-4d28-aebb-0d26b99248fa",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "val_inputs = [\n",
+    "    {\n",
+    "        \"document\": \"Jack went to the hill with Jill. Rupert went to the diner.\",\n",
+    "        \"template\": \"\"\"{\"names\": [\"verbatim-string\"]}\"\"\",\n",
+    "        \"label\": \"\"\"{\"names\": [\"jack\", \"jill\", \"rupert\"]}\"\"\", # lowercase result\n",
+    "    },\n",
+    "    {\n",
+    "        \"document\": \"My dog Clifford likes to play fetch with Emily and Peter.\",\n",
+    "        \"template\": \"\"\"{\"names\": [\"verbatim-string\"]}\"\"\",\n",
+    "        \"label\": \"\"\"{\"names\": [\"clifford\", \"emily\", \"peter\"]}\"\"\", # lowercase result\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "val_messages = [\n",
+    "    construct_messages(\n",
+    "        x[\"document\"], \n",
+    "        x[\"template\"], \n",
+    "        x[\"label\"],\n",
+    "        x[\"examples\"] if \"examples\" in x else None\n",
+    "    ) for x in val_inputs\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8c129755-5a99-43e6-98d4-02fa8e1e16d1",
+   "metadata": {},
+   "source": [
+    "The data is now structued in message format that the processor will be able to reformat via the chat template before tokenization. We will do that on the fly during training via the collate function below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "dfac13eb-8ede-4939-b40b-a515c0d73110",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "def collate_fn(examples):\n",
+    "    # process input/prompt part of conversations\n",
+    "    user_texts = [processor.apply_chat_template(example[:1], tokenize=False) for example in examples]\n",
+    "    \n",
+    "    # process full conversations (user + assistant)\n",
+    "    full_texts = [processor.apply_chat_template(example, tokenize=False) for example in examples]\n",
+    "    \n",
+    "    # process images\n",
+    "    images = process_vision_info(examples)[0]\n",
+    "    \n",
+    "    # tokenize sequences\n",
+    "    user_batch = processor(text=user_texts, images=images, return_tensors=\"pt\", padding=True)\n",
+    "    full_batch = processor(text=full_texts, images=images, return_tensors=\"pt\", padding=True)\n",
+    "    \n",
+    "    # mask padding tokens\n",
+    "    labels = full_batch[\"input_ids\"].clone()\n",
+    "    labels[labels == processor.tokenizer.pad_token_id] = -100\n",
+    "    \n",
+    "    # mask user message tokens for each example in the batch\n",
+    "    for i in range(len(examples)):\n",
+    "        # length of prompt message (accounting for possible padding)\n",
+    "        user_len = user_batch[\"attention_mask\"][i].sum().item()\n",
+    "        \n",
+    "        # mask prompt part of label\n",
+    "        labels[i, :user_len - 1] = -100\n",
+    "    \n",
+    "    full_batch[\"labels\"] = labels\n",
+    "    return full_batch"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "a3311e1d-8d53-4a0f-bc5a-4197e316ae1d",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "torch.Size([2688, 1176])"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "collate_fn(messages[:1])['pixel_values'].shape"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6139f83c-d83c-4b4b-8e87-e28320799b5c",
+   "metadata": {},
+   "source": [
+    "## Fine-Tune the Model\n",
+    "\n",
+    "We will use the `SFTTrainer` from the `trl` library, which abstracts a lot of the complexities of training for us. For your own use-case you should adjust various hyper-parameters like learning rate, epochs, etc. according to your problem."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "8df597dd-7eac-4331-ad2c-3fc6800ea4e3",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from trl import SFTConfig, SFTTrainer\n",
+    "\n",
+    "# Configure training arguments\n",
+    "training_args = SFTConfig(\n",
+    "    output_dir=\"test_finetune\",  # Directory to save the model\n",
+    "    num_train_epochs=5,  # Number of training epochs\n",
+    "    per_device_train_batch_size=1,  # Batch size for training\n",
+    "    per_device_eval_batch_size=1,  # Batch size for evaluation\n",
+    "    gradient_accumulation_steps=4,  # Steps to accumulate gradients\n",
+    "    learning_rate=1e-5,  # Learning rate for training\n",
+    "    lr_scheduler_type=\"constant\",  # Type of learning rate scheduler\n",
+    "    logging_steps=1,  # Steps interval for logging\n",
+    "    eval_steps=2,  # Steps interval for evaluation\n",
+    "    eval_strategy=\"steps\",  # Strategy for evaluation\n",
+    "    # save_strategy=\"steps\",  # Strategy for saving the model\n",
+    "    # save_steps=20,  # Steps interval for saving\n",
+    "    bf16=True,  # Use bfloat16 precision\n",
+    "    max_grad_norm=0.3,  # Maximum norm for gradient clipping\n",
+    "    warmup_ratio=0.03,  # Ratio of total steps for warmup\n",
+    "    report_to=\"none\",  # Reporting tool for tracking metrics\n",
+    "    gradient_checkpointing=True,  # Enable gradient checkpointing for memory efficiency\n",
+    "    gradient_checkpointing_kwargs={\"use_reentrant\": False},  # Options for gradient checkpointing\n",
+    "    # max_seq_length=1024  # Maximum sequence length for input\n",
+    ")\n",
+    "\n",
+    "# allow for proper loading of images during collation\n",
+    "training_args.remove_unused_columns = False\n",
+    "training_args.dataset_kwargs = {\"skip_prepare_dataset\": True}\n",
+    "\n",
+    "trainer = SFTTrainer(\n",
+    "    model=model,\n",
+    "    args=training_args,\n",
+    "    data_collator=collate_fn,\n",
+    "    train_dataset=messages,\n",
+    "    eval_dataset=val_messages,\n",
+    "    processing_class=processor.tokenizer,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "0d05bdce-dc6e-4979-a573-8aed27972291",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "    <div>\n",
+       "      \n",
+       "      <progress value='10' max='10' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
+       "      [10/10 01:34, Epoch 5/5]\n",
+       "    </div>\n",
+       "    <table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       " <tr style=\"text-align: left;\">\n",
+       "      <th>Step</th>\n",
+       "      <th>Training Loss</th>\n",
+       "      <th>Validation Loss</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <td>2</td>\n",
+       "      <td>0.170400</td>\n",
+       "      <td>0.269275</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>4</td>\n",
+       "      <td>0.018300</td>\n",
+       "      <td>0.042456</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>6</td>\n",
+       "      <td>0.001800</td>\n",
+       "      <td>0.012193</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>8</td>\n",
+       "      <td>0.000300</td>\n",
+       "      <td>0.011337</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>10</td>\n",
+       "      <td>0.000300</td>\n",
+       "      <td>0.011667</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table><p>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/plain": [
+       "TrainOutput(global_step=10, training_loss=0.057332569236314156, metrics={'train_runtime': 96.9988, 'train_samples_per_second': 0.412, 'train_steps_per_second': 0.103, 'total_flos': 261136381470720.0, 'train_loss': 0.057332569236314156})"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "trainer.train()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "7c800e94-9494-4178-b987-d1eebbd80ec0",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "trainer.save_model(training_args.output_dir)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1c8cb789-9168-4b68-ae57-2d67ab104743",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "## Test Generation\n",
+    "\n",
+    "Now, let's run actual generation of outputs for our validation examples to see if what the model has learned."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "a66eb36d-8301-4e43-8bcd-8dc831a625b7",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# reload processor with left padding (for generation)\n",
+    "processor = AutoProcessor.from_pretrained(model_name, \n",
+    "                                          trust_remote_code=True, \n",
+    "                                          padding_side='left',\n",
+    "                                          use_fast=True)\n",
+    "\n",
+    "# reconstruct validation messages without labels\n",
+    "test_messages = [\n",
+    "    construct_messages(\n",
+    "        x[\"document\"], \n",
+    "        x[\"template\"], \n",
+    "    ) for x in val_inputs\n",
+    "]\n",
+    "\n",
+    "texts = processor.tokenizer.apply_chat_template(\n",
+    "    test_messages,\n",
+    "    tokenize=False,\n",
+    "    add_generation_prompt=True,\n",
+    ")\n",
+    "\n",
+    "image_inputs = process_vision_info(messages[2][:1])[0]\n",
+    "inputs = processor(\n",
+    "    text=texts,\n",
+    "    images=image_inputs,\n",
+    "    padding=True,\n",
+    "    return_tensors=\"pt\",\n",
+    ").to(\"cuda\")\n",
+    "\n",
+    "# we choose greedy sampling here, which works well for most information extraction tasks\n",
+    "generation_config = {\"do_sample\": True, \"temperature\": 1.0, \"max_new_tokens\": 2048}\n",
+    "\n",
+    "# Inference: Generation of the output\n",
+    "generated_ids = model.generate(\n",
+    "    **inputs,\n",
+    "    **generation_config\n",
+    ")\n",
+    "generated_ids_trimmed = [\n",
+    "    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)\n",
+    "]\n",
+    "output_texts = processor.batch_decode(\n",
+    "    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "14609ef3-7121-4956-b25e-6ded1a12bc70",
+   "metadata": {},
+   "source": [
+    "As can be seen below, the model is now generating extractions with lowercase strings by default."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "091058ee-347f-45dc-aed8-21693a818334",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "=== Prompt ===\n",
+      "<|im_start|>system\n",
+      "You are a helpful assistant.<|im_end|>\n",
+      "<|im_start|>user\n",
+      "# Template:\n",
+      "{\"names\": [\"verbatim-string\"]}\n",
+      "# Context:\n",
+      "Jack went to the hill with Jill. Rupert went to the diner.<|im_end|>\n",
+      "<|im_start|>assistant\n",
+      "\n",
+      "=== Output ===\n",
+      "{\"names\": [\"jack\", \"jill\", \"rupert\"]}\n",
+      "\n",
+      "=== Prompt ===\n",
+      "<|im_start|>system\n",
+      "You are a helpful assistant.<|im_end|>\n",
+      "<|im_start|>user\n",
+      "# Template:\n",
+      "{\"names\": [\"verbatim-string\"]}\n",
+      "# Context:\n",
+      "My dog Clifford likes to play fetch with Emily and Peter.<|im_end|>\n",
+      "<|im_start|>assistant\n",
+      "\n",
+      "=== Output ===\n",
+      "{\"names\": [\"clifford\", \"emily\", \"peter\"]}\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "for i in range(len(texts)):\n",
+    "    print(f\"=== Prompt ===\\n{texts[i]}\")\n",
+    "    print(f\"=== Output ===\\n{output_texts[i]}\\n\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "eb987296-d418-4cbb-b97c-37f4dd27632b",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "environment": {
+   "kernel": "conda-base-py",
+   "name": "workbench-notebooks.m127",
+   "type": "gcloud",
+   "uri": "us-docker.pkg.dev/deeplearning-platform-release/gcr.io/workbench-notebooks:m127"
+  },
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel) (Local)",
+   "language": "python",
+   "name": "conda-base-py"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.16"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

pyproject.toml

@@ -0,0 +1,23 @@
+[project]
+name = "numarkapp"
+version = "0.1.0"
+description = "Add your description here"
+readme = "README.md"
+requires-python = ">=3.12,<3.13"
+dependencies = [
+    "flash-attn",
+    "gradio",
+    "huggingface-hub",
+    "qwen-vl-utils>=0.0.14",
+    "torch==2.9.0",
+    "transformers>=5.14.1",
+]
+
+[tool.uv.sources]
+torch = { index = "pytorch-cu130" }
+flash-attn = { url = "https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.9.28/flash_attn-2.8.3+cu130torch2.9-cp312-cp312-win_amd64.whl" }
+
+[[tool.uv.index]]
+name = "pytorch-cu130"
+url = "https://download.pytorch.org/whl/cu130"
+explicit = true