gemma-4-12b-it-structure-lora-GGUF

GGUF LoRA adapter for the document-structure adapter on google/gemma-4-12B-it: policy page images in, one nested JSON tree per document out. This repo holds only the adapter; load it on top of a Gemma 4 12B-it base GGUF with its multimodal projector. The PEFT version of the same weights, with the evaluation and extract.py, is at webAI-Official/gemma-4-12b-it-structure-lora-peft.

File	Format	Size
gemma-4-12b-it-structure-lora-f16.gguf	LoRA adapter, f16	~130 MB

Untested with llama.cpp vision. The adapter was trained and evaluated with vLLM only. The weights convert cleanly (all 640 LoRA tensors), but llama.cpp's Gemma 4 image path, its image token budget, and whether it reproduces the 560-token reading the adapter was trained on have not been checked. Use the PEFT adapter with vLLM for the evaluated behaviour.

Usage (llama.cpp)

hf download webAI-Official/gemma-4-12b-it-structure-lora-GGUF gemma-4-12b-it-structure-lora-f16.gguf --local-dir .
llama-server -m gemma-4-12b-it-Q8_0.gguf --mmproj mmproj-gemma-4-12b-it-f16.gguf --lora gemma-4-12b-it-structure-lora-f16.gguf

The client must do what extract.py (in the PEFT repo) does: render pages at 200 DPI, send each page as two overlapping full-width strips with the strips note and the open-path prompt, under system_prompt.txt, thinking off, greedy, up to 4,096 tokens, pages in order, replies merged with page_merge.py.

Serving recipe the adapter was trained on

Callers must follow this input recipe; the adapter was trained on it and nothing else.

1. Render each PDF page at 200 DPI.
2. Send each page as two full-width strips, top half then bottom half, each reaching 4% of the page height into the other (extract.py, page_strips).
3. Serve with an image budget of 560 tokens per image (--mm-processor-kwargs '{"max_soft_tokens": 560}', or "mm_processor_kwargs": {"max_soft_tokens": 560} per request). Do not use 1120: vLLM 0.28 garbles a 1,120-token Gemma 4 image (Transformers reads it fine), and the base model and the adapter both fall into repetition loops.
4. User turn: the two strips, then the text The page is shown as 2 overlapping strips, top to bottom., a blank line, and the page prompt (Open path: [] (this is the first page) + Extract this page. for page 1; the open path of the tree merged so far for later pages).
5. System prompt system_prompt.txt; thinking off (chat_template_kwargs: {"enable_thinking": false}); greedy (temperature: 0); max_tokens: 4096; no presence or repetition penalty (they corrupt verbatim text).
6. Pages of one document in order; parse each reply with page_merge.parse_page and attach it with page_merge.merge_lenient (extract.py does all of this).

Adapter details

• Type: LoRA, rank 16, alpha 32, dropout 0.05, on down_proj, gate_proj, k_proj, o_proj, q_proj, up_proj, v_proj of the language model; the image embedder is frozen
• The k_proj of the eight full-attention layers (5, 11, ..., 47) is left out: their single projection serves keys and values, and vLLM 0.28 would apply a LoRA there to the keys only
• Training: 4836 conversations (2665 extraction, 721 contrast, 478 retention text, 972 retention image), 3 epochs, lr 0.0002, cosine, 8 conversations per step, max length 8192; final train loss 0.1139, validation loss by epoch 0.03079, 0.03599, 0.01924
• Images: 560 tokens each; an extraction page as two strips
• Data: the 484-conversation mix of Ashish-Ranjan/Policy-Structure-Pages (its contrast and retention prompts answered again by Gemma 4 itself) plus pages typeset from public-domain eCFR regulations in sampled one- and two-column layouts, with exact page trees
• Recipe after Ashish-Ranjan/Qwen3.5-9B-Policy-Structure-LoRA (same task, prompts, page loop and merge code)

Evaluation (vLLM, PEFT weights): see the PEFT repo. Converted with llama.cpp convert_lora_to_gguf.py (commit 5cdadfc).

License and provenance

The task design, system prompt, prompts, page loop and page_merge.py come from Ashish-Ranjan/Qwen3.5-9B-Policy-Structure-LoRA (Apache-2.0; LICENSE-policy-structure). The eCFR pages are U.S. government text in the public domain. Gemma 4 is Apache-2.0.