README.md

---
license: apache-2.0
base_model: meta-llama/Llama-3.2-3B-Instruct
library_name: peft
pipeline_tag: text-generation
tags:
  - lora
  - peft
  - nlg-evaluation
  - semantic-fidelity
  - slot-error-rate
  - data-to-text
language:
  - en
---

# xdomain-ser-extractor

A LoRA adapter for Llama-3.2-3B-Instruct that extracts structured meaning
representations (slot-value pairs) from generated text, for measuring semantic
fidelity in meaning-to-text NLG. Part of the cross-domain slot-error-rate (SER)
evaluation framework described in our GEM 2026 paper.

## What it does

Given a natural-language realization and a domain hint map, the adapter extracts
the slot-value pairs expressed in the text. Comparing the extracted MR against a
gold MR yields per-example SER, slot F1, and substitution/deletion/insertion
counts. The framework reaches 86.8% SER agreement across 23 domains when
combined with over-generate-and-rank and NLI routing, without per-domain rules.

## Usage

This is a PEFT adapter. The base model must be obtained separately from Meta and
is governed by the Llama 3.2 Community License (see License below).

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")
model = PeftModel.from_pretrained(base, "DavanHarrison/xdomain-ser-extractor")
tokenizer = AutoTokenizer.from_pretrained("DavanHarrison/xdomain-ser-extractor")
```

For the full pipeline (extraction, over-generate-and-rank, NLI routing, CLI), see
the code repository: https://github.com/Vrindiesel/xdomain-ser

## Training

- Base model: meta-llama/Llama-3.2-3B-Instruct
- Method: 4-bit QLoRA supervised fine-tuning
- LoRA config: r=4, alpha=16, dropout=0.05, rank-stabilized (rsLoRA), EVA initialization
- Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
- Training data: multi_ser_v9, 20 topics x 200 examples (~4,000 examples)
- Trainable parameters: ~0.34% of the base model

## Intended use

Automatic slot-level semantic fidelity evaluation of data-to-text and
task-oriented dialogue generation. Research use. The adapter is English-only and
was trained on task-oriented domains; behavior on other languages or open-domain
text is not characterized.

## Limitations

Extraction accuracy degrades on meaning representations with many slots (7-8+).
The adapter has not been evaluated by human annotators on the extraction task
itself; reported numbers use a synthetic modified-MR evaluation protocol and a
cross-validation set of 1,000 gold-annotated examples. English-only.

## License

The adapter weights and accompanying code in this repository are released under
the Apache License 2.0, covering our contribution only.

Use of this adapter requires the base model, meta-llama/Llama-3.2-3B-Instruct,
which is distributed under the Llama 3.2 Community License Agreement. You must
obtain the base model from Meta and accept that license to use this adapter.
This release does not redistribute any Meta weights and does not grant any
rights to the base model.

## Citation

```bibtex
@inproceedings{harrison2026xdomainser,
  title     = {Cross-Domain Semantic Fidelity Evaluation for Meaning-to-Text NLG},
  author    = {Harrison, Davan and Walker, Marilyn},
  booktitle = {Proceedings of the GEM Workshop at ACL 2026},
  year      = {2026}
}
```