soodashima91/meeting-summarization-domain-balancing

Meeting Summarisation — Domain Balancing

QLoRA adapters for multi-domain meeting summarisation, from the paper Token Distribution versus Data Volume: Domain Balancing in Multi-Domain Meeting Summarisation (INLG 2026).

This repository collects the adapters from the study in one place. Each adapter lives in its own subfolder and is loaded with the subfolder= argument (see Usage).

TL;DR

Fine-tuning one model jointly on meeting corpora of wildly different size, we separate the effect of how tokens are distributed across domains from the effect of how much data is seen, by comparing a balanced (equal-token) and a natural (size-proportional) mixture at matched token budgets. Balancing redistributes quality: it raises the data-scarce minority domains (AMI, ICSI, ELITR) at a small cost to the data-rich ones (EuroParlMin, MeetingBank), rather than adding quality uniformly.

What's in here

Subfolder	Base	Scheme	Budget
mistral-7b/balanced-32m	Mistral-7B-Instruct-v0.3	balanced (equal-token)	32M
mistral-7b/natural-32m	Mistral-7B-Instruct-v0.3	natural (proportional)	32M
mistral-7b/balanced-2m	Mistral-7B-Instruct-v0.3	balanced (equal-token)	2M
mistral-7b/natural-2m	Mistral-7B-Instruct-v0.3	natural (proportional)	2M
llama-3.2-3b/balanced-32m	Llama-3.2-3B-Instruct	balanced (equal-token)	32M
llama-3.2-3b/natural-32m	Llama-3.2-3B-Instruct	natural (proportional)	32M

mistral-7b/balanced-32m is the primary model; it is the system used for the paper's fact-level human-validation study. The Llama-3.2-3B adapters are the cross-family scaling control (RQ5) — they load a different base model, so match the base to the subfolder.

Headline result (32M, pruned, seed 42)

Balanced vs. natural allocation per domain, at the matched 32M budget:

Scheme	AMI	ICSI	ELITR	EPM	MB	Macro	Micro
Balanced ROUGE-Lsum	0.495	0.445	0.367	0.547	0.648	0.500	0.615
Natural ROUGE-Lsum	0.393	0.281	0.204	0.534	0.694	0.421	0.637
Balanced BERTScore-F1	0.875	0.852	0.846	0.887	0.928	0.877	0.915
Natural BERTScore-F1	0.865	0.833	0.837	0.889	0.938	0.872	0.923

Macro weights the five domains equally; micro weights meetings. Balanced leads all three minority domains and the macro average; natural leads the majority domains and the micro average. Full budget ladder, judge scores, seeds, and CIs are in the paper.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

REPO = "soodashima91/meeting-summarization-domain-balancing"
SUB  = "mistral-7b/balanced-32m"        # pick a subfolder from the table
BASE = "mistralai/Mistral-7B-Instruct-v0.3"   # use the base that matches SUB

bnb = BitsAndBytesConfig(
    load_in_4bit=True, bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16,
)

tok = AutoTokenizer.from_pretrained(REPO, subfolder=SUB)
model = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=bnb, device_map="auto")
model = PeftModel.from_pretrained(model, REPO, subfolder=SUB)
model.eval()

system = ("You are a meeting summarizer. Given a meeting transcript, write the meeting minutes "
          "that faithfully capture the substantive content of the meeting. Base the minutes only "
          "on what is stated in the transcript; do not introduce information that is not present.")
user = "Here is the meeting transcript. Write the meeting minutes.\n\n" + transcript

msgs = [{"role": "system", "content": system}, {"role": "user", "content": user}]
inputs = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=2048, do_sample=False)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

To load a Llama variant, set SUB = "llama-3.2-3b/balanced-32m" and BASE = "meta-llama/Llama-3.2-3B-Instruct".

Training configuration

QLoRA: 4-bit NF4, double quant, bf16 compute. LoRA r=32, α=16, dropout=0.05, applied to q/k/v/o/gate/up/down. Paged AdamW-8bit, lr 1e-4 cosine (3% warmup), weight decay 0.01, max grad norm 0.3, effective batch size 16, max sequence length 16,384. Up to 5 epochs with patience-2 early stopping on dev loss; each adapter is the lowest-dev-loss checkpoint. Transcripts are pruned of conversational filler before training; test transcripts are never pruned.

License

The Mistral-7B adapters are released under Apache-2.0 (the base model's licence). The Llama-3.2-3B adapters are derived from Llama-3.2-3B-Instruct and are subject to the Llama 3.2 Community License. Match the licence to the base model of the subfolder you use.

Citation

The official INLG 2026 proceedings citation is not yet available. In the meantime, please cite the arXiv preprint:

@article{sood2026token,
  title   = {Token Distribution versus Data Volume: Domain Balancing in Multi-Domain Meeting Summarisation},
  author  = {Sood, Ashima and Gardiner, Bryan and Condell, Joan},
  journal = {arXiv preprint arXiv:2608.15935},
  year    = {2026}
}