Inishds/Llama-3.2-1B-Instruct-KMerge-LoRA-library

Llama-3.2-1B-Instruct K-Merge LoRA library

Built with Llama

40 LoRA adapters for meta-llama/Llama-3.2-1B-Instruct, one for each of 5 tasks in 8 languages (en, de, es, fr, it, ja, ko, zh). This is the LoRA library used in the online continual merging experiments of K-Merge: Online Continual Merging of Adapters for On-device Large Language Models (Shenaj et al., ACL 2026), retrained with the authors' code from donaldssh/K-Merge.

This is an independent reproduction, not an official release by the paper's authors. 27 of the 40 adapters are within one point of the paper's test scores and 32 within two; content rephrasing and some text-correction adapters score higher than the paper, and Korean QA lower (see Evaluation).

Adapters

Each adapter is in its own folder, named <task>_attn_lora_l32_1b_tc_<lang>_v0 (content rephrasing: cr_attn_lora_l32_1b_tc_generic_<lang>_v0).

Folder prefix	Task	Training data
correction	Text (grammatical error) correction	en: Write & Improve (BEA-2019); de, es, fr, ja, ko: GitHub Typo Corpus; it: MERLIN (from the MultiGEC-2025 dataset); zh: ECSpell
qa	Extractive question answering	SQuAD v1.1
rp	Reply suggestion	Synthetic-Persona-Chat (consecutive turn pairs)
sum	Dialogue summarization	SAMSum
cr	Content rephrasing in 4 tones (professional, casual, witty, paraphrase), one adapter for all tones	inputs of facebook/content_rephrasing, outputs rewritten in each tone with llm-toys/RedPajama-INCITE-Base-3B-v1-paraphrase-tone

For every task except text correction, the non-English data is machine-translated from English: Helsinki-NLP/opus-mt-en-{de,es,fr,it} for de, es, fr and it, and facebook/m2m100_418M for ja, ko and zh.

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "meta-llama/Llama-3.2-1B-Instruct"
repo_id = "Inishds/Llama-3.2-1B-Instruct-KMerge-LoRA-library"

tokenizer = AutoTokenizer.from_pretrained(base_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(model, repo_id, subfolder="sum_attn_lora_l32_1b_tc_en_v0")

dialogue = "Anna: Are we still on for lunch?\nBen: Yes, 12:30 at the usual place.\nAnna: Great, see you there!"
messages = [{"role": "user", "content": "Summarize the following text:\n\n" + dialogue}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
output = model.generate(**inputs, max_new_tokens=64, do_sample=False, pad_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

More adapters can be added to the same model with model.load_adapter(repo_id, adapter_name=..., subfolder=...).

Prompt format

The adapters were trained with the Llama chat template, with a single user turn made of a task instruction in the target language, a blank line and the input. The English instructions are:

Task	Instruction
correction	Remove all grammatical errors from this text:
qa	Answer the following question: (input: question: <question> context: <context>)
rp	Suggest a reply for the following text:
sum	Summarize the following text:
cr	Changes a given user's input sentence or text to the '<Professional / Casual / Witty>' style: or Paraphrase the following text:

The instructions for the other languages are in prompts.py. For translated QA data, the question: and context: markers were translated along with the text.

Training details

• LoRA on the attention projections (q_proj, k_proj, v_proj, o_proj), r = 32, alpha = 128, dropout = 0.05.
• 1 epoch, learning rate 5e-5 with cosine schedule and 10 warmup steps, 8-bit AdamW, batch size 3 with gradient accumulation 64 (4 for content rephrasing and non-English text correction), gradient checkpointing, maximum source and target length 512 tokens.
• Trained with run_experiments.py and the arguments of scripts/train_loras_llama.sh from the K-Merge repository, on NVIDIA A100 80GB GPUs.

Evaluation

Test scores of each adapter on its own task, with the single-LoRA scores reported in the paper in brackets. Test splits follow the K-Merge code; the SQuAD and Synthetic-Persona-Chat test sets are capped at 1,000 examples.

Task (metric)	en	de	es	fr	it	ja	ko	zh
Text correction (F0.5)	35.1 (35.1)	25.9 (26.0)	33.3 (34.0)	24.5 (20.5)	31.6 (27.2)	12.3 (12.9)	10.8 (9.2)	36.1 (37.9)
Question answering (F1)	66.7 (63.5)	37.5 (36.5)	43.3 (42.3)	34.3 (34.7)	37.2 (37.1)	22.4 (22.5)	18.5 (28.2)	23.4 (22.6)
Reply suggestion (weighted ROUGE)	23.6 (23.0)	11.9 (10.7)	13.4 (13.2)	13.0 (12.2)	9.4 (8.8)	9.2 (9.1)	4.7 (5.0)	6.9 (7.5)
Summarization (ROUGE-L)	38.0 (38.2)	29.1 (28.6)	31.8 (31.3)	30.5 (30.2)	29.2 (28.3)	27.7 (27.5)	14.0 (14.0)	22.6 (22.7)
Content rephrasing (ROUGE-L)	57.8 (58.1)	44.9 (43.8)	48.5 (44.4)	47.2 (46.2)	44.8 (42.8)	42.5 (39.4)	33.4 (30.5)	41.1 (35.2)

The Korean QA adapter is the one clear gap: its exact match (10.3) is in line with Japanese and Chinese, but it gives longer answers than the references (5.9 vs 2.7 words on average), which lowers its F1.

Intended use and limitations

The adapters are meant for research on LoRA merging, continual merging and multilingual adaptation of small language models. They are small task-specific adapters on a 1B model, mostly trained on machine-translated data, and their outputs can be wrong or poorly phrased; they are not intended for production use.

License

The adapters are derivatives of Llama 3.2 and are distributed under the Llama 3.2 Community License, subject to the Acceptable Use Policy. Llama 3.2 is licensed under the Llama 3.2 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.

The training datasets have their own terms, and some of them are limited to non-commercial or research use, for example SAMSum (CC BY-NC-ND 4.0) and the Write & Improve and MERLIN/MultiGEC learner corpora. Check the terms of the relevant datasets before any commercial use.

Citation

If you use these adapters, please cite the K-Merge paper:

@inproceedings{shenaj2026k,
  title={K-merge: Online continual merging of adapters for on-device large language models},
  author={Shenaj, Donald and Bohdal, Ondrej and Ceritli, Taha and Ozay, Mete and Zanuttigh, Pietro and Michieli, Umberto},
  booktitle={Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  pages={3013--3029},
  year={2026}
}