Llama2-13B-Syntax-Instruct

Llama-2-13b-chat fine-tuned to give students feedback on essay syntax

Reads an English student essay and returns structured feedback on its syntax across eight categories — spelling, conjunctions and connectives, modifiers, prepositions, modal verbs, punctuation, articles and pronouns — with examples from the essay and suggestions for each. A LoRA adapter for Llama-2-13b-chat trained on Essay-Syntax-Instruct.

From Advancing Student Writing Through Automated Syntax Feedback (AIEER 2024). Code: Essay-Syntax-Feedback.

Results

ROUGE against the reference feedback, base → fine-tuned:

Model	ROUGE-1	ROUGE-2	ROUGE-L
Llama-2-7b-chat	0.351 → 0.477	0.146 → 0.337	0.185 → 0.343
Llama-2-13b-chat	0.328 → 0.476	0.148 → 0.340	0.214 → 0.347
Mistral-7B-Instruct-v0.2	0.366 → 0.514	0.144 → 0.375	0.205 → 0.369

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-13b-chat-hf")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-13b-chat-hf", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "Kamyar-zeinalipour/Llama2-13B-Syntax-Instruct").merge_and_unload()

Training

LoRA r = 32, α = 64, dropout 0.1; 3 epochs; AdamW, learning rate 1e-4 with cosine decay; 8 sequences per GPU × 4 accumulation steps; up to 2,048 tokens; bf16, DeepSpeed. Script: scripts/.

Models from this paper

Base model	Adapter	ROUGE-L (base → fine-tuned)
Llama-2-7b-chat	Llama2-7B-Syntax-Instruct	0.185 → 0.343
Llama-2-13b-chat	Llama2-13B-Syntax-Instruct (this model)	0.214 → 0.347
Mistral-7B-Instruct-v0.2	Mistral7B-Syntax-Instruct	0.205 → 0.369

Citation

@inproceedings{zeinalipour2024advancing,
  title     = {Advancing Student Writing Through Automated Syntax Feedback},
  author    = {Kamyar Zeinalipour and Mehak Mehak and Fatemeh Parsamotamed and Marco Maggini and Marco Gori},
  booktitle = {AI in Education and Educational Research (AIEER 2024)},
  pages     = {52--66},
  year      = {2024},
  publisher = {Springer},
  doi       = {10.1007/978-3-031-93409-4_5},
  url       = {https://arxiv.org/abs/2501.07740}
}