README.md

metadata

language:
  - en
library_name: transformers
license: cc-by-4.0
pipeline_tag: question-answering

Model Card for Llama-3.1-8B-Instruct LoRA for Knowledge Incorporation

This model is a Low-Rank Adaptation (LoRA) of Llama-3.1-8B-Instruct, designed to enhance its question-answering capabilities by incorporating new knowledge, as described in the paper How Much Knowledge Can You Pack into a LoRA Adapter without Harming LLM?.

Model Details

• Developed by: Sergey Pletenev et al.
• Model type: LlamaForCausalLM with LoRA
• Language(s) (NLP): English
• License: CC-BY-4.0
• Finetuned from model: meta-llama/Meta-Llama-3.1-8B-Instruct

Model Sources

• Repository: https://github.com/memyprokotow/knowledge_lora
• Paper: How Much Knowledge Can You Pack into a LoRA Adapter without Harming LLM?
• Datasets:
  • Dbpedia dump
  • Precollected triples and questions
  • Questions with labelled knowledge categories

Uses

Direct Use

This model can be used for question-answering tasks, particularly those involving the new knowledge incorporated during fine-tuning. It is designed to be used with the base model meta-llama/Meta-Llama-3.1-8B-Instruct.

Downstream Use

This model can be further fine-tuned or used as a starting point for research on knowledge incorporation into LLMs.

Out-of-Scope Use

This model should not be used for generating harmful, biased, or misleading content. Its performance on general question-answering benchmarks might be impacted after fine-tuning with specific knowledge.

Bias, Risks, and Limitations

This model inherits the biases present in the base Llama-3.1-8B-Instruct model. Furthermore, the focused fine-tuning may introduce biases related to the new knowledge incorporated. The paper highlights potential performance decline on external question-answering benchmarks and a tendency to over-represent answers related to prominent entities in the training data.

Recommendations

Users should be aware of the potential biases and limitations of the model. Careful attention should be paid to the composition and balance of the training data to mitigate biases and preserve general question-answering capabilities.

How to Get Started with the Model

See the Github repository for detailed instructions on training and using the LoRA adapter with the base Llama model.

Training Details

Training Data

The model is fine-tuned on a dataset generated using the head-to-tail pipeline with DBpedia as the knowledge source. The data includes known facts, potentially known facts, and unknown facts categorized based on the base model's pre-training knowledge. See the "Data" section of the Github README for details.

Training Procedure

The model is trained using the LoRA technique. Refer to the lora_train_llama.py script in the Github repository for training parameters and instructions.

Evaluation

The paper evaluates the model's performance using a reliability score and investigates different knowledge integration scenarios. See the paper for detailed results and analysis.

Environmental Impact

The environmental impact information is not available in the original README. Users can estimate the carbon emissions using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

Citation

@misc{pletenev2025knowledgepackloraadapter,
      title={How Much Knowledge Can You Pack into a LoRA Adapter without Harming LLM?}, 
      author={Sergey Pletenev and Maria Marina and Daniil Moskovskiy and Vasily Konovalov and Pavel Braslavski and Alexander Panchenko and Mikhail Salnikov},
      year={2025},
      eprint={2502.14502},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.14502}, 
}