base_model:
- meta-llama/Llama-3.1-8B
datasets:
- allenai/winogrande
- allenai/ai2_arc
- google/boolq
- wentingzhao/obqa
license: llama3.1
tags:
- peft
- bayesian
pipeline_tag: text-generation
library_name: transformers
This repository contains a low-rank adapter model, based on Meta-Llama-3.1-8B, which was presented in the paper Training-Free Bayesianization for Low-Rank Adapters of Large Language Models.
Training-Free Bayesianization (TFB) is a simple yet theoretically grounded framework that efficiently transforms trained low-rank adapters into Bayesian ones without additional training. TFB systematically searches for the maximally acceptable level of variance in the weight posterior, constrained within a family of low-rank isotropic Gaussian distributions. This approach aims to achieve superior uncertainty estimation and generalization compared to existing methods, while eliminating the need for complex Bayesianization training procedures.
For the code, installation instructions, and further details on how to use the TFB framework, please refer to the official GitHub repository: https://github.com/Wang-ML-Lab/bayesian-peft