license: mit
language:
- en
tags:
- peft
- low-rank-adaptation
- efficient-finetuning
- basis
BASIS: [这里填写你的论文全名或副标题]
Storage cleanup note (2026-04-14): this repository currently preserves the directory layout and model card only. Checkpoint artifacts were removed from the Hub to reclaim storage and can be re-uploaded later if needed.
📖 Model Overview
This repository hosts the trained checkpoints and projection matrices for BASIS, an efficient and structurally unified fine-tuning framework. BASIS optimizes low-rank adaptation by aligning the trainable subspace with the intrinsic data distribution.
These checkpoints represent various base models adapted using the BASIS methodology, achieving competitive performance under strict rank and parameter budgets.
🚀 Methodology Quick Glance
BASIS operates through a meticulously decoupled pipeline to maximize both local and global parameter efficiency:
- Calibration-Aware Basis Construction: We perform robust symmetric eigendecomposition on the empirical covariance of activations, scaling the base weights to construct a subspace that preserves the most critical directions.
- Cross-Layer Rank Allocation: Instead of uniform rank distribution, we dynamically allocate rank budgets across different layers. This is guided by Fisher information (for sensitivity) and managed via a streaming approach to ensure scalability for 70B+ models.
- In-Subspace Adaptation: The final fine-tuning occurs strictly within this optimized, calibration-aware subspace.
📂 Available Checkpoints
Please refer to the specific sub-directories or branches for different base models and rank configurations.
| Base Model | Rank Budget | Target Task/Domain | Download Link |
|---|---|---|---|
| [e.g., Llama-2-7b] | [e.g., R=32] | [e.g., Commonsense QA] | [Link] |
| [e.g., Mistral-7B] | [e.g., R=64] | [e.g., Math Reasoning] | [Link] |
💻 How to Use
To load and use these checkpoints for inference or further evaluation, you can use the following snippet:
# [TODO: 提供一段你们官方代码库中加载 BASIS 权重的 Python 伪代码或真实代码]
# Example:
# from basis import BASISModel
# model = BASISModel.from_pretrained("Zishan-Shao/BASIS", subfolder="llama-2-7b-r32")