Yewei-Liu/SHINE-Pretrain

SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass

SHINE (Scalable Hyper In-context NEtwork) is a scalable hypernetwork that can map diverse meaningful contexts into high-quality LoRA adapters for large language models (LLM).

By reusing the frozen LLM's own parameters in an in-context hypernetwork design, SHINE transforms in-context knowledge into in-parameter knowledge in a single forward pass. This allows the model to handle complex question-answering tasks related to a specific context without needing to process that context again during inference.

• Paper: SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass
• Repository: https://github.com/Yewei-Liu/SHINE

Introduction

SHINE overcomes key limitations of prior hypernetworks by achieving strong expressive power with a relatively small number of parameters. It updates LLM parameters without any fine-tuning, significantly saving time, computation, and memory costs compared to standard supervised fine-tuning (SFT) adaptation.

Usage

This is the hypernetwork checkpoint after pretraining.

For detailed instructions on environment setup, downloading model checkpoints, and performing inference (including the inference.ipynb notebook), please refer to the official GitHub repository.