DistillLens-gpt2-base

paper | code

DistillLens-gpt2-base is a gpt2-base (120M) model distilled from gpt2-xlarge (1.5B) on databricks-dolly-15k.

Method

Note: DistillLens requires a GPT2-Base for initilization to perform the distillation.

Citation

@article{dhakal2026distilllens,
  title={DistillLens: Symmetric Knowledge Distillation Through Logit Lens},
  author={Dhakal, Manish and Jinadu, Uthman and Budathoki, Anjila and Sunderraman, Rajshekhar and Ding, Yi},
  journal={arXiv preprint arXiv:2602.13567},
  year={2026}
}