gpt2-cbctg-prior

Concept-bottleneck controllable-generation modules for GPT-2 (124M), from the paper "The Illusion of Control: Why Bare Classifier Inversion Silently Fails in Concept-Bottleneck Text Generation" (EMNLP 2026).

Paper: arXiv:2608.22956

Primary backbone for Table 1 and for most appendix diagnostics (off-manifold distance, regulariser sweeps, the $k$-sample control, and the normalising-flow baseline).

This repository contains only our own trained parameters: the per-axis concept encoder MLPs, the per-axis classifier heads, the concept injector, the LoRA adapter, and the post-hoc label prior. No base-model weights are included; download gpt2 separately.

Contents

Each split directory holds one trained checkpoint.

Path	Description
hold-out/	Fyelp Hold-Out idx=-0 split (39 seen / 1 unseen configuration)
acd/	Fyelp ACD split (half of all configurations held out)

Within each split directory:

File	Description
config.json	Architecture and training configuration
full_model_best.pt	Concept encoder MLPs, classifier heads, injector, internal prior
best_lora_adapter.pt	LoRA adapter weights (rank 8, alpha 16); optimiser state stripped
label_prior.pt	Post-hoc label-conditioned prior $g_\gamma$ (the paper's recommended z-source)
retrieval_prior.pt	Nearest-seen-configuration retrieval baseline (Appendix L), where available
label_prior_flow.pt	Conditional normalising-flow baseline (Appendix K), where available
label_prior_factorized.pt	Factorised per-axis prior (Appendix N), where available
clsinv_reg_*.pt	Manifold-regulariser statistics for the inversion sweeps (Appendix I), where available

Architecture

• Concept code: 4 axes (cuisine, gender, sentiment, tense), 32 dimensions per axis
• Injector: AdaLN-zero, injected at every block
• Generator: GPT-2 (124M) with LoRA (rank 8, alpha 16) on the attention projections
• Prior $g_\gamma$: one hidden layer, 128 GELU units, fitted post-hoc in under 30 seconds

Usage

Clone the code repository and point the generation script at a downloaded checkpoint directory:

git clone https://github.com/BiancaBing/cbctg-illusion-of-control
cd cbctg-illusion-of-control

huggingface-cli download kabing/gpt2-cbctg-prior --local-dir ckpt/gpt2-cbctg-prior

python evaluation/generate_v3.py \
    --checkpoint ckpt/gpt2-cbctg-prior/hold-out/full_model_best.pt \
    --data_dir   data/fyelp_hold-out_idx0_4ax \
    --output_dir out/ \
    --model_name gpt2-cbctg-prior_prior \
    --z_source prior \
    --prior_path ckpt/gpt2-cbctg-prior/hold-out/label_prior.pt

Swap --z_source prior for oracle (reference-text encoding) or mode_b (classifier inversion) to reproduce the other two protocols compared in the paper. Full reproduction instructions are in the code repository.

Licence

GPT-2 is released under the MIT License.

Our released parameters (encoder MLPs, classifier heads, injector, LoRA adapter, label prior) are made available for research use under the same terms as the corresponding base model.

Citation

@inproceedings{bing2026illusion,
  title     = {The Illusion of Control: Why Bare Classifier Inversion
               Silently Fails in Concept-Bottleneck Text Generation},
  author    = {Bing, Qi and Shao, Xiaowei},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in
               Natural Language Processing (EMNLP)},
  year      = {2026},
  eprint    = {2608.22956},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url       = {https://arxiv.org/abs/2608.22956}
}