| ---
|
| license: mit
|
| base_model: gpt2
|
| tags:
|
| - controllable-text-generation
|
| - concept-bottleneck
|
| - compositional-generalization
|
| - lora
|
| library_name: peft
|
| ---
|
|
|
| # gpt2-cbctg-prior
|
|
|
| Concept-bottleneck controllable-generation modules for **GPT-2 (124M)**, from
|
| the paper "The Illusion of Control: Why Bare Classifier Inversion Silently Fails in Concept-Bottleneck Text Generation" (EMNLP 2026).
|
|
|
| Paper: [arXiv:2608.22956](https://arxiv.org/abs/2608.22956)
|
|
|
| Primary backbone for Table 1 and for most appendix diagnostics (off-manifold distance, regulariser sweeps, the $k$-sample control, and the normalising-flow baseline).
|
|
|
| This repository contains **only our own trained parameters**: the per-axis
|
| concept encoder MLPs, the per-axis classifier heads, the concept injector,
|
| the LoRA adapter, and the post-hoc label prior. No base-model weights are
|
| included; download **[gpt2](https://huggingface.co/gpt2)** separately.
|
|
|
| ## Contents
|
|
|
| Each split directory holds one trained checkpoint.
|
|
|
| | Path | Description |
|
| |---|---|
|
| | `hold-out/` | Fyelp Hold-Out idx=-0 split (39 seen / 1 unseen configuration) |
|
| | `acd/` | Fyelp ACD split (half of all configurations held out) |
|
|
|
| Within each split directory:
|
|
|
| | File | Description |
|
| |---|---|
|
| | `config.json` | Architecture and training configuration |
|
| | `full_model_best.pt` | Concept encoder MLPs, classifier heads, injector, internal prior |
|
| | `best_lora_adapter.pt` | LoRA adapter weights (rank 8, alpha 16); optimiser state stripped |
|
| | `label_prior.pt` | Post-hoc label-conditioned prior $g_\gamma$ (the paper's recommended z-source) |
|
| | `retrieval_prior.pt` | Nearest-seen-configuration retrieval baseline (Appendix L), where available |
|
| | `label_prior_flow.pt` | Conditional normalising-flow baseline (Appendix K), where available |
|
| | `label_prior_factorized.pt` | Factorised per-axis prior (Appendix N), where available |
|
| | `clsinv_reg_*.pt` | Manifold-regulariser statistics for the inversion sweeps (Appendix I), where available |
|
|
|
| ## Architecture
|
|
|
| - **Concept code**: 4 axes (cuisine, gender, sentiment, tense), 32 dimensions per axis
|
| - **Injector**: AdaLN-zero, injected at every block
|
| - **Generator**: GPT-2 (124M) with LoRA (rank 8, alpha 16) on the attention projections
|
| - **Prior** $g_\gamma$: one hidden layer, 128 GELU units, fitted post-hoc in under 30 seconds
|
|
|
| ## Usage
|
|
|
| Clone the code repository and point the generation script at a downloaded
|
| checkpoint directory:
|
|
|
| ```bash
|
| git clone https://github.com/BiancaBing/cbctg-illusion-of-control
|
| cd cbctg-illusion-of-control
|
|
|
| huggingface-cli download kabing/gpt2-cbctg-prior --local-dir ckpt/gpt2-cbctg-prior
|
|
|
| python evaluation/generate_v3.py \
|
| --checkpoint ckpt/gpt2-cbctg-prior/hold-out/full_model_best.pt \
|
| --data_dir data/fyelp_hold-out_idx0_4ax \
|
| --output_dir out/ \
|
| --model_name gpt2-cbctg-prior_prior \
|
| --z_source prior \
|
| --prior_path ckpt/gpt2-cbctg-prior/hold-out/label_prior.pt
|
| ```
|
|
|
| Swap `--z_source prior` for `oracle` (reference-text encoding) or `mode_b`
|
| (classifier inversion) to reproduce the other two protocols compared in the
|
| paper. Full reproduction instructions are in the code repository.
|
|
|
| ## Licence
|
|
|
| GPT-2 is released under the MIT License.
|
|
|
| Our released parameters (encoder MLPs, classifier heads, injector, LoRA
|
| adapter, label prior) are made available for research use under the same
|
| terms as the corresponding base model.
|
|
|
| ## Citation
|
|
|
| ```bibtex
|
| @inproceedings{bing2026illusion,
|
| title = {The Illusion of Control: Why Bare Classifier Inversion
|
| Silently Fails in Concept-Bottleneck Text Generation},
|
| author = {Bing, Qi and Shao, Xiaowei},
|
| booktitle = {Proceedings of the 2026 Conference on Empirical Methods in
|
| Natural Language Processing (EMNLP)},
|
| year = {2026},
|
| eprint = {2608.22956},
|
| archivePrefix = {arXiv},
|
| primaryClass = {cs.CL},
|
| url = {https://arxiv.org/abs/2608.22956}
|
| }
|
| ```
|
|
|
