---
license: mit
base_model: gpt2
tags:
  - controllable-text-generation
  - concept-bottleneck
  - compositional-generalization
  - lora
library_name: peft
---

# gpt2-cbctg-prior

Concept-bottleneck controllable-generation modules for **GPT-2 (124M)**, from
the paper "The Illusion of Control: Why Bare Classifier Inversion Silently Fails in Concept-Bottleneck Text Generation" (EMNLP 2026).

Paper: [arXiv:2608.22956](https://arxiv.org/abs/2608.22956)

Primary backbone for Table 1 and for most appendix diagnostics (off-manifold distance, regulariser sweeps, the $k$-sample control, and the normalising-flow baseline).

This repository contains **only our own trained parameters**: the per-axis
concept encoder MLPs, the per-axis classifier heads, the concept injector,
the LoRA adapter, and the post-hoc label prior. No base-model weights are
included; download **[gpt2](https://huggingface.co/gpt2)** separately.

## Contents

Each split directory holds one trained checkpoint.

| Path | Description |
|---|---|
| `hold-out/` | Fyelp Hold-Out idx=-0 split (39 seen / 1 unseen configuration) |
| `acd/` | Fyelp ACD split (half of all configurations held out) |

Within each split directory:

| File | Description |
|---|---|
| `config.json` | Architecture and training configuration |
| `full_model_best.pt` | Concept encoder MLPs, classifier heads, injector, internal prior |
| `best_lora_adapter.pt` | LoRA adapter weights (rank 8, alpha 16); optimiser state stripped |
| `label_prior.pt` | Post-hoc label-conditioned prior $g_\gamma$ (the paper's recommended z-source) |
| `retrieval_prior.pt` | Nearest-seen-configuration retrieval baseline (Appendix L), where available |
| `label_prior_flow.pt` | Conditional normalising-flow baseline (Appendix K), where available |
| `label_prior_factorized.pt` | Factorised per-axis prior (Appendix N), where available |
| `clsinv_reg_*.pt` | Manifold-regulariser statistics for the inversion sweeps (Appendix I), where available |

## Architecture

- **Concept code**: 4 axes (cuisine, gender, sentiment, tense), 32 dimensions per axis
- **Injector**: AdaLN-zero, injected at every block
- **Generator**: GPT-2 (124M) with LoRA (rank 8, alpha 16) on the attention projections
- **Prior** $g_\gamma$: one hidden layer, 128 GELU units, fitted post-hoc in under 30 seconds

## Usage

Clone the code repository and point the generation script at a downloaded
checkpoint directory:

```bash
git clone https://github.com/BiancaBing/cbctg-illusion-of-control
cd cbctg-illusion-of-control

huggingface-cli download kabing/gpt2-cbctg-prior --local-dir ckpt/gpt2-cbctg-prior

python evaluation/generate_v3.py \
    --checkpoint ckpt/gpt2-cbctg-prior/hold-out/full_model_best.pt \
    --data_dir   data/fyelp_hold-out_idx0_4ax \
    --output_dir out/ \
    --model_name gpt2-cbctg-prior_prior \
    --z_source prior \
    --prior_path ckpt/gpt2-cbctg-prior/hold-out/label_prior.pt
```

Swap `--z_source prior` for `oracle` (reference-text encoding) or `mode_b`
(classifier inversion) to reproduce the other two protocols compared in the
paper. Full reproduction instructions are in the code repository.

## Licence

GPT-2 is released under the MIT License.

Our released parameters (encoder MLPs, classifier heads, injector, LoRA
adapter, label prior) are made available for research use under the same
terms as the corresponding base model.

## Citation

```bibtex
@inproceedings{bing2026illusion,
  title     = {The Illusion of Control: Why Bare Classifier Inversion
               Silently Fails in Concept-Bottleneck Text Generation},
  author    = {Bing, Qi and Shao, Xiaowei},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in
               Natural Language Processing (EMNLP)},
  year      = {2026},
  eprint    = {2608.22956},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url       = {https://arxiv.org/abs/2608.22956}
}
```