Built with Llama

Llama-3.2-1B-cbctg-prior

Concept-bottleneck controllable-generation modules for Llama 3.2 1B, from the paper "The Illusion of Control: Why Bare Classifier Inversion Silently Fails in Concept-Bottleneck Text Generation" (EMNLP 2026).

Paper: arXiv:2608.22956

Largest main-table backbone; source of the cross-scale result discussed in Section 7.1 and of the activation over-shoot analysis in Section 5.2.

This repository contains only our own trained parameters: the per-axis concept encoder MLPs, the per-axis classifier heads, the concept injector, the LoRA adapter, and the post-hoc label prior. No base-model weights are included; download meta-llama/Llama-3.2-1B separately.

Contents

Each split directory holds one trained checkpoint.

Path	Description
hold-out/	Fyelp Hold-Out idx=-0 split (39 seen / 1 unseen configuration)
acd/	Fyelp ACD split (half of all configurations held out)

Within each split directory:

File	Description
config.json	Architecture and training configuration
full_model_best.pt	Concept encoder MLPs, classifier heads, injector, internal prior
best_lora_adapter.pt	LoRA adapter weights (rank 8, alpha 16); optimiser state stripped
label_prior.pt	Post-hoc label-conditioned prior $g_\gamma$ (the paper's recommended z-source)
retrieval_prior.pt	Nearest-seen-configuration retrieval baseline (Appendix L), where available
label_prior_flow.pt	Conditional normalising-flow baseline (Appendix K), where available
label_prior_factorized.pt	Factorised per-axis prior (Appendix N), where available
clsinv_reg_*.pt	Manifold-regulariser statistics for the inversion sweeps (Appendix I), where available

Architecture

• Concept code: 4 axes (cuisine, gender, sentiment, tense), 32 dimensions per axis
• Injector: AdaLN-zero, injected at every block
• Generator: Llama 3.2 1B with LoRA (rank 8, alpha 16) on the attention projections
• Prior $g_\gamma$: one hidden layer, 128 GELU units, fitted post-hoc in under 30 seconds

Usage

Clone the code repository and point the generation script at a downloaded checkpoint directory:

git clone https://github.com/BiancaBing/cbctg-illusion-of-control
cd cbctg-illusion-of-control

huggingface-cli download kabing/Llama-3.2-1B-cbctg-prior --local-dir ckpt/Llama-3.2-1B-cbctg-prior

python evaluation/generate_v3.py \
    --checkpoint ckpt/Llama-3.2-1B-cbctg-prior/hold-out/full_model_best.pt \
    --data_dir   data/fyelp_hold-out_idx0_4ax \
    --output_dir out/ \
    --model_name Llama-3.2-1B-cbctg-prior_prior \
    --z_source prior \
    --prior_path ckpt/Llama-3.2-1B-cbctg-prior/hold-out/label_prior.pt

Swap --z_source prior for oracle (reference-text encoding) or mode_b (classifier inversion) to reproduce the other two protocols compared in the paper. Full reproduction instructions are in the code repository.

Licence

The base model is governed by the Llama 3.2 Community License. These adapter weights are a derivative work and are distributed under the same terms. Meta's base weights are not redistributed here and must be obtained separately from the link above.

Our released parameters (encoder MLPs, classifier heads, injector, LoRA adapter, label prior) are made available for research use under the same terms as the corresponding base model.

Citation

@inproceedings{bing2026illusion,
  title     = {The Illusion of Control: Why Bare Classifier Inversion
               Silently Fails in Concept-Bottleneck Text Generation},
  author    = {Bing, Qi and Shao, Xiaowei},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in
               Natural Language Processing (EMNLP)},
  year      = {2026},
  eprint    = {2608.22956},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url       = {https://arxiv.org/abs/2608.22956}
}