Detecting Model Degeneration with Probe

Model degeneration is a failure mode of autoregressive language models in which generation collapses into repetitive, low-information output. The model gets trapped in a loop, producing the same phrase, sentence, or reasoning pattern over and over, while the text loses all semantic value. It does not stop generating; it simply stops communicating.

Autoregressive models generate text one token at a time: at each step the model looks at all previous tokens and picks the next one according to a probability distribution. When that distribution becomes too concentrated on a narrow set of tokens, often ones that are already present in context, the model enters a self-reinforcing cycle it cannot escape on its own.[2, 3]

Across the completions we collected, degeneration takes several recurring forms. The repeated unit may be a single token, a digit or a word emitted hundreds of times, a short phrase, a full sentence reproduced verbatim, a mathematical expression, or a structural marker such as a row of asterisks or hashtags. In each case the model stops advancing the task and instead recycles material already present in its own output.

The effect is strongly correlated with the difficulty of the prompt: simple questions are answered cleanly, whereas hard, multi-step reasoning, competition mathematics above all, degenerates far more often. Once a loop sets in, the model frequently, though not always, fails to recover a natural stopping signal and keeps emitting tokens at full confidence until the generation is cut off by the token limit, as illustrated by Apertus-8B-Instruct[1] in the example below.

The exchange below was generated by Apertus-8B-Instruct[1] on a one-line arithmetic question drawn from our balanced dataset. The model factorises the number correctly and even spots that 13 is prime, the answer is right there, but as it starts to list the factors it falls into a loop, repeating the digit "2" thousands of times and never delivering the answer. Text highlighted in red marks the degenerate region.

Model output: Apertus-8B-Instruct

The starting point is the dataset published on Hugging Face by Luca Sartori ( luca-sartori/degeneration-probe-instruct-token-level). It contains 83,350 completions generated by Apertus-8B-Instruct[1] across five public instruction datasets, one response per prompt, each annotated at the token level with a continuous repetition score.

A generation is labeled positive (degenerating) if it contains at least one token whose averaged repetition score is ≥ 0.8; otherwise it is labeled negative (clean). In the full 83k corpus, the vast majority of completions are clean: the natural positive-to-negative ratio is roughly 1 : 72. Training a probe on such an imbalanced set would bias the model towards always predicting the negative class, so we downsampled the negative examples.

Negative examples were randomly removed using a fixed seed (42) to keep the experiment reproducible. After a cleanup step that discarded rows with no scored tokens, the final ratio settled at approximately 1 : 1.6. The balanced dataset is published at lorenzo0312/degeneration-probe-instruct-token-level-balanced.

Every completion in the corpus was generated by Apertus-8B-Instruct[1]. Each token then receives a continuous repetition score between 0 and 1. A score of 0 means every bigram in the surrounding context is unique: healthy, diverse output. A score of 1 means all bigrams repeat: a fully degenerate loop. A token is flagged as degenerating when its score is ≥ 0.8.

The core idea is simple: rather than looking at a single token in isolation, we look at the neighbourhood around it. Concretely, we slide a window of 256 tokens along the generation and compute a bigram repetition score for each window position. A token then inherits the average score of all windows that contain it.

Tokens that fall outside any complete 256-token window (only possible in generations shorter than 256 tokens) receive repetition: null and are excluded from training.

A probe is a lightweight linear head attached to one internal layer of the language model. During a forward pass the prompt and response are fed through Apertus-8B-Instruct[1]; at a chosen layer X the probe intercepts the hidden state \(\mathbf{h}_t\) for every token t and maps it to a scalar score in [0, 1] via a linear projection followed by sigmoid. The backbone is kept frozen except for lightweight LoRA[4] adapters. LoRA (Low-Rank Adaptation) works by injecting two small trainable matrices alongside each frozen attention weight matrix W: a down-projection \(A \in \mathbb{R}^{d \times r}\) and an up-projection \(B \in \mathbb{R}^{r \times d}\), so the effective weight update is \(\Delta W = BA\) with rank \(r \ll d\) (typically \(r = 8\text{–}64\)). Only A and B are trained; W stays unchanged. This reduces trainable parameters by several orders of magnitude while still letting the representations adapt to the probe task.

The training signal is Mean Squared Error between the probe’s predicted score and the pre-computed repetition label \(y_t\) for each token:

Tokens with repetition: null are excluded. The regression formulation gives the probe a continuous gradient signal across the full score range rather than a hard binary boundary at 0.8. The pipeline is illustrated in the diagram below.

The probe attaches to a chosen layer, reads its 4,096-dimensional activations, and predicts a per-token repetition score ŷ[t]; the MSE loss against the deterministic target y[t] trains the probe weights W. Hover a layer to see how the activations and scores change with depth.