arxiv:2611.06976

On the Opportunities and Challenges of Prefill-to-Autoregressive Cross-Model Transfer

Published on Nov 4

· Submitted by

Zhenyi Shen on Nov 6

King's College London

Upvote

Authors:

Abstract

Prefill-to-Autoregressive Cross-model Transfer (PACT), a form of latent communication between language models, combines a larger model's prompt representations with a smaller model's efficient generation, offering a promising way to balance capability and inference cost. However, existing systems provide limited support for independently evaluating large-small model collaboration, and improved task performance alone does not establish whether the receiver benefits from the sender's information. We introduce PLatour, an open-source PACT framework that connects frozen models through lightweight, zero-initialized linear bridges trained directly with task rewards and supports hybrid-attention architectures. PLatour significantly outperforms open-source baselines on generation tasks and surpasses Mostik's reported results on shared mathematics and coding benchmarks using a substantially smaller sender. However, controlled experiments show that the gains on some tasks can be reproduced by uninformative sender signals, self-communication, or receiver-only training. We find that trainable bridges can act as parameter-efficient adaptation mechanisms without exploiting useful sender information. We therefore propose PACT-Check to quantify how much of a trained PACT system's gain genuinely comes from communication. Finally, we discuss the opportunities and challenges of PACT systems. Code is available at https://github.com/zhenyi4/platour.

View arXiv page View PDF GitHub 9 Add to collection

Community

zen-E

Paper submitter about 3 hours ago

How much can a small model benefit from a larger model that only processes the prompt—and what actually drives the improvement?

We explore these questions in this new paper.

In prefill-to-autoregressive cross-model transfer (PACT), a larger sender processes the prompt and passes internal representations to a smaller receiver, which then generates the response independently.

We introduce PLatour, a simple and scalable framework that connects frozen models through lightweight linear bridges trained directly with reinforcement learning. It also supports hybrid-attention architectures.

On generation tasks, PLatour substantially outperforms the evaluated open-source PACT baselines. Using a 35B sender, it also exceeds Mostik AI’s reported results on shared math and coding benchmarks, compared with Mostik's 753B sender—although the training setups may differ.

But stronger benchmark results do not, by themselves, establish useful information transfer.

On some tasks, replacing sender representations with question-independent signals or training the receiver alone reproduces much of the improvement. Trainable bridges can provide substantial adaptation capacity even when both model backbones are frozen.

The evidence is task-dependent: controls nearly match PLatour on GSM8K, while PLatour retains gains beyond all tested controls on Chemistry with chain-of-thought generation.

To examine this systematically, we introduce PACT-Check. It combines interventions on sender information with separately trained adaptation baselines. PACT-Score measures the accuracy margin over the strongest applicable control.

Our takeaway: PACT shows promising performance, but understanding its benefits requires distinguishing useful sender information from gains achievable through adaptation.

We hope PLatour and PACT-Check help the community investigate when prefill-only communication helps—and how to make those benefits more reliable.