Papers
arxiv:2610.17175

FunDM: Cheap & Label-Free Decision Models from Frozen Language Models

Published on Oct 12
Authors:

Abstract

Decision models such as Jev score candidate answers or actions. A small decision head trained label-free can recover about 99% of a frozen language model's multiple-choice accuracy while using only two-thirds of its transformer layers. It learns the full model's answer probabilities with KL divergence, without correct-answer labels for training. One backbone pass reads the question and all options together; the head returns probabilities for every option. We call this model FunDM (Fun Decision Model). Using 24 of Qwen3-8B's 36 layers and skipping the LM head, its +1T variant reaches 75.0% mean accuracy over eight benchmarks, versus 75.7% for the full model. On MMLU-Pro, both heads achieve 1.56 to 1.57 times batched vLLM throughput with about one-third lower peak GPU allocator memory. Both heads score higher than released Contrastive Language Models (CLM), fine-tuned CLM, and CLM-shaped heads trained from scratch on the eight multiple-choice benchmarks and coding-agent selection. Separately trained agent heads learn judge probabilities without pass/fail labels and select passing runs for 74.4 to 74.6 of 100 held-out tasks on average, versus 74 for the full-model judge and 52.0 for a CLM-shaped action-trained head. Code is available at: https://github.com/Ujjawal-K-Panchal/FunDM

Community

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.17175 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.17175 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.17175 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.