File size: 1,883 Bytes
878cadc
5c5dd05
 
 
 
878cadc
5c5dd05
 
 
 
 
 
 
878cadc
 
5c5dd05
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
---
title: Long-Context Preference Optimization
emoji: 📖
colorFrom: indigo
colorTo: blue
sdk: static
app_file: index.html
short_description: Explore bandit chunk sampling for long-context DPO
tags:
  - long-context
  - preference-optimization
  - dpo
  - multi-armed-bandit
---

# Long-Context Preference Optimization

How can a model generate useful preference pairs when the evidence is scattered through a long document? This walkthrough follows chunk selection and shows what the LongMab ablations measured.

Open the App tab to explore the examples or reported measurements.

## Where do the bandit arms come from?

Split the source document into chunks. LongMab initializes chunk utility with a probe based on embedding similarity, then uses an upper confidence bound to balance promising chunks against less-tested ones. The language model generates candidate answers from selected chunks, and feedback from those answers updates the bandit scores.

## How much did the bandit contribute?

The same Llama-3.1-8B-Instruct backbone appears in the full and ablated runs. These are averages over five long-context question answering datasets in Table 2.

## Where does this method fit in a long-context pipeline?

The bandit runs when collecting candidate responses for preference training. Chosen and rejected responses then become DPO examples. In comparisons with inference-time retrieval or truncation, separate the training cost, the chunk budget during data construction and the evaluation-time context.

## Paper and citation

[Chunks as Arms: Multi-Armed Bandit-Guided Sampling for Long-Context LLM Preference Optimization](https://aclanthology.org/2026.acl-long.161/)  
Shaohua Duan, Pengcheng Huang, Xinze Li, and colleagues. ACL 2026.

[arXiv](https://arxiv.org/abs/2508.13993) · [Ablations in Tables 2 and 3](https://aclanthology.org/2026.acl-long.161.pdf#page=7)