| \section{Experiments} |
| \label{sec:experiments} |
|
|
| \subsection{Setup} |
| \begin{figure}[h] |
| \centering |
| \includegraphics[width=\linewidth]{figs/radar_32b.pdf} |
| \caption{The radar chart of long-context reasoning benchmarks for DS-R1-Distill-32B trained with varying length.} |
| \label{fig:length_main_radar} |
| \end{figure} |
|
|
| \definecolor{rowgray}{gray}{0.92} |
| \begin{table*}[t] |
| \centering |
| \small |
| \caption{\textbf{Performance Comparison on Long-Context and Reasoning Benchmarks.} |
| We compare base models against their variants optimized via Reinforcement Learning on our dataset (denoted as \textbf{+ Ours}). |
| The best results within each model family are highlighted in \textbf{bold}. |
| \textsc{LB-v2}: LongBench v2; \textsc{Brows}: BrowsCompLong; \textsc{GSM-Inf}: GSM-Infinite; \textsc{Oolong}: Oolong-Synth. |
| For model names, Qwen2.5-32B-Inst denotes Qwen2.5-32B-Instruct, and DS-R1-Distill refers to Deepseek-R1-Distill-Qwen.} |
| \label{tab:main_results} |
| |
| \setlength{\tabcolsep}{5pt} |
| \begin{tabular}{lcccccccc} |
| \toprule |
| \textbf{Model} & \textbf{\textsc{LB-v2}} & \textbf{\textsc{Loong}} & \textbf{\textsc{Brows}} & \textbf{\textsc{GSM-Inf}} & \textbf{\textsc{Oolong}} & \textbf{\textsc{Ruler}} & \textbf{\textsc{MRCR}} & \textbf{Avg.} \\ |
| \midrule |
| |
| \multicolumn{9}{l}{\textit{\textbf{Reference Flagships \& Baselines}}} \\ |
| \midrule |
| Gemini-3.0-pro & \textbf{69.38} & \textbf{65.43} & \textbf{88.07} & \textbf{87.06} & \textbf{78.41} & \textbf{83.01} & \textbf{75.30} & \textbf{78.09} \\ |
| Deepeek-v3.1 & 52.88 & 50.55 & 56.27 & 47.40 & 51.90 & 42.97 & 46.62 & 49.80 \\ |
| Qwen-Long-L1 & 43.74 & 44.68 & 69.93 & 9.00 & 27.37 & 47.70 & 27.70 & 38.59 \\ |
| |
| \midrule |
| \multicolumn{9}{l}{\textit{\textbf{Main Results: Effectiveness of Our Method}}} \\ |
| \midrule |
| |
| Qwen3-32B & 46.33 & 39.96 & 59.33 & 12.22 & 31.66 & 50.86 & 42.45 & 40.40 \\ |
| \rowcolor{rowgray} |
| Qwen3-32B \textbf{+ Ours} & \textbf{46.92} & \textbf{43.10} & \textbf{65.44} & \textbf{23.40} & \textbf{42.31} & \textbf{53.55} & \textbf{42.66} & \textbf{45.34} \\ |
| \addlinespace[4pt] |
| |
| Qwen2.5-32B-Inst & 38.17 & 33.22 & 52.56 & 9.60 & \textbf{41.03} & 45.25 & 33.19 & 36.15 \\ |
| \rowcolor{rowgray} |
| Qwen2.5-32B-Inst \textbf{+ Ours} & \textbf{43.54} & \textbf{38.18} & \textbf{60.55} & \textbf{13.00} & 35.50 & \textbf{62.65} & \textbf{33.19} & \textbf{40.94} \\ |
| \addlinespace[4pt] |
| |
| DS-R1-Distill-14B & 35.19 & 25.07 & 51.38 & 9.00 & 42.05 & 61.28 & 29.02 & 36.14 \\ |
| \rowcolor{rowgray} |
| DS-R1-Distill-14B \textbf{+ Ours} & \textbf{38.97} & \textbf{35.44} & \textbf{74.31} & \textbf{9.60} & \textbf{45.95} & \textbf{80.72} & \textbf{30.48} & \textbf{45.07} \\ |
| \addlinespace[4pt] |
| |
| DS-R1-Distill-32B & 42.35 & 38.17 & 64.22 & 6.60 & 39.43 & 58.11 & 31.94 & 40.12 \\ |
| \rowcolor{rowgray} |
| DS-R1-Distill-32B \textbf{+ Ours} & \textbf{45.73} & \textbf{45.30} & \textbf{74.31} & \textbf{14.80} & \textbf{51.41} & \textbf{66.41} & \textbf{40.57} & \textbf{48.36} \\ |
| |
| \bottomrule |
| \end{tabular} |
| \end{table*} |
|
|
| \begin{figure*}[!h] |
| \begin{subfigure}{0.49\textwidth} |
| \centering |
| \includegraphics[width=\linewidth]{figs/needle_haystack_heatmap_chinese_14b_baseline.pdf} |
| \subcaption{DS-R1-Distill-14B} |
| \end{subfigure} |
| \hfill |
| \begin{subfigure}{0.49\textwidth} |
| \centering |
| \includegraphics[width=\linewidth]{figs/needle_haystack_heatmap_chinese_14b_ours.pdf} |
| \subcaption{DS-R1-Distill-14B + Ours} |
| \end{subfigure} |
| \hfill |
| \begin{subfigure}{0.49\textwidth} |
| \centering |
| \includegraphics[width=\linewidth]{figs/needle_haystack_heatmap_chinese_32b_baseline.pdf} |
| \subcaption{DS-R1-Distill-32B} |
| \end{subfigure} |
| \hfill |
| \begin{subfigure}{0.49\textwidth} |
| \centering |
| \includegraphics[width=\linewidth]{figs/needle_haystack_heatmap_chinese_32b_ours.pdf} |
| \subcaption{DS-R1-Distill-32B + Ours} |
| \end{subfigure} |
| \caption{Needle in a Haystack retrieval across document depths. Our approach significantly enhances long-context robustness, boosting the 14B model's accuracy from 69.30\% to 91.20\% and the 32B model's accuracy from 87.95\% to 99.40\%, achieving near-perfect performance.} |
| \label{fig:retrival_exps} |
| \end{figure*} |
|
|
| \begin{figure*}[!h] |
| \begin{subfigure}{0.49\textwidth} |
| \centering |
| \includegraphics[width=\linewidth]{figs/structure_exps_blue.pdf} |
| \caption{Impact of semantics, delimiters, and noise.} |
| \end{subfigure} |
| \hfill |
| \begin{subfigure}{0.49\textwidth} |
| \centering |
| \includegraphics[width=\linewidth]{figs/train_step_reward.pdf} |
| \subcaption{RL reward curves: Convergence and stability analysis.} |
| \end{subfigure} |
| \caption{\textbf{Decomposition experiments for DS-R1-Distill-32B.} (a) While structure alone (``no semantics'') boosts baseline performance (+1.67\%), semantics remain essential for peak results. Removing delimiters or adding \textbf{noise} yields negligible drops, confirming the primacy of intrinsic structure. (b) Models with ``no semantics'' suffer premature convergence, whereas ``no visible delimiters'' settings recover from low initial rewards.} |
| \label{fig:ablation_study} |
| \label{fig:structure_exps} |
| \end{figure*} |
|
|
| \textbf{Benchmarks.} To evaluate OOD universality and scalability, we conduct evaluation across diverse long-context tasks. We utilize \textbf{LongBench-v2} \cite{bai2025longbench} and \textbf{Loong} \cite{wang2024leave} for holistic real-world assessment. Furthermore, we include specific diagnostic datasets: \textbf{Browscomplong} \cite{BrowseComplong} and \textbf{MRCR} \cite{MRCR} for retrieval; the \textbf{Ruler} \cite{hsieh2024ruler} subset for variable tracking; and \textbf{GSM-Infinite} \cite{zhou2025gsm} alongside \textbf{Oolong} \cite{bertsch2025oolong} for numerical reasoning. To assess out-of-distribution (OOD) generalization, we further evaluate on \textbf{LiveCodeBench}\cite{jain2024livecodebench}, \textbf{AIME 2025}\cite{balunovic_srimatharena_2025}, \textbf{GPQA-Diamond} \cite{rein2024gpqa}, and \textbf{MultiChallenge} \cite{deshpande2025multichallenge}. Details of benchmarks are provided in Appendix~\ref{app:benchmark_details}. |
|
|
| \textbf{Baselines.} We take four distinct backbones as our baselines: Qwen2.5-32B-Instruct \cite{qwen2.5}, Qwen3-32B \cite{qwen3technicalreport}, Deepseek-R1-Distill-Qwen-14B, and Deepseek-R1-Distill-Qwen-32B \cite{deepseekai2025deepseekr1incentivizingreasoningcapability}, and apply RL training on them using our TableLong data. Additionally, we include Gemini-3-Pro, Deepseek-v3.1 \cite{deepseekai2024deepseekv3technicalreport}and Qwen-Long-L1 \cite{wan2025qwenlong} as external baselines. Among them, Qwen-Long-L1, also based on Deepseek-R1-Distill-Qwen-32B and trained for long-context reasoning via RL, serves as a relevant reference for same-backbone performance. |
|
|
| \textbf{Training Details.} Experiments are conducted on 64 H20 GPUs using the Verl \cite{sheng2024hybridflow} framework based on the GRPO \cite{shao2024deepseekmath} algorithm. Details of training configurations are in Appendix~\ref{app:training_details}. |
|
|
| \subsection{Main Results} |
| In this subsection, we evaluate the effectiveness of our TableLong in enhancing the general long-context reasoning capabilities of LLMs across multiple OOD general long-context reasoning benchmarks using several open-source backbone models. |
|
|
| \emph{\textbf{(a) Scalable table data is effective for long-context reasoning.}} Specifically, as shown in Table~\ref{tab:main_results}, for Deepseek-R1-Distill-Qwen-32B, our TableLong achieves a remarkable improvement, with an average accuracy gain of 8.24\% across seven OOD benchmarks of long contexts. Similarly, for other backbones (Qwen3-32B, Qwen2.5-32B-Instruct, and Deepseek-R1-Distill-Qwen-14B), the improvements are 4.94\%, 4.79\%, and 8.93\%, respectively. Deepseek-R1-Distill-Qwen-32B with our TableLong clearly outperforms Qwen-Long-L1 and approaches the performance of Deepseek-v3.1. This demonstrates the robustness of our TableLong in enhancing long-context reasoning capabilities. |
|
|
| \emph{\textbf{(b) Length Scalability: Train short, generalize long.}} Table data is inherently scalable in length. To investigate the effect of length scalability of table data on long-context reasoning, we conduct RL training with input lengths of 4k, 8k, and 16k. The results show consistent performance improvements across multiple long-context benchmarks as the length of structured table data scales up. Specifically, for Deepseek-R1-Distill-Qwen-32B, we observe relative gains of 0.48\% and 1.39\% at 8k and 16k, respectively, compared with the 4k experiment in Figure~\ref{tab:length_results}. |
| The 14B results are similar, with details in Appendix~\ref{app:detailed_results}. |
|
|
| Moreover, we investigate the impact of length scaling on accuracy across different benchmark length ranges in Appendix~\ref{app:scalability_length_range}. The results show that TableLong, when trained on sequences within 16k, generalizes effectively to contexts beyond 16k and even up to 128k, with performance further improving as the training length scales. |
|
|
| \emph{\textbf{(c) Significantly improved long-context retrieval.}} We adopt the Needle-in-a-Haystack benchmark to evaluate the gains of our TableLong in long-context retrieval, which measures a model's ability to locate a ``needle'' embedded at different depths within long documents. |
|
|
| As shown in Figure~\ref{fig:retrival_exps}, compared with Deepseek-R1-Distill-Qwen-14B, our TableLong significantly improves long-context retrieval performance by 31.60\%, increasing the retrieval score from 69.30\% to 91.20\%. Moreover, compared with Deepseek-R1-Distill-Qwen-32B, the retrieval score increases by 13.02\%, from 87.95\% to 99.40\%, approaching near-perfect retrieval performance. These results indicate that structured table data, potentially benefiting from its periodic non-vanishing structural properties, can substantially enhance long-context retrieval for RL-based post-training. |
| \begin{table*}[t] |
| \centering |
| \caption{\textbf{Performance comparison across varying scales of table cells.} Bold values indicate the best performance within each backbone.} |
| \label{tab:cell_count_results} |
| \begin{tabular}{lcccccccc} |
| \toprule |
| \textbf{Model} & \textbf{\textsc{LB-v2}} & \textbf{\textsc{Loong}} & \textbf{\textsc{Brows}} & \textbf{\textsc{GSM-Inf}} & \textbf{\textsc{Oolong}} & \textbf{\textsc{Ruler}} & \textbf{\textsc{MRCR}} & \textbf{Avg.} \\ |
| \midrule |
| |
| DS-R1-Distill-32B & & & & & & & & \\ |
| \quad$|-$ 0$\sim$30 & 44.33 & 42.87 & 71.87 & \textbf{18.00} & 40.29 & 69.30 & 37.44 & 46.30 \\ |
| \quad$|-$ 0$\sim$100 & 45.33 & 43.46 & 74.31 & 10.00 & 44.10 & \textbf{71.82} & 36.46 & 46.50 \\ |
| \quad$|-$ 0$\sim$300+ & \textbf{45.73} & \textbf{45.30} & \textbf{74.31} & 14.80 & \textbf{51.41} & 66.41 & \textbf{40.57} & \textbf{48.36} \\ |
|
|
| \midrule |
| DS-R1-Distill-14B & & & & & & & & \\ |
| \quad$|-$ 0$\sim$30 & 36.98 & 35.60 & 72.17 & 6.40 & 41.70 & \textbf{82.16} & 27.63 & 43.23 \\ |
| \quad$|-$ 0$\sim$100 & 37.38 & \textbf{36.45} & \textbf{74.31} & 6.40 & 34.47 & 80.44 & 29.16 & 42.65 \\ |
| \quad$|-$ 0$\sim$300+ & \textbf{38.97} & 35.44 & \textbf{74.31} & \textbf{9.60} & \textbf{45.95} & 80.72 & \textbf{30.48} & \textbf{45.07} \\ |
| \bottomrule |
| \end{tabular} |
| \end{table*} |
|
|
| \begin{table*}[t] |
| \centering |
| \caption{\textbf{Performance comparison across varying scales of table count.} Bold values indicate the best performance within each backbone.} |
| \label{tab:table_count_results} |
| \begin{tabular}{lcccccccc} |
| \toprule |
| \textbf{Model} & \textbf{\textsc{LB-v2}} & \textbf{\textsc{Loong}} & \textbf{\textsc{Brows}} & \textbf{\textsc{GSM-Inf}} & \textbf{\textsc{Oolong}} & \textbf{\textsc{Ruler}} & \textbf{\textsc{MRCR}} & \textbf{Avg.} \\ |
| \midrule |
| |
| DS-R1-Distill-32B & & & & & & & & \\ |
| \quad$|-$ 1 & 44.93 & 43.65 & 73.39 & \textbf{16.80} & 50.78 & 62.97 & 34.10 & 46.66 \\ |
| \quad$|-$ 1$\sim$5 & 45.33 & 44.27 & 73.37 & 15.00 & 49.63 & 64.37 & 37.79 & 47.11 \\ |
| \quad$|-$ 1$\sim$30 & \textbf{45.73} & \textbf{45.30} & \textbf{74.31} & 14.80 & \textbf{51.41} & \textbf{66.41} & \textbf{40.57} & \textbf{48.36} \\ |
|
|
| \midrule |
| DS-R1-Distill-14B & & & & & & & & \\ |
| \quad$|-$ 1 & 34.39 & \textbf{37.05} & 71.25 & 6.20 & 42.09 & 77.03 & \textbf{31.59} & 42.80 \\ |
| \quad$|-$ 1$\sim$5 & \textbf{39.56} & 36.24 & 72.78 & 6.80 & 41.38 & 79.08 & 30.48 & 43.76 \\ |
| \quad$|-$ 1$\sim$30 & 38.97 & 35.44 & \textbf{74.31} & \textbf{9.60} & \textbf{45.95} & \textbf{80.72} & 30.48 & \textbf{45.07} \\ |
| |
| \bottomrule |
| \end{tabular} |
| \end{table*} |
| \subsection{Decomposing Tabular Capabilities for Long-Context Reasoning} |
| In this subsection, we further decompose the capabilities of tabular data from three perspectives, including structural properties, multi-hop reasoning, and grounding. We also systematically analyze their contributions to the long-context reasoning capabilities of LLMs, yielding several inspiring insights. |
|
|
| \subsubsection{\textbf{structural property}} |
| To investigate the effect of the periodic non-vanishing structural properties inherent to table data, we design and conduct experiments from two perspectives: the semantics of table cell contents and the visible delimiter structures of tables, yielding the following insights: |
| \begin{tcolorbox}[myfinding, title=Insight 1] |
| The inherent structure of table data serves as a foundation for long-range dependencies, while semantic content provides the reasoning signal for complex long-context reasoning. |
| \end{tcolorbox} |
| Specifically, we construct ``no semantic'' table data (Figure~\ref{fig:nosematic}) and ``no visible delimiters'' table data (Figure~\ref{fig:delimiters}), with details provided in Appendix~\ref{app:pipelinedetails}. |
|
|
| Figure~\ref{fig:structure_exps}(a) shows that, relative to the baseline, ``no semantic'' with simple instruction prompting achieves an average performance improvement of 1.67\%, indicating that the enhancement in long-context reasoning mainly stems from the table's inherent organizational structure, namely its periodic non-vanishing structural properties. |
|
|
| Relative to ``Ours'', the ``no semantic'' with only simple instruction prompting converges rapidly (within approximately 40 steps) in Figure~\ref{fig:structure_exps}(b), leading to an average performance drop of 6.57\%. This indicates that both the semantic content of the table and the complexity of the instructions are also crucial for long-context reasoning. |
| \begin{tcolorbox}[myfinding, title=Insight 2] |
| The visible delimiter structures do not affect the inherent structural properties of tables during RL training. |
| \end{tcolorbox} |
| Figure~\ref{fig:structure_exps} shows that the ``no visible delimiters'' setting does not exhibit a significant performance drop compared to ``Ours'', with a reduction of about 0.77\%, and exhibits a similar asymptotic performance. in the RL reward curve. This indicates that visible delimiter structures do not affect the intrinsic periodic non-vanishing structural properties of tables, which are the key factor. |
|
|
| Moreover, the lower initial reward and consistently inferior performance of ``no visible delimiters'' suggest that removing visible delimiters makes table data harder for LLMs to interpret during training. |
|
|
| We further randomly replace delimiters with noise text (illustrated in Figure~\ref{fig:noise}) and observe results that are largely consistent with the ``no visible delimiters'' setting. This further demonstrates the robustness of table data for RL training. |
|
|
| \subsubsection{\textbf{multi-hop reasoning}} |
| \begin{tcolorbox}[myfinding, title=Insight 3] |
| |
| Table linearization induces multi-hop reasoning that enhances long-context reasoning, and increasingly multi-hop patterns further strengthen this effect. |
| \end{tcolorbox} |
| We characterize the ``multi-hop'' reasoning behavior of LLMs by the number of table cells involved in instruction operations, particularly along the column dimension. Specifically, when a two-dimensional table is linearized into a one-dimensional token sequence $\phi(\mathcal{T})$, cells within the same column become widely separated, forcing LLMs to attend to non-adjacent tokens and perform ``multi-hop'' reasoning. Representative examples are shown in Figure~\ref{fig:multi-hop reasoning case}, and further details are in Appendix~\ref{app:pipelinedetails}. |
|
|
| As shown in Table~\ref{tab:cell_count_results}, the model maintains strong long-context reasoning performance even with a small number of involved table cells (46.30\% on average for the 0$\sim$30 range). Moreover, scaling up the number of involved cells further improves performance on OOD long-context reasoning benchmarks. These findings suggest that the ``multi-hop'' property induced by table linearization effectively enhances long-context reasoning, and that increasing task complexity and multi-hop behavior can further strengthen this capability. |
|
|
| \subsubsection{\textbf{grounding}} |
| \begin{tcolorbox}[myfinding, title=Insight 4] |
| Tables in $\phi(\mathcal{T})$ provide ``grounding'', and scaling up quantity strengthens these signals, guiding attention to key tokens and improving long-context reasoning. |
| \end{tcolorbox} |
| Similarly, the number of tables in the prompt reflects the ``grounding'' capability of LLMs. Specifically, multi-table instruction operations compel the model to distinguish tables distributed across long contexts and to \textbf{associate} natural language queries with specific tables, making these tables act as sources of ``grounding''. |
|
|
| The results show that even a single table enables strong long-context reasoning in Table~\ref{tab:table_count_results}. As the number of tables scales up, the model achieves better performance on OOD long-context reasoning benchmarks, with gains of +0.45\% for 1$\sim$5 tables and +1.7\% for ``all'' tables compared to the single-table setting. This indicates that linearized tables in the token sequence $\phi(\mathcal{T})$ act as ``grounding'' elements that the model need to attend to. As their number scales up, the distributed grounding signals across $\phi(\mathcal{T})$ increase, forcing the model to learn to allocate attention to key tokens in the sequence, thereby further enhancing long-context reasoning capability. |
|
|
| \begin{table}[H] |
| \centering |
| \footnotesize |
| \setlength{\tabcolsep}{1.8pt} |
| \caption{\textbf{Generalization on OOD Benchmarks.} |
| We evaluate models on GPQA-Diamond (\textbf{GPQA}), AIME 2025 (\textbf{AIME}), MultiChallenge (\textbf{MC}), and LiveCodeBench (\textbf{LCB}). |
| \colorbox{rowgray}{Shaded rows} indicate our RL-finetuned models (\textbf{+ Ours}). |
| Our method achieves significant gains, notably \textbf{+11.9\%} on LCB for the 32B model.} |
| \label{tab:generalization} |
| |
| \begin{tabular}{lcccc} |
| \toprule |
| \textbf{Model} & \textbf{GPQA} & \textbf{AIME} & \textbf{MC} & \textbf{LCB} \\ |
| \midrule |
| |
| DS-R1-Distill-32B & 56.06 & 60.00 & 30.28 & 46.71 \\ |
| \rowcolor{rowgray} |
| DS-R1-Distill-32B \textbf{+ Ours} & \textbf{63.64} & \textbf{70.00} & \textbf{32.97} & \textbf{58.68} \\ |
| \addlinespace[4pt] |
| |
| DS-R1-Distill-14B & 55.56 & 43.33 & 23.08 & 43.71 \\ |
| \rowcolor{rowgray} |
| DS-R1-Distill-14B \textbf{+ Ours} & \textbf{59.60} & \textbf{46.67} & \textbf{26.86} & \textbf{45.51} \\ |
| |
| \bottomrule |
| \end{tabular} |
| \end{table} |
| \subsection{Generalization Results to Other Domains} |
| In this subsection, we evaluate the performance on out-of-domain (OOD) benchmarks from diverse fields, as shown in Table~\ref{tab:generalization}. Specifically, we consider GPQA-Diamond (science), AIME 2025 (math), MultiChallenge (multi-turn dialogue), and LiveCodeBench (code) to assess the generalization ability of TableLong across heterogeneous OOD domains. The results demonstrate that for Deepseek-R1-Distill-Qwen-32B, models trained with TableLong achieve substantial improvements of +7.58\%, +10.00\%, +2.69\%, and +11.97\% on these benchmarks, respectively. These significant gains indicate that TableLong strengthens their long-context retrieval and reasoning abilities, enabling effective generalization to a wide range of OOD domains. |
|
|
