source/tex/src/methods.tex

\section{TableLong: a Scalable and Verifiable Pipeline for Long-context Reasoning}

In this section, we propose a simple yet scalable pipeline, namely TableLong, to construct high-quality, diverse, and verifiable structured table data, tailored for RL. 

\subsection{Overview of the Construction Pipeline}
As illustrated in Figure~\ref{fig:pipeline}, our framework achieves a pipeline to transform hybrid tabular sources into high-quality RL tasks. The pipeline proceeds in three stages:

\begin{itemize}
    \item \textbf{Environment Initialization:} We first aggregate a diverse repository of tables from both open-source datasets and documents, parsing them into executable SQL environments.
    
    \item \textbf{Sample Construction:} Leveraging these environments, we employ LLMs to generate diverse SQL queries and corresponding natural language questions, and subsequently execute the SQL queries to obtain verifiable answers. We construct each instance as a tuple: (Raw Table, Question, Answer).
    
    \item \textbf{Verification and Filtration:} We then apply a consistency-based filtration mechanism to prune noise ($P=0$) and triviality ($P=1$), ensuring the selected tasks offer optimal training value.
\end{itemize}

\begin{figure}[h]
    \centering
    \includegraphics[width=0.99\linewidth]{figs/table_overview.pdf}
    \caption{Overview of TableLong: An end-to-end table data construction pipeline for long-context reasoning.}
    \label{fig:pipeline}
\end{figure}

\subsection{Environment Setup}
To ensure the diversity and realism of the reasoning environment, we construct a hybrid repository by aggregating over 10,000 real-world tables from established datasets (e.g., BIRD \cite{li2023can}, CoSQL \cite{yu2019cosql}, Spider \cite{yu2018spider}) and extracting scalable table data from information-dense documents via LLMs.

Following rigorous cleaning and sampling, the resulting corpus exhibits high diversity in both content and structure. Thematically, the data spans a broad spectrum of domains, including Finance, Sports, Healthcare, and Science. Linguistically, it covers both English and Chinese. Crucially, the corpus is tailored for long-context reasoning: context lengths range from a few hundred up to 32k tokens, with an average cell density of 5.2 tokens. Furthermore, to support complex structural grounding, the number of tables per instance ranges from single to 30 tables, creating a challenging multi-table environment. Finally, all collected tables are parsed into a unified SQLite database to support executable SQL queries.
\subsection{Sample Construction}

We generate diverse SQL queries by scalable prompt constraints, targeting \textbf{three distinct dimensions} of table tasks for long-context reasoning:

\begin{itemize}
    \item \textbf{Precise Retrieval:} Targeting atomic information locating. We constrain the generation to \texttt{SELECT} queries with specific \texttt{WHERE} clauses, requiring the model to filter and extract precise values from contexts.
    
    \item \textbf{Multi-hop Reasoning:} Targeting long-range multi-hop reasoning. We constrain the generation to column-wise aggregation and calculation (e.g., \texttt{SUM}, \texttt{AVG}), requiring the model to attend to non-adjacent tokens separated by substantial distances due to table linearization.
    
    \item \textbf{Grounding:} Targeting query-to-table association. We constrain the generation to multi-table operations (e.g., \texttt{JOIN}), requiring the model to distinguish specific tables distributed across the long context as ``grounding''.
\end{itemize}

Upon generating diverse SQL queries and corresponding natural language questions, we execute the queries to obtain verifiable answers. Finally, we construct each training sample as a triplet: (Raw Table, Question, Answer).

\subsection{Verification and Filtration}
To ensure appropriate task difficulty, we implement a consistency-based filtration mechanism. For each instance $(x, y_{gt})$, we generate $N$ candidate responses using the model and evaluate their correctness against the ground truth $y_{gt}$. Based on the calculated pass rate ($P$), we apply a \textbf{dual-sided filter}:

\begin{itemize}
    \item \textbf{Discard $P=0$:} Eliminates tasks that are ambiguous, erroneous, or exceed reasoning capacity.
    \item \textbf{Discard $P=1$:} Prunes trivial tasks that the model has already mastered, ensuring high training efficiency.
    \item \textbf{Retain $0 < P < 1$:} Preserves non-trivial tasks within the effective learning boundary, offering optimal gradient signals for RL training.
\end{itemize}