\documentclass[11pt]{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{microtype}
\usepackage{amsmath,amssymb}
\usepackage{booktabs}
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{xcolor}
\usepackage[margin=1in]{geometry}
\usepackage{authblk}
\usepackage{natbib}
\usepackage{hyperref}
\hypersetup{colorlinks=true,linkcolor=blue,citecolor=blue,urlcolor=blue}

\title{\textbf{Distinguishing Language Difficulty from Mathematical\\ Difficulty in Children's Word Problems}}
\author{CosmicMicra}
\affil{\textit{Bilingual Adaptive Math Project}\\\texttt{https://github.com/CosmicMicra/bilingual-math-tutor}}
\date{\today}

\begin{document}
\maketitle

\begin{abstract}
A bilingual child who fails a math word problem may understand the mathematics
perfectly and be stuck only on the English. Standard adaptive systems collapse
this distinction: a wrong answer lowers difficulty, which is exactly the wrong
response when the mathematics was never the obstacle. Building on the
three-component framework of \citet{daroczy2015}, who decompose word-problem
difficulty into linguistic complexity, numerical complexity, and their
interaction, we ask an empirical question: \emph{are the linguistic and
mathematical difficulty of a children's word problem genuinely separable
signals, or two faces of one underlying difficulty?} We extract an interpretable
twelve-feature linguistic block (readability formulas, CEFR-graded vocabulary
with protected mathematics terms, and dependency-parse syntax) and a
mathematical block (solution-step count, operation count and type, and number
magnitude) for 1{,}218 grade-labeled ASDiv problems and 700 SVAMP problems. Three
results support separability. (i) In a five-fold cross-validated regression
predicting a problem's grade level, the linguistic block alone reaches
$R^2{=}0.18$ and the mathematical block $R^2{=}0.31$; combining them reaches
$R^2{=}0.45$ --- a $+0.14$ lift, indicating each block carries signal the other
lacks. (ii) Variance partitioning attributes $0.223$ of explained variance
uniquely to mathematical features, $0.072$ uniquely to linguistic features, and
only $0.039$ to their shared component, so the two blocks explain mostly
non-overlapping variance. (iii) A single-dimensional difficulty index from each
block correlates at only Spearman $\rho{=}0.12$, i.e.\ the two axes are
near-orthogonal; and on SVAMP, holding the mathematics constant, linguistic
features still vary substantially. We argue these findings justify treating
language and mathematics as separate dials in adaptive instruction and
assessment for bilingual learners, and we release all code, features, and a
reproducible pipeline.
\end{abstract}

\section{Introduction}
Word problems are the point where mathematics meets language. To solve one, a
child must first read a short text, build a mental model of the situation it
describes, and only then carry out the arithmetic \citep{kintsch1985,cummins1988}.
Failure can therefore originate in two very different places: the
\emph{mathematics} (the child cannot perform the required operations) or the
\emph{language} (the child cannot decode the text well enough to recover the
mathematical structure). For bilingual learners and English-language learners
(ELLs) this distinction is not academic. \citet{abedi2001} showed that
\emph{linguistically modifying} mathematics test items---simplifying the
non-mathematical language without changing the mathematics---raises the scores of
ELL and low-income students, establishing unnecessary language load as a source
of \emph{construct-irrelevant variance}: difficulty that the test did not intend
to measure. \citet{martiniello2008} confirmed this at the item level, finding
that linguistic complexity produces differential item functioning (DIF) against
ELLs even after controlling for mathematical ability.

The practical consequence motivates this work. An adaptive tutor that lowers
mathematical difficulty whenever a bilingual student struggles will, for a
student whose mathematics is fine but whose English is not, repeatedly make the
mathematics easier while never addressing the real barrier. The correct response
is to vary the \emph{language}, not the mathematics. This is only possible if the
system can read the two difficulties as separate quantities.

\citet{daroczy2015}, in their review of linguistic and numerical factors in word
problems, give the conceptual foundation: word-problem difficulty decomposes into
(i)~linguistic complexity of the text, (ii)~numerical complexity of the
arithmetic, and (iii)~the interaction between them. Our contribution is to test
this decomposition \emph{empirically and computationally} on children's word
problems, and to quantify how separable the two axes actually are. Concretely:

\begin{itemize}
  \item We define two interpretable, orthogonally-motivated feature blocks---a
  \textbf{linguistic} block and a \textbf{mathematical} block---each grounded in
  the factors named by \citet{daroczy2015}, \citet{abedi2001}, and
  \citet{martiniello2008}, and each computable from problem text alone
  (\S\ref{sec:method}).
  \item We show, via block-wise regression and \emph{variance partitioning} on
  grade-labeled problems, that the two blocks explain mostly non-overlapping
  variance in problem difficulty (\S\ref{sec:results}, E1, E4).
  \item We show the two difficulty axes are near-orthogonal (Spearman
  $\rho{=}0.12$) and, using the SVAMP linguistic-perturbation paradigm
  \citep{patel2021}, that language varies even when the mathematics is held fixed
  (E2, E3).
  \item We release the full pipeline, features, and results for reproduction.
\end{itemize}

We are explicit about scope: we measure the \emph{separability of difficulty
signals from text}, not student response data. The grade label of a problem is a
proxy for its overall (mathematical) difficulty; human solve-rate or
item-response-theory difficulty would be a stronger target and is the natural
next step (\S\ref{sec:limits}).

\section{Related Work}
\paragraph{Language as construct-irrelevant variance.}
The foundational result is \citet{abedi2001}: simplifying the non-mathematical
language of NAEP items raised ELL and low-SES scores without altering the
mathematics, framing excess language load as construct-irrelevant variance.
\citet{martiniello2008} operationalized this at the item level with DIF analysis
on a grade-4 state test, reporting that linguistic complexity predicts DIF
against ELLs ($\beta{=}0.141$, $p<.001$; model adj.\ $R^2{=}0.66$), and that
schematic (visual) representations attenuate the effect. Think-aloud studies in
that line identify the specific culprits: low-frequency and polysemous
vocabulary, long noun and prepositional phrases, multiclausal sentences with
embedded relative/adverbial clauses, and passive voice. \citet{shaftel2006} add a
crucial methodological caution---individual linguistic features are often
non-significant in isolation; the effect emerges only when features are
\emph{aggregated}---which directly motivates our use of a multi-feature block
rather than any single readability number.

\paragraph{Comprehension models and relational language.}
\citet{kintsch1985} and \citet{cummins1988} model word-problem solving as a
text$\rightarrow$situation-model translation, locating many errors before any
computation. \citet{hegarty1992} showed that \emph{inconsistent} relational
language (e.g.\ ``more than'' when subtraction is required) induces systematic
reversal errors via a keyword strategy---a linguistic difficulty with no
mathematical counterpart. \citet{daroczy2015} synthesize this literature into the
three-component framework we adopt.

\paragraph{Computational difficulty and the NLP thread.}
A recent NLP line operationalizes these ideas. \citet{patel2021} construct SVAMP
by applying controlled \emph{linguistic} perturbations (question sensitivity,
lexical and structural variation) to simple problems, exposing that solvers lean
on shallow lexical cues rather than mathematical structure---precisely a tool for
separating surface language from mathematics. Grade- and standards-annotated
corpora such as ASDiv \citep{miao2020} and EDUMATH/STEM \citep{christ2025} supply
the difficulty labels we use. On the language side, readability is measured with
classic formulas \citep{textstat} and, increasingly, with CEFR-graded vocabulary
\citep{arase2022}. We combine these threads: CEFR/readability features for
language, step/operation features for mathematics, evaluated against grade labels.

\section{Method}
\label{sec:method}
\subsection{Data}
We use two complementary corpora, both verified on the Hugging Face Hub.
\textbf{ASDiv} (\texttt{MU-NLPC/Calc-asdiv\_a}; 1{,}218 problems) provides an
integer \emph{grade} label (1--6) and a step-annotated solution
\texttt{chain}, giving both a difficulty target and a clean mathematical-feature
source. \textbf{SVAMP} (\texttt{ChilleD/SVAMP}; 700 problems) provides an
operation \texttt{Type} and an \texttt{Equation}, and is built specifically to
vary phrasing while controlling the mathematics, which we exploit for the
perturbation experiment. The ASDiv grade distribution is skewed toward grade~3
(631 problems) with thin grade-5/6 tails (21/11), so we report rank-based
statistics (Spearman $\rho$) alongside $R^2$.

\subsection{Linguistic feature block}
Each problem text yields twelve language features, grouped by the factors named
in the assessment literature:
\begin{itemize}
  \item \textbf{Readability formulas:} Flesch--Kincaid grade, Flesch Reading
  Ease, Dale--Chall \citep{textstat}.
  \item \textbf{Lexical / vocabulary:} number of words, words per sentence, mean
  word length, and three CEFR-graded statistics---mean and maximum CEFR word
  level and the fraction of words at B1 or above \citep{arase2022}. Crucially,
  Common-Core/NCTM mathematics terms (\emph{area}, \emph{ratio}, \emph{median},
  \ldots) are \emph{protected}: they are excluded from the CEFR statistics so
  that vocabulary difficulty never double-counts mathematical content.
  \item \textbf{Syntactic:} dependency-parse depth, mean dependency length, and
  clause count (counts of \texttt{ccomp}/\texttt{advcl}/\texttt{relcl}/
  \texttt{acl}/\texttt{xcomp} relations) from spaCy, targeting the multiclausal,
  embedded structures flagged by \citet{martiniello2008}.
\end{itemize}

\subsection{Mathematical feature block}
From the solution \texttt{chain} (ASDiv) or \texttt{Equation} (SVAMP) we extract:
number of solution steps, number of arithmetic operations, maximum number
magnitude, and indicators for fractions and decimals---the numerical factors of
\citet{daroczy2015}. These features deliberately do \emph{not} read the problem
prose, keeping the two blocks methodologically independent.

\subsection{Experiments}
\textbf{E1 (block-wise prediction).} We predict the ASDiv grade label from
each block alone and from their union, using a Random Forest (and a linear Ridge
variant) under five-fold cross-validation, reporting $R^2$.
\textbf{E2 (axis separability).} We reduce each block to a single
difficulty index (its first principal component, sign-aligned to grade) and
measure the Spearman correlation between the two indices; we also report each
feature's correlation with grade.
\textbf{E3 (perturbation).} On SVAMP we group problems by mathematical
signature (operation type $\times$ operation count) and measure the within-group
coefficient of variation (CoV) of linguistic vs.\ mathematical features: if
language is a separate dial, it should still vary within a fixed-mathematics
group.
\textbf{E4 (variance partitioning).} From the Ridge $R^2$ of the
language-only, math-only, and combined models we compute the variance uniquely
attributable to each block and the shared component, the standard commonality
analysis.

\section{Results}
\label{sec:results}

\begin{table}[t]
\centering
\caption{E1: five-fold cross-validated $R^2$ predicting ASDiv grade level.
Each block carries signal; combining them adds a clear lift, indicating
complementary (non-redundant) information.}
\label{tab:e1}
\begin{tabular}{lccc}
\toprule
Feature block & Random Forest $R^2$ & Ridge $R^2$ \\
\midrule
Language only      & $0.179 \pm 0.050$ & $0.112 \pm 0.038$ \\
Mathematics only   & $0.313 \pm 0.079$ & $0.263 \pm 0.055$ \\
Combined           & $\mathbf{0.452 \pm 0.059}$ & $\mathbf{0.335 \pm 0.026}$ \\
\bottomrule
\end{tabular}
\end{table}

\paragraph{The two blocks are complementary (E1).}
Table~\ref{tab:e1} and Figure~\ref{fig:r2} show that language alone predicts
grade ($R^2{=}0.18$) and mathematics alone predicts it better ($R^2{=}0.31$), as
expected since grade is a mathematics-anchored label. The key number is the
\emph{combination}: $R^2{=}0.45$, a $+0.14$ gain over mathematics alone. If
language were merely a noisy restatement of mathematical difficulty, adding it
could not improve prediction this much.

\paragraph{Mostly non-overlapping variance (E4).}
Commonality analysis on the Ridge models (Figure~\ref{fig:partition}) attributes
$0.223$ of explained variance \emph{uniquely} to the mathematical block, $0.072$
\emph{uniquely} to the linguistic block, and only $0.039$ to the shared
component. In relative terms, shared variance is roughly $12\%$ of the total
explained---the two blocks measure largely different things.

\begin{figure}[t]
\centering
\begin{subfigure}{0.62\textwidth}
  \includegraphics[width=\linewidth]{fig_block_r2.png}
  \caption{Block-wise predictive $R^2$ (E1).}
  \label{fig:r2}
\end{subfigure}\hfill
\begin{subfigure}{0.36\textwidth}
  \includegraphics[width=\linewidth]{fig_variance_partition.png}
  \caption{Variance partition (E4).}
  \label{fig:partition}
\end{subfigure}
\caption{Language and mathematics features contribute complementary,
largely non-overlapping information about a problem's grade-level difficulty.}
\end{figure}

\paragraph{Near-orthogonal difficulty axes (E2).}
The single-dimensional linguistic and mathematical difficulty indices correlate
at only Spearman $\rho{=}0.12$ ($p{=}2.5\!\times\!10^{-5}$): the correlation is
statistically detectable but practically negligible
(Figure~\ref{fig:scatter}). Each axis correlates with grade on its own
(language $\rho{=}0.27$, mathematics $\rho{=}0.14$). Per-feature
(Figure~\ref{fig:featcorr}), the strongest \emph{linguistic} predictors of grade
are words-per-sentence ($\rho{=}0.27$), mean dependency length ($\rho{=}0.26$),
clause count ($\rho{=}0.26$), and parse depth ($\rho{=}0.22$)---exactly the
syntactic-complexity factors \citet{martiniello2008} identified---while the
strongest \emph{mathematical} predictor is the presence of fractions
($\rho{=}0.41$).

\begin{figure}[t]
\centering
\begin{subfigure}{0.46\textwidth}
  \includegraphics[width=\linewidth]{fig_axes_scatter.png}
  \caption{The two difficulty indices (E2).}
  \label{fig:scatter}
\end{subfigure}\hfill
\begin{subfigure}{0.52\textwidth}
  \includegraphics[width=\linewidth]{fig_feature_corr.png}
  \caption{Per-feature correlation with grade.}
  \label{fig:featcorr}
\end{subfigure}
\caption{Left: the linguistic and mathematical difficulty axes are
near-orthogonal. Right: syntactic-complexity features drive the language axis;
fraction/decimal presence drives the mathematics axis.}
\end{figure}

\paragraph{Language varies when mathematics is fixed (E3).}
On SVAMP, grouping by mathematical signature and measuring within-group
variability, linguistic features retain a mean coefficient of variation of
$0.33$---substantial phrasing variation persists even when the underlying
operation and operation count are held constant. (Mathematical features show
higher residual CoV, $1.42$, because number \emph{magnitudes} still differ within
a signature; the relevant point is that language is far from constant once the
mathematics is pinned.) This is the perturbation-style evidence that language is
an independently movable dial \citep{patel2021}.

\section{Discussion}
Across four analyses the same conclusion recurs: in children's word problems,
linguistic difficulty and mathematical difficulty are \emph{separable} signals.
They are not independent in the strict statistical sense---a weak positive
coupling ($\rho{=}0.12$) is expected, since higher grades tend to bring both
harder mathematics and denser prose---but the coupling is small enough that
treating them as one dial discards most of the linguistic signal. The
$+0.14$ predictive lift (E1) and the $0.072$ of \emph{uniquely} linguistic
variance (E4) are precisely the information an adaptive system throws away when it
responds to every wrong answer by lowering mathematical difficulty.

\paragraph{Implications for bilingual instruction.}
This is the empirical justification for a two-dial design. If a tutor can
estimate a problem's linguistic load separately from its mathematical load, it
can respond to a struggling bilingual student by first \emph{reducing language
load while holding the mathematics constant}---e.g.\ simplifying syntax or
providing CEFR-appropriate vocabulary and bilingual glosses---rather than making
the mathematics easier. This is exactly the linguistic-modification intervention
\citet{abedi2001} showed to recover ELL performance, now usable as an online,
per-problem adaptation. The protected-mathematics-term design (vocabulary
substitution that never touches \emph{ratio} or \emph{median}) ensures such
adaptation changes language without changing mathematical meaning.

\paragraph{Limitations and next steps.}
\label{sec:limits}
First, our difficulty target is a problem's \emph{grade label}, a coarse,
mathematics-anchored proxy; the stronger target is human solve-rate or an
item-response-theory difficulty parameter estimated from student responses, which
would let us test separability against \emph{behavioral} difficulty directly.
Second, the ASDiv grade distribution is skewed toward grade~3, which depresses
absolute $R^2$; the relative, rank-based separability results are more robust to
this than the raw $R^2$. Third, our analysis is English-only. \citet{daroczy2015}
caution that linguistic complexity is language-specific---manipulations that
raise difficulty in English need not transfer to Spanish, where morphology rather
than word order carries much relational meaning. Extending the linguistic block
to Spanish (via a Spanish dependency parser and a Spanish CEFR resource) and
testing whether the \emph{same} problem has different linguistic difficulty in its
two languages is the central bilingual experiment this paper sets up. Finally, the
mathematical block reads gold solution annotations; predicting difficulty from the
problem text alone (without the equation) is a harder and more deployable variant.

\section{Conclusion}
We tested, empirically, whether the linguistic and mathematical difficulty of
children's word problems are separable, operationalizing the three-component
framework of \citet{daroczy2015} with interpretable language and mathematics
feature blocks over 1{,}218 grade-labeled ASDiv problems and 700 SVAMP problems.
Block-wise regression, variance partitioning, axis-correlation, and a
linguistic-perturbation test all converge: the two difficulties carry mostly
non-overlapping signal and correspond to near-orthogonal axes. For bilingual
mathematics instruction and assessment, this is the quantitative case for
treating language and mathematics as separate dials---and for responding to a
struggling bilingual learner by adjusting the language before touching the
mathematics.

\paragraph{Reproducibility.}
All feature-extraction code, the extracted feature matrices, the analysis
script, and the figures are released with this paper.

\bibliographystyle{plainnat}
\bibliography{refs}

\end{document}
