source/primary/appendix/7_onpolicyresults.tex

\section{On-Policy RL Combination Results} \label{app:onpolicyresults}

To expand upon the results of \cref{sec:discobench_results}, in this section we provide a focused analysis of how changing the \textit{editable} modules affects the performance of agents. Specifically, in On-Policy RL, we sweep over all 15 possible module combinations with the with the same dataset splits as in \cref{sec:discobench_results} (i.e., meta-train is comprised of Breakout and Freeway, and meta-test of Asterix and Space Invaders, from \citet{young_minatar_2019}). Due to the vast number of domains supported in DiscoGen, such analysis is unaffordable across all possible domains. 
Before more in-depth analysis, we  examine the performance of each LLM on the different On-Policy RL module combinations in Table \ref{tab:elos_onpolicy}.

\begin{table}[h]
\centering
\caption{ADA performance over all On-Policy RL module combinations (ELO Scores with 95\% CIs).}
\label{tab:elos_onpolicy}
\setlength{\tabcolsep}{5pt}
\begin{tabular}{@{}lccc@{}}
\toprule
& \multicolumn{3}{c}{\textit{On-Policy RL Experiments}} \\
\cmidrule(lr){2-4} 
\textbf{Model} & \textbf{Succ.} & \textbf{Meta-Train} & \textbf{Meta-Test} \\
\midrule
Baseline      & --- & 1182 \tiny{[1107, 1260]} & \textbf{1267} \tiny{[1185, 1350]} \\
Deepseek-v3.2 & \textbf{41.1\%} & \textbf{1192} \tiny{[1111, 1281]} & 1218 \tiny{[1138, 1311]} \\
GPT-OSS 120B  & 20.0\% & 762 \tiny{[672, 837]} & 631 \tiny{[467, 742]} \\
Devstral2     & 18.9\% & 865 \tiny{[756, 961]} & 883 \tiny{[777, 978]} \\
\bottomrule
\end{tabular}
\end{table}

Interestingly, we find that Deepseek-v3.2 performs \textit{within confidence} of the `\textit{all fixed}' baseline, both in meta-train and in meta-test, over the different On-Policy RL tasks. Additionally, we consider how the performance of each agent changes with respect to the number of editable modules. Firstly, we examine how the success rate of each agent changes as we change the number of modules in \cref{tab:change_correlation}.

\begin{table}[h]
    \centering
    \caption{Success rate by number of editable modules in On-Policy RL.}\label{tab:change_correlation}

    % We define a new column type 'Y' that is Centered and expands equally
    \newcolumntype{Y}{>{\centering\arraybackslash}X} 
    
    % Set table width to roughly fit the text (e.g., 0.7 or 0.8 of text width)
    \begin{tabularx}{0.5\textwidth}{l *{4}{Y}}
        \toprule
        & \multicolumn{4}{c}{\textbf{Success Rate (\%) by Editable Modules}} \\
        \cmidrule(lr){2-5}
        \textbf{Model} & \textbf{1} & \textbf{2} & \textbf{3} & \textbf{4} \\
        \midrule
        Deepseek-v3.2 & 75.0 & 47.2 & 8.3 & 0.0 \\
        GPT-OSS-120b  & 50.0 & 11.1 & 8.3 & 0.0 \\
        Devstral2     & 29.2 & 27.8 & 0.0 & 0.0 \\
        \bottomrule
    \end{tabularx}
\end{table}
There is a consistent trend: adding more \textit{editable} modules, even when meta-train and meta-test datasets are kept the same, leads to reduced success rates. Clearly, the tasks that include more editable modules are more difficult, providing direct evidence that changing modules leads to semantically different tasks. To investigate this further, we consider how success rates compare for each module combination when aggregated over all language models in Table \ref{tab:successconfig}. Furthermore, we show the \textit{maximum} score achieved by any LLM in each environment, to explore if there are any signs that different module combinations can lead agents to produce better or worse scores.

\begin{table}[h]
    \centering
    \caption{Average success rate and maximum per-environment performance by module configuration. Any environment for which no ADA implemented a successful algorithm is marked with ---.}\label{tab:successconfig}
    \newcolumntype{Y}{>{\centering\arraybackslash}X} 
    
    \begin{tabularx}{\textwidth}{l *{5}{Y}}
        \toprule
         & \textbf{Success} & \multicolumn{2}{c}{\textbf{Meta-Train (Eval Return)}} & \multicolumn{2}{c}{\textbf{Meta-Test (Eval Return)}} \\
        \cmidrule(lr){3-4} \cmidrule(lr){5-6}
        \textbf{Configuration} & \textbf{Rate (\%)} & \textbf{Breakout} & \textbf{Freeway} & \textbf{Asterix} & \textbf{SpaceInv} \\
        \midrule
        
        \multicolumn{6}{c}{\textit{1 Editable Module}} \\
        \cmidrule(lr){1-6}
        Optimiser & 83.33 & 74.97 & 62.86 & 18.11 & 181.25 \\
        Loss & 77.78 & 83.91 & 62.58 & 39.60 & 179.50 \\
        Network & 33.33 & 99.97 & 68.07 & 14.20 & 189.75 \\
        Train & 11.11 & 8.41 & 8.41 & 3.73 & 177.12 \\
        \addlinespace
        
        \multicolumn{6}{c}{\textit{2 Editable Module}} \\
        \cmidrule(lr){1-6}
        Loss + Optimiser & 61.11 & 84.47 & 62.89 & 20.50 & 181.38 \\
        Network + Optimiser & 44.44 & 91.44 & 65.25 & 19.77 & 191.62 \\
        Loss + Network & 38.89 & 106.12 & 66.16 & 69.42 & 184.00 \\
        Loss + Train & 22.22 & 8.56 & 61.91 & 3.91 & 169.38 \\
        Optimiser + Train & 5.56 & 34.61 & 29.45 & --- & --- \\
        Network + Train & 0.00 & --- & --- & --- & --- \\
        \addlinespace
        
        \multicolumn{6}{c}{\textit{3 Editable Module}} \\
        \cmidrule(lr){1-6}
        Loss + Network + Optimiser & 11.11 & 88.58 & 65.20 & 64.85 & 186.75 \\
        Loss + Optimiser + Train & 11.11 & 0.30 & 2.95 & --- & --- \\
        Loss + Network + Train & 0.00 & --- & --- & --- & --- \\
        Network + Optimiser + Train & 0.00 & --- & --- & --- & --- \\
        \addlinespace
        
        \multicolumn{6}{c}{\textit{4 Editable Module}} \\
        \cmidrule(lr){1-6}
        Loss + Network + Optimiser + Train & 0.00 & --- & --- & --- & --- \\
        \bottomrule
    \end{tabularx}
\end{table}

Table \ref{tab:successconfig} demonstrates a clear pattern in success rates; developing optimisation algorithms is easier than loss functions, which are easier to implement than networks, and developing correct training loop logic is most difficult. This translates into the combinations with more editable modules, where implementing a loss and optimiser has a higher success rate than networks and train, for example. Interestingly, it also appears that the maximum score for \textit{every} environment was achieved in a setting with 2 editable modules. While it is impossible to confirm whether such a trend could continue into 3 and 4 editable modules, due to low success rates, there are signs that tasks with more editable modules \textit{can} exhibit higher performance due to greater ADA flexibility, emphasising further the semantic diversity of tasks. In fact, for \textit{every} individual module, a higher max score was attained by combining it with another module.

Our analysis demonstrates the importance and validity of introducing more diversity by expanding module combinations. In addition to a general pattern of increasing complexity through expanding the editable module count, we show how each module and module combination performs slightly differently, revealing a hierarchy of difficulty. This diversity of difficulty also demonstrates the natural application of autocurriculum for DiscoGen (Section \ref{sec:new_research}).