source/primary/appendix/9_implementation_details.tex

\section{DiscoGen Details} \label{app:implementation}


\subsection{Technical DiscoGen Implementation}\label{app:technical_implementation}


Here, we clarify the implementation details of DiscoGen; in particular, how tasks are generated.

DiscoGen operates by creating \textit{file systems}. To sample a random task from DiscoGen, a random DiscoGen configuration must be created. Doing so involves selecting a task domains, which defines the availability of modules, datasets and backends, and randomising each of these categories.

After a configuration has been created, DiscoGen is queried to build the meta-train portion of the task. To do so, DiscoGen creates: (1) all necessary files for \textit{each} meta-training dataset, including downloading and caching any data; (2) a script for running an inner-loop on all meta-training datasets; (3) a directory which includes all discovered algorithms; and (4) a procedurally generated \texttt{description} file which describes the task domain, the meta-training datasets, the backend and the editable modules based on the pre-written per-domain text in \ref{app:task_information}. 

Any ADA operates in a meta-loop over the meta-train files to develop new algorithms. When the meta-loop is complete, DiscoGen is queried again to build the meta-\textit{test} task. As this is created, \textbf{all} files besides the editable modules in \texttt{discovered/} are overwritten, and rebuilt from scratch for different datasets. This is done to lower the risk of evaluation hacking. Since the meta-test datasets are not known to the agent, we also dramatically reduce the risk of train-test leakage.



\subsection{Deriving Task Counts} \label{app:derivation}

For $m$ different modules, $d$ different datasets, and $b$ different backends in a task domain, we derive the number of valid tasks below.

Each dataset can be marked as part of the \textit{meta-train} set, \textit{meta-test} set, or \textit{excluded} set, given 3 possible options per dataset. The same dataset can not be included more than once, to prevent leakage between the meta-train and meta-test sets (i.e., $\mathcal{D}_{train} \cap \mathcal{D}_{test} = \varnothing$. Therefore, the number of possible combinations of datasets is $3^d$. However, we require \textit{at least} one dataset to be in the meta-train, and \textit{at least} one dataset in meta-test. We remove the two $2^d$ cases where this is not true (i.e., where either meta-train or meta-test are excluded as options), but add back the double-counted case of \texttt{all exclude}. This produces
$(3^d-2\times2^{d}+1) = (3^d-2^{(d+1)}+1)$ valid dataset configurations.

For $m$ modules, each modules can be marked as \textit{editable} or \textit{fixed}, meaning each module has $2$ possible states. This means there are $2^m$ possible module combinations. By removing the case where all modules are \textit{fixed}, this gives $2^m-1$ valid modules configurations.

We currently support $2$ types of initialisation (start-from-interface and start-from-baseline), and $3$ types of evaluation (performance, energy and time).

By combining the number of configurations for datasets and modules, and multiplying by the number of backends, initialisations and evaluation types, this gives
\begin{equation*}
    N_{tasks} = 2 \cdot 3 \cdot b \cdot (2^m-1)\cdot\left(3^d-2^{(d+1)}+1\right).
\end{equation*}

For Model Unlearning, where we can provide one of $n$ different base models for each dataset, the computation changes slightly. As opposed to counting $3$ options for every dataset there are $2n+1$ possible choices; any combination of meta-train/model ($n$), meta-test/model ($n$), or exclude ($1$). This provides $(2n+1)^d$ combinations. The degenerate cases now count as removing $(n+1)^d$ combinations each, meaning we remove $2(n+1)^d$ in place of the previous $2\times2^d$. This gives 
\begin{equation*}    
N_{tasks} = 2 \cdot 3 \cdot b \cdot (2^m-1)\cdot\left((2n+1)^d-2(n+1)^{d}+1\right).
\end{equation*}