Buckets:

MihirRajeshPanchal
/

Athena-Data

Files

xet

MihirRajeshPanchal/Athena-Data / data /rl_data_prep.tex

MihirRajeshPanchal

about 1 month ago

download

raw

7.66 kB

\documentclass[11pt]{article}
\usepackage[margin=1in]{geometry}
\usepackage{booktabs}
\usepackage{hyperref}
\usepackage{tabularx}
\usepackage{longtable}

\newcommand{\game}[1]{\path{#1}}
\setlength{\emergencystretch}{3em}

\title{RL Data Preparation Report}
\author{}
\date{\today}

\begin{document}
\maketitle

\section{Purpose}

This report summarizes the preparation of newly generated visual puzzle test
cases for reinforcement-learning data use. The objective was to create new
examples without reusing pre-existing reference test cases from the checked-in
datasets.

\section{Generation Method}

Two puzzle collections were prepared: AlgoPuzzleVQA and PuzzleVQA. For each
puzzle category, candidate examples were created using the collection's existing
generation procedure. Each candidate was compared with the corresponding
pre-existing reference dataset before being retained.

For AlgoPuzzleVQA, comparison was based on the complete underlying puzzle
solution state. For PuzzleVQA, comparison considered both puzzle metadata and
rendered image content, while ignoring presentation-only differences such as
image file names and answer-option ordering. Candidates matching a reference
example, or an example already accepted during the same run, were discarded.

Each retained example consists of a generated image and a JSONL record containing
the question, answer, answer choices where applicable, and supporting puzzle
metadata. The final outputs were checked to confirm that every record has a
corresponding image and that no retained case overlaps its reference dataset.

\section{Generated Test Cases}

\begin{table}[h]
\centering
\begin{tabular}{lrr}
\toprule
Collection & Puzzle Categories & Test Cases \\
\midrule
AlgoPuzzleVQA & 18 & 1,800 \\
PuzzleVQA & 16 & 1,600 \\
\midrule
Total & 34 & 3,400 \\
\bottomrule
\end{tabular}
\caption{Generated non-overlapping test cases. Each generated category contains
100 test cases.}
\end{table}

\section{Dataset Organization}

All generated datasets have been collected under the repository's
\texttt{dataset/} directory. The AlgoPuzzleVQA outputs are located in
\texttt{dataset/AlgoPuzzleVQA/data/}, while the PuzzleVQA outputs are located in
\texttt{dataset/PuzzleVQA/data/}. Each collection contains one JSONL file per
puzzle type and an \texttt{images/} directory containing a matching subdirectory
for that puzzle type. The \texttt{puzzle\_type} identifiers used by the ontology
files match these JSONL file and image-directory names.

\section{Ontology Files}

Two ontology files describe how the generated puzzle types are organized:
\path{algopuzzlevqa_ontology.json} and
\path{puzzlevqa_ontology.json}. They provide a compact, machine-readable
index for selecting subsets of games during training or evaluation.

\subsection{AlgoPuzzleVQA Ontology}

The AlgoPuzzleVQA ontology contains all 18 available puzzle types. Each entry
records its \texttt{puzzle\_type}, its required visual features, and its
algorithmic reasoning features. A puzzle may belong to several feature groups.
The visual features describe what must be interpreted from the image, whereas
the algorithmic features describe the reasoning needed to solve the puzzle.

Table~\ref{tab:algo-puzzles} lists each puzzle first, followed by its visual and
algorithmic categories and its generation status.

\small
\begin{longtable}{p{0.18\textwidth}p{0.18\textwidth}p{0.38\textwidth}p{0.12\textwidth}}
\caption{AlgoPuzzleVQA ontology and generation status.}\label{tab:algo-puzzles}\\
\toprule
Puzzle Name & Visual Categories & Algorithmic Categories & Generated \\
\midrule
\endfirsthead
\toprule
Puzzle Name & Visual Categories & Algorithmic Categories & Generated \\
\midrule
\endhead
\game{board_tile} & Colour, position & Arithmetic, boolean logic & Yes \\
\game{calendar} & Position, text & Arithmetic & Yes \\
\game{chain_link} & Position, shape and size & Arithmetic, optimization, sets & Yes \\
\game{checker_move} & Colour, position & Arithmetic, boolean logic, search & Yes \\
\game{clock} & Position, shape and size, text & Arithmetic & Yes \\
\game{colour_hue} & Colour, position & Arithmetic, optimization & Yes \\
\game{map} & Colour, position, shape and size, text & Arithmetic, boolean logic, combinatorics, graphs, search & Yes \\
\game{maze} & Colour, position & Arithmetic, graphs, optimization, search & Yes \\
\game{move_box} & Colour, position & Arithmetic, graphs, optimization, search & Yes \\
\game{n_queens} & Position & Arithmetic, boolean logic, search & Yes \\
\game{number_slide} & Position, text & Arithmetic, combinatorics, optimization, search, sets & Yes \\
\game{rotting_kiwi} & Position & Arithmetic, boolean logic, graphs, optimization, search & Yes \\
\game{rubiks_cube} & Colour, position & Arithmetic, graphs & Yes \\
\game{think_dot} & Colour, position, shape and size & Arithmetic, boolean logic, sets & Yes \\
\game{tower_of_hanoi} & Position, shape and size & Arithmetic, boolean logic, optimization, search & Yes \\
\game{water_jugs} & Position, shape and size, text & Arithmetic, boolean logic, optimization, search & Yes \\
\game{wheel_of_fortune} & Position, shape and size, text & Arithmetic & Yes \\
\game{wood_slide} & Position, shape and size & Arithmetic, optimization, search & Yes \\
\bottomrule
\end{longtable}
\normalsize

\subsection{PuzzleVQA Ontology}

The PuzzleVQA ontology organizes games using four visual concepts:
\emph{numbers}, \emph{colours}, \emph{size}, and \emph{shapes}. Each entry
contains a concept list, a derived category name, a concept count, and the
boolean field \texttt{has\_new\_generated\_data}. The boolean distinguishes
games included in the new dataset from ontology entries for which no new data
could be retained.

Table~\ref{tab:puzzlevqa-puzzles} lists all 20 ontology entries, including the
four puzzle types for which new data could not be generated.

\small
\begin{longtable}{p{0.32\textwidth}p{0.40\textwidth}p{0.16\textwidth}}
\caption{PuzzleVQA ontology and generation status.}\label{tab:puzzlevqa-puzzles}\\
\toprule
Puzzle Name & Category & New Data Generated \\
\midrule
\endfirsthead
\toprule
Puzzle Name & Category & New Data Generated \\
\midrule
\endhead
\game{grid_number} & Numbers & Yes \\
\game{triangle} & Numbers & Yes \\
\game{color_grid} & Colours & Yes \\
\game{color_hexagon} & Colours & Yes \\
\game{size_cycle} & Size & Yes \\
\game{size_grid} & Size & No \\
\game{shape_morph} & Shapes & No \\
\game{shape_reflect} & Shapes & No \\
\game{polygon_sides_number} & Numbers and shapes & Yes \\
\game{venn} & Numbers and shapes & Yes \\
\game{color_number_hexagon} & Numbers and colours & Yes \\
\game{grid_number_color} & Numbers and colours & Yes \\
\game{circle_size_number} & Numbers and size & Yes \\
\game{rectangle_height_number} & Numbers and size & Yes \\
\game{color_overlap_squares} & Shapes and colours & No \\
\game{polygon_sides_color} & Shapes and colours & Yes \\
\game{shape_size_grid} & Shapes and size & Yes \\
\game{shape_size_hexagon} & Shapes and size & Yes \\
\game{color_size_circle} & Colours and size & Yes \\
\game{rectangle_height_color} & Colours and size & Yes \\
\bottomrule
\end{longtable}
\normalsize

\section{Result}

The prepared dataset contains 3,400 new test cases across 34 puzzle categories.
The four excluded PuzzleVQA categories would require an approved expansion of
their original puzzle designs before 100 additional non-overlapping cases could
be generated for each category.

\end{document}

Xet Storage Details

Size:: 7.66 kB
Xet hash:: 363957b2909dbcae2868e58efbd0aa17d1ef379dc13d18644e6eb0c24d8ac246

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	\documentclass[11pt]{article}
	\usepackage[margin=1in]{geometry}
	\usepackage{booktabs}
	\usepackage{hyperref}
	\usepackage{tabularx}
	\usepackage{longtable}

	\newcommand{\game}[1]{\path{#1}}
	\setlength{\emergencystretch}{3em}

	\title{RL Data Preparation Report}
	\author{}
	\date{\today}

	\begin{document}
	\maketitle

	\section{Purpose}

	This report summarizes the preparation of newly generated visual puzzle test
	cases for reinforcement-learning data use. The objective was to create new
	examples without reusing pre-existing reference test cases from the checked-in
	datasets.

	\section{Generation Method}

	Two puzzle collections were prepared: AlgoPuzzleVQA and PuzzleVQA. For each
	puzzle category, candidate examples were created using the collection's existing
	generation procedure. Each candidate was compared with the corresponding
	pre-existing reference dataset before being retained.

	For AlgoPuzzleVQA, comparison was based on the complete underlying puzzle
	solution state. For PuzzleVQA, comparison considered both puzzle metadata and
	rendered image content, while ignoring presentation-only differences such as
	image file names and answer-option ordering. Candidates matching a reference
	example, or an example already accepted during the same run, were discarded.

	Each retained example consists of a generated image and a JSONL record containing
	the question, answer, answer choices where applicable, and supporting puzzle
	metadata. The final outputs were checked to confirm that every record has a
	corresponding image and that no retained case overlaps its reference dataset.

	\section{Generated Test Cases}

	\begin{table}[h]
	\centering
	\begin{tabular}{lrr}
	\toprule
	Collection & Puzzle Categories & Test Cases \\
	\midrule
	AlgoPuzzleVQA & 18 & 1,800 \\
	PuzzleVQA & 16 & 1,600 \\
	\midrule
	Total & 34 & 3,400 \\
	\bottomrule
	\end{tabular}
	\caption{Generated non-overlapping test cases. Each generated category contains
	100 test cases.}
	\end{table}

	\section{Dataset Organization}

	All generated datasets have been collected under the repository's
	\texttt{dataset/} directory. The AlgoPuzzleVQA outputs are located in
	\texttt{dataset/AlgoPuzzleVQA/data/}, while the PuzzleVQA outputs are located in
	\texttt{dataset/PuzzleVQA/data/}. Each collection contains one JSONL file per
	puzzle type and an \texttt{images/} directory containing a matching subdirectory
	for that puzzle type. The \texttt{puzzle\_type} identifiers used by the ontology
	files match these JSONL file and image-directory names.

	\section{Ontology Files}

	Two ontology files describe how the generated puzzle types are organized:
	\path{algopuzzlevqa_ontology.json} and
	\path{puzzlevqa_ontology.json}. They provide a compact, machine-readable
	index for selecting subsets of games during training or evaluation.

	\subsection{AlgoPuzzleVQA Ontology}

	The AlgoPuzzleVQA ontology contains all 18 available puzzle types. Each entry
	records its \texttt{puzzle\_type}, its required visual features, and its
	algorithmic reasoning features. A puzzle may belong to several feature groups.
	The visual features describe what must be interpreted from the image, whereas
	the algorithmic features describe the reasoning needed to solve the puzzle.

	Table~\ref{tab:algo-puzzles} lists each puzzle first, followed by its visual and
	algorithmic categories and its generation status.

	\small
	\begin{longtable}{p{0.18\textwidth}p{0.18\textwidth}p{0.38\textwidth}p{0.12\textwidth}}
	\caption{AlgoPuzzleVQA ontology and generation status.}\label{tab:algo-puzzles}\\
	\toprule
	Puzzle Name & Visual Categories & Algorithmic Categories & Generated \\
	\midrule
	\endfirsthead
	\toprule
	Puzzle Name & Visual Categories & Algorithmic Categories & Generated \\
	\midrule
	\endhead
	\game{board_tile} & Colour, position & Arithmetic, boolean logic & Yes \\
	\game{calendar} & Position, text & Arithmetic & Yes \\
	\game{chain_link} & Position, shape and size & Arithmetic, optimization, sets & Yes \\
	\game{checker_move} & Colour, position & Arithmetic, boolean logic, search & Yes \\
	\game{clock} & Position, shape and size, text & Arithmetic & Yes \\
	\game{colour_hue} & Colour, position & Arithmetic, optimization & Yes \\
	\game{map} & Colour, position, shape and size, text & Arithmetic, boolean logic, combinatorics, graphs, search & Yes \\
	\game{maze} & Colour, position & Arithmetic, graphs, optimization, search & Yes \\
	\game{move_box} & Colour, position & Arithmetic, graphs, optimization, search & Yes \\
	\game{n_queens} & Position & Arithmetic, boolean logic, search & Yes \\
	\game{number_slide} & Position, text & Arithmetic, combinatorics, optimization, search, sets & Yes \\
	\game{rotting_kiwi} & Position & Arithmetic, boolean logic, graphs, optimization, search & Yes \\
	\game{rubiks_cube} & Colour, position & Arithmetic, graphs & Yes \\
	\game{think_dot} & Colour, position, shape and size & Arithmetic, boolean logic, sets & Yes \\
	\game{tower_of_hanoi} & Position, shape and size & Arithmetic, boolean logic, optimization, search & Yes \\
	\game{water_jugs} & Position, shape and size, text & Arithmetic, boolean logic, optimization, search & Yes \\
	\game{wheel_of_fortune} & Position, shape and size, text & Arithmetic & Yes \\
	\game{wood_slide} & Position, shape and size & Arithmetic, optimization, search & Yes \\
	\bottomrule
	\end{longtable}
	\normalsize

	\subsection{PuzzleVQA Ontology}

	The PuzzleVQA ontology organizes games using four visual concepts:
	\emph{numbers}, \emph{colours}, \emph{size}, and \emph{shapes}. Each entry
	contains a concept list, a derived category name, a concept count, and the
	boolean field \texttt{has\_new\_generated\_data}. The boolean distinguishes
	games included in the new dataset from ontology entries for which no new data
	could be retained.

	Table~\ref{tab:puzzlevqa-puzzles} lists all 20 ontology entries, including the
	four puzzle types for which new data could not be generated.

	\small
	\begin{longtable}{p{0.32\textwidth}p{0.40\textwidth}p{0.16\textwidth}}
	\caption{PuzzleVQA ontology and generation status.}\label{tab:puzzlevqa-puzzles}\\
	\toprule
	Puzzle Name & Category & New Data Generated \\
	\midrule
	\endfirsthead
	\toprule
	Puzzle Name & Category & New Data Generated \\
	\midrule
	\endhead
	\game{grid_number} & Numbers & Yes \\
	\game{triangle} & Numbers & Yes \\
	\game{color_grid} & Colours & Yes \\
	\game{color_hexagon} & Colours & Yes \\
	\game{size_cycle} & Size & Yes \\
	\game{size_grid} & Size & No \\
	\game{shape_morph} & Shapes & No \\
	\game{shape_reflect} & Shapes & No \\
	\game{polygon_sides_number} & Numbers and shapes & Yes \\
	\game{venn} & Numbers and shapes & Yes \\
	\game{color_number_hexagon} & Numbers and colours & Yes \\
	\game{grid_number_color} & Numbers and colours & Yes \\
	\game{circle_size_number} & Numbers and size & Yes \\
	\game{rectangle_height_number} & Numbers and size & Yes \\
	\game{color_overlap_squares} & Shapes and colours & No \\
	\game{polygon_sides_color} & Shapes and colours & Yes \\
	\game{shape_size_grid} & Shapes and size & Yes \\
	\game{shape_size_hexagon} & Shapes and size & Yes \\
	\game{color_size_circle} & Colours and size & Yes \\
	\game{rectangle_height_color} & Colours and size & Yes \\
	\bottomrule
	\end{longtable}
	\normalsize

	\section{Result}

	The prepared dataset contains 3,400 new test cases across 34 puzzle categories.
	The four excluded PuzzleVQA categories would require an approved expansion of
	their original puzzle designs before 100 additional non-overlapping cases could
	be generated for each category.

	\end{document}