This is a simplified version of Rikka's model. During training, its primary objective is to predict one of the 30 subcategories from the danbooru dataset.
Its secondary goal was to predict the input token, but due to its size, it was not designed to perform the fill in the mask function.