nebulette/puro-2b-contrastive

Puro 2B-C

The text encoder can:

• predict the logits behind the <|mask|> tokens
• predict similarity embeddings for both natural language and comma-separated text behind a special token for the pooled embedding

Regardless of the captioning strategy, it can understand both booru captions and natural language and provide a pooled embedding.

The loss was calculated in the ForMaskedLM mode during training.

Anime images

This encoded text can be used as input data for a compatible T2I model.

Compared to the small LFM2.5 encoders, this model uses a higher hidden dimension (2048).

Contrastive Text Model

The pooled embedding can be computed from any of the previously described formats while preserving the underlying semantic representation.

Unlike image encoder models, the embedding were learned from pairs of texts.

Tokenization

The tokenizer has remained unchanged.

The English stop words have been removed from the captions.

The comma-seperated keywords were scrambled and adjusted for the model.

Initial epochs

The model was initialised using a low learning rate and natural language captions from CC (Moondream) and Danbooru (Qwen3.5).

The masked language model then continued to learn from a combination of comma-separated and natural language texts.

The artists' given names, nicknames and fantasy names have been deliberately excluded from the dataset, as it would be difficult to predict these based on the text alone.

Fixed danbooru

Grio's patched tags were applied to the danbooru dataset. This makes it the most recent and accurate source.

Other booru sites were included as well.

Encoding syntax

Just use the tokenizer(['...']) for any of the input text without the chat template.

Masking syntax

A non-whitespace character precedes the mask token.

word1 word2<|mask|> ...

Source data

• anime_captions
• artbench_captions
• cc12m_2mp_realistic
• danbooru_multitier_captions_202606
• dtg_character_tags
• furry_e621
• gelbooru_characters_enriched
• gelbooru_tags_full
• tag_cleaning
• wikiart_215k
• samples for a certain locked tag