LFM 350M-C
The text encoder can:
- predict the logits behind the <|mask|> tokens
- predict similarity embeddings for both natural language and comma-separated text behind the <|img_thumbnail|> token for the pooled embedding
Regardless of the captioning strategy, it can understand both booru captions and natural language and provide a pooled embedding.
Anime images
The model has been verified for both the LFM (1024) and the CLIP (768) dimensions.
This encoded text can be used as input data for a compatible T2I model.
Contrastive Text Model
The pooled embedding can be computed from any of the previously described formats while preserving the underlying semantic representation.
Unlike image encoder models, the embedding were learned from pairs of texts.
Tokenization
The tokenizer has remained unchanged.
The English stop words have been removed from the captions.
The comma-seperated keywords were scrambled and adjusted for the model.
Initial epochs
The model was initialised using a low learning rate and natural language captions from CC (Moondream) and Danbooru (Qwen3.5).
The masked language model then continued to learn from a combination of comma-separated and natural language texts.
The artists' given names, nicknames and fantasy names have been deliberately excluded from the dataset, as it would be difficult to predict these based on the text alone.
Changes
Compared to the 230M model, this has a cleaner, more recent dataset.
Compared to the 350M model, this includes the optional pooled embedding.
Syntax
Just use the tokenizer(['...']) for any of these.
[bos]...
[bos][pooled]A cartoon character, sits on the ground with a grumpy expression, holding a submachine gun
[bos][pooled]anthro, grumpy, submachine gun, holding weapon
Source data
- anime_captions
- artbench_captions
- cc12m_2mp_realistic
- danbooru_multitier_captions_202606
- dtg_character_tags
- furry_e621
- gelbooru_characters_enriched
- gelbooru_tags_full
- wikiart_215k
- samples for a certain locked tag
- Downloads last month
- -