anushreeberlia/fashion-florence

Fashion Florence: Structured Fashion Tagging with Florence-2

A fine-tuned Florence-2-large model that analyzes clothing images and returns structured fashion tags as JSON.

Florence-2 is an advanced vision foundation model that uses a prompt-based approach to handle a wide range of vision and vision-language tasks. This fine-tuned version specializes in structured fashion attribute extraction.

Model Summary

Base model	microsoft/Florence-2-large (0.77B params)
Method	LoRA (r=16, alpha=32) merged into base weights
Task	Clothing image → structured JSON fashion tags
Dataset	Marqo/iMaterialist

Output Schema

Given a clothing image, the model outputs structured JSON with:

Field	Type	Description	Example
category	string	One of: top, bottom, dress, layer, shoes, accessory	"dress"
primary_color	string	Dominant color	"navy blue"
material	string	Fabric/material	"cotton"
style_tags	list	Style descriptors	["casual", "minimalist"]

Evaluation Results

Evaluated on 3,915 held-out test examples (5% split from iMaterialist).

Overall Metrics

Metric	Score
JSON parse rate	100.0%
Category accuracy	89.5%
Color exact-match	52.7%
Material exact-match	42.6%
Style-tag Jaccard (IoU)	0.644
Style-tag macro F1	0.888

Per-Category Performance

Category	Precision	Recall	F1	Support
top	0.87	0.89	0.88	1,137
bottom	0.94	0.88	0.91	778
dress	0.91	0.92	0.91	1,512
layer	0.84	0.90	0.87	435
shoes	0.82	0.49	0.61	47
accessory	1.00	0.17	0.29	6

Color and material use exact-match scoring; near-synonyms (e.g. "navy" vs "navy blue") count as misses. Shoes and accessory have low support in the test set.

Comparison with Other Approaches

While direct comparisons are approximate (different datasets and task setups), these benchmarks provide context:

Model	Task	Category F1	Attribute Score	Notes
Fashion Florence (ours)	Structured JSON (4 fields)	0.89 weighted	Style-tag F1: 0.888	Fine-tuned LoRA, 0.77B
GPT-4o mini	18 fashion attributes	—	Macro F1: 0.433	Zero-shot
Gemini 2.0 Flash	18 fashion attributes	—	Macro F1: 0.568	Zero-shot
FashionCLIP	Category classification	0.71	—	Contrastive learning
CLIP	Category classification	0.66	—	General purpose
Atlas (attention-based)	52-class categorization	0.92 micro	—	Specialized classifier

Fashion Florence outperforms zero-shot LLMs by a wide margin on fashion attribute prediction, and approaches specialized classifiers on category accuracy while also producing full structured output.

How to Get Started with the Model

import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoProcessor

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model = AutoModelForCausalLM.from_pretrained(
    "anushreeberlia/fashion-florence",
    torch_dtype=torch_dtype,
    trust_remote_code=True,
).to(device)
processor = AutoProcessor.from_pretrained(
    "anushreeberlia/fashion-florence", trust_remote_code=True,
)

image = Image.open("clothing_image.jpg").convert("RGB")
prompt = "Analyze this clothing item image and return structured fashion tags as JSON."

inputs = processor(text=prompt, images=image, return_tensors="pt").to(device, torch_dtype)
generated_ids = model.generate(
    input_ids=inputs["input_ids"],
    pixel_values=inputs["pixel_values"],
    max_new_tokens=256,
    num_beams=3,
)
result = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(result)

Training Details

• Epochs: 2
• Learning rate: 2e-4
• Effective batch size: 16 (1 × 16 gradient accumulation)
• LoRA rank: 16, alpha: 32, dropout: 0.05
• Target modules: all linear layers
• Precision: fp16
• Dataset: ~78k training examples from iMaterialist

Citation

@article{xiao2023florence,
  title={Florence-2: Advancing a unified representation for a variety of vision tasks},
  author={Xiao, Bin and Wu, Haiping and Xu, Weijian and Dai, Xiyang and Hu, Houdong and Lu, Yumao and Zeng, Michael and Liu, Ce and Yuan, Lu},
  journal={arXiv preprint arXiv:2311.06242},
  year={2023}
}