Post
84
New blog post: Seeing ViT Registers 🔍
I reproduced the main observation of "Vision Transformers Need Registers" (Darcet et al.) on six DINOv2 checkpoints from the Hub: Base, Large and Giant, each with and without registers.
Three findings:
1️⃣ Model size matters. In DINOv2 Giant, ~1.6% of patch tokens form a separate high-norm group. In Large the effect is weak, and in Base I found none.
2️⃣ Where you measure matters. The artifact tokens have very high norms before the final LayerNorm (hidden_states[-1]), but the lowest norms after it (last_hidden_state).
3️⃣ Registers take over. With registers, patch maps are smooth, and one register token carries a norm of ~1,346 instead.
Everything is inference-only and reproducible with one command. The post also includes practical tips for people who use DINOv2 features for dense tasks or probing.
📝 Blog: https://huggingface.co/blog/turhancan97/seeing-vit-registers
💻 Code: https://github.com/turhancan97/seeing-vit-registers
I reproduced the main observation of "Vision Transformers Need Registers" (Darcet et al.) on six DINOv2 checkpoints from the Hub: Base, Large and Giant, each with and without registers.
Three findings:
1️⃣ Model size matters. In DINOv2 Giant, ~1.6% of patch tokens form a separate high-norm group. In Large the effect is weak, and in Base I found none.
2️⃣ Where you measure matters. The artifact tokens have very high norms before the final LayerNorm (hidden_states[-1]), but the lowest norms after it (last_hidden_state).
3️⃣ Registers take over. With registers, patch maps are smooth, and one register token carries a norm of ~1,346 instead.
Everything is inference-only and reproducible with one command. The post also includes practical tips for people who use DINOv2 features for dense tasks or probing.
📝 Blog: https://huggingface.co/blog/turhancan97/seeing-vit-registers
💻 Code: https://github.com/turhancan97/seeing-vit-registers
