I reproduced the main observation of "Vision Transformers Need Registers" (Darcet et al.) on six DINOv2 checkpoints from the Hub: Base, Large and Giant, each with and without registers.
Three findings:
1️⃣ Model size matters. In DINOv2 Giant, ~1.6% of patch tokens form a separate high-norm group. In Large the effect is weak, and in Base I found none.
2️⃣ Where you measure matters. The artifact tokens have very high norms before the final LayerNorm (hidden_states[-1]), but the lowest norms after it (last_hidden_state).
3️⃣ Registers take over. With registers, patch maps are smooth, and one register token carries a norm of ~1,346 instead.
Everything is inference-only and reproducible with one command. The post also includes practical tips for people who use DINOv2 features for dense tasks or probing.
@Banaxi-Tech stop hiding my comments. AND STOP STEALING PAPERS AND SPREADING MISINFORMATION. your BGA blog is a copy of NSA (deepseek, 2025) branded under your name. literally the same top16 selected blocks, 512 local window, router over block summaries, all you did was change block size from 64 to 128. you didnt cite NSA once but you put a “please cite BGA” bibtex at the bottom. i commented under your post and said that there is no way that you can support claims like: “The Accuracy Should BE WAy better than DSA but untested yet.” you didnt run a single experiment. and the 256x isnt from BGA, its just n/2k with k=2048 so the exact same k DSA uses. if opus wrote this for you, at least read it before posting. i commented again after you hid my comment despite it having constructive and correct feedback and you hid that too. and again. you can hide the truth and just try to get hf post likes..... but is it really the thing that needs to be done? do you really want to take papers and make them yours while barely even changing the params?
admitting your mistakes and doing something about them needs humbleness, intelligence, humanness. i encourage you to admit your mistakes and try to do better next time (at least read what blog your ai wrote or do proper experiments to back your stuff up).
Pretrained on 4x more tokens than the previous releases (20b vs 5b). Instruct tuned versions are coming soon. Very interesting models are coming soon too (hint: super long context).
Just hit #14 and #15 with out FIRST models on Open SLM Leaderboard. The models were trained on 5B tokens, while competing with similarly sized models trained on more than 6-20x the data.
A new base model Speck1.5-140M being trained right now on a higher quality corpus and will be released soon. SpeckChat3 is coming very soon with 1 million samples, specifically designed to post train small base models.
Also, just to clarify stuff, we will NOT release anything that is NOT MIT licensed EVER. Openness is needed in small language research.
Thanks to everyone supporting the project, and stay tuned for new releases!
SPECK UPDATES: 1 New instruct model tuned on top of Speck1-140M: specklabs/Speck1-140M-Instruct 2 Instruction tuning datasets 2 GGUFs
Much more coming soon: Speck1.1-140M-Instruct that is post trained on SpeckChat2 will be coming very soon New base model Speck1.5-140M is coming with a much higher quality corpus
Thanks to everyone who is already supporting the project, and stay tuned for new releases!
new models coming very soon (both instruct and much better models), with much much higher training scale as i am getting marenostrum5 access soon! we will be looking at 100b-2t token budgets :)
SKT AI Labs, we are pushing the boundaries of AI architecture and research—and today, we are thrilled to open our doors to the global research community!
We warmly welcome researchers, developers, and AI enthusiasts to join us and contribute to our R&D efforts.
🧪 What You Can Explore:
We invite you to experiment with our WMF (Weight Manifold Fusion) technology. You can test this high-dimensional fusion technique on smaller models to gain a deeper understanding of its behavior and token convergence.
If it works: Fantastic! Share your results with us and contribute directly to the core vision of SKT AI Labs.
If it doesn't work: No problem at all! Your critical feedback is just as valuable to us. Every experiment and anomaly helps us refine this architecture to make it more stable and robust.
We firmly believe that true innovation stems from community collaboration and transparent testing. Let's build the future of advanced AI together. Your ideas, test results, and feedback are always welcome!
You Can Still Research and Development On WMF Only SKT-SURYA-H Model is Dismissed.
We’re excited to release NRS_QWEN_MYTHOS_1M — a powerful reasoning model built on Qwen 3.5 9B! At SKT AI LABS, we’ve supercharged this 9B model with our proprietary Neural Reasoning System (NRS) to deliver next-level performance.
🔥 Why This Model is a Game-Changer: ✅ 100x Reasoning Capacity — Exceptional deep logical thinking and complex problem-solving ✅ 1 Million Token Context — Perfect for massive codebases, long documents, and multi-turn agentic workflows ✅ Advanced Thinking Mode — Native <think> tags for true step-by-step Chain-of-Thought reasoning ✅ Tool-Use Ready — Optimized for Python execution, Web Search, and self-correction ✅ Blazing Fast — Runs smoothly on consumer GPUs like RTX 3090/4090
Whether you’re a developer building coding agents, a researcher working with long-context data, or someone who loves powerful reasoning — this model is built for you.