Hugging Face's logo
John Locke's picture
🏗️ Building on HF

John Locke

johnlockejrr
85 26 182
darkc0de's profile picture cantillation's profile picture MikeHarris's profile picture
·

AI & ML interests

OCR, HTR, ATR, NLP, AI

Recent Activity

liked a dataset about 10 hours ago
oddadmix/dialectal-arabic-lahgtna-v2
reacted to AbstractPhil's post with 👀 about 10 hours ago
The AlephLM results are rolling in and I'm very excited for the possibilities. I am very much looking forward to the coming weeks as I train the first AlephLM distillations from MANY teachers into AMOE arms. The AMOE arms hook cleanly to AlephLM structures and provide pos/neg learning elements. Hard positive and hard negatives coalesce to extend the capacity. https://huggingface.co/AbstractPhil/alephlm-0 https://huggingface.co/AbstractPhil/alephlm-adopt-0 As it stands they are structurally sound enough to fully pretrain. As or more stable than a standard Bert experimentally to distill using InfoNCE. AMOE legs improve these structures substantially. Structural behavior can be expanded in many ways on distilled and pretrained models alike. Attaching the AMOE to any model I've tried has created expanded or improved behavioral accumulations. They do have downsides but their upsides are very experimentally exciting. I've distilled multiple vits, multiple berts, and have begun distilling berts into AlephLM structures successfully. This is overall very exciting for me. I've begun formatting larger variants such as including GPT-2 and Qwen 3.5 4b as a paired combinator utilizing pathological T5 learned distilled encodings. It sounds odd, but the results show everything can be expanded and even be taught to cooperate. The CaptionBert-8192-v2 and v2-b are both structurally collapsing after token 480 or so, which is expected due to the small train. By distilling an AMOE arm to V2 by training with a longformer expert, the results are cutting through like butter. V2 has begun stabilizing rapidly for considerably longer token chains and sequences, the structure is repairing and building reusable capacity. I have discovered an improved methodology for sampling the AlephLM for text encoder benchmarks, which is predominantly L2 normalized outputs. Upcoming large paper for the distillation experiments and results within the next week or two. It's going to be a big one.
upvoted a paper about 24 hours ago
ChronoLens: Measuring Language Change Across Time, Languages, and Linguistic Levels
View all activity

Organizations

Samaritan AI's profile picture Syriac AI's profile picture Ethiopic AI's profile picture