Distillation comparison: Escarda x Byrne
Yo I want to ask about the distillation process of Byrne and Escarda. I want to do a comparison of the pair and just wondered if JEPA was the only intended variable between them, or are there more variables I should account for that might have not been highlighted on the model cards? I have been studying novel architecture and model compositions that don't fit in with the current popular designs. I would really like to look at Escarda with Byrne as a control. They are deadly candidates to observe and in my opinion worth spending some time making adaptations to my current tools and process. If there is any real structural difference that might make a comparison inaccurate let me know and I will switch the parameters an look at doing an independent model study..
Hey, sorry for the late reply! Yes — JEPA is the only intended variable; both share the same HRM, and the rest of the recipe (teacher, distillation data, training schedule) is identical, so Byrne works fine as a control.
One heads-up: I'm pushing a quality update to all the models later today. I'd hold off starting your comparison until the new versions are up, or pin the current revisions explicitly, so you're not comparing across checkpoint versions.
Thanks for the heads up! I'll wait for the update before I get started and make sure anything I write and share is includes the version I worked with.
Updated.