Token Averaging A comprehensive research framework for analyzing whether averaging adjacent tokens in a LLM can reduce the compute compared to standard model. FAIRC/token-averaging-matched_mean-avg_50m_k8 Updated 5 days ago • 11 FAIRC/token-averaging-model1_50m Updated 5 days ago • 23 FAIRC/token-averaging-avg_50m_k8_wexp Updated 5 days ago • 11 FAIRC/token-averaging-avg_50m_k8_learnable_pos Updated 5 days ago • 7
MoWE Paper Releases associated with the MoWE paper written by Tavish, Vardhaman, Keshava, Dr. Deepan. Coded and trained with openlanguagemodel. FAIRC/mowe-mowe-768-19l 0.3B • Updated 6 days ago • 3 FAIRC/mowe-dense-768-19l 0.1B • Updated 6 days ago • 3 FAIRC/mowe-moe-768-19l 0.4B • Updated 6 days ago • 5
Token Averaging A comprehensive research framework for analyzing whether averaging adjacent tokens in a LLM can reduce the compute compared to standard model. FAIRC/token-averaging-matched_mean-avg_50m_k8 Updated 5 days ago • 11 FAIRC/token-averaging-model1_50m Updated 5 days ago • 23 FAIRC/token-averaging-avg_50m_k8_wexp Updated 5 days ago • 11 FAIRC/token-averaging-avg_50m_k8_learnable_pos Updated 5 days ago • 7
MoWE Paper Releases associated with the MoWE paper written by Tavish, Vardhaman, Keshava, Dr. Deepan. Coded and trained with openlanguagemodel. FAIRC/mowe-mowe-768-19l 0.3B • Updated 6 days ago • 3 FAIRC/mowe-dense-768-19l 0.1B • Updated 6 days ago • 3 FAIRC/mowe-moe-768-19l 0.4B • Updated 6 days ago • 5
