Hugging Face's logo
Papers
arxiv:2105.03994

Dispatcher: A Message-Passing Approach To Language Modelling

Published on May 9, 2021
Authors:

Abstract

A message-passing mechanism with a Dispatcher layer replaces self-attention for unidirectional generation, achieving comparable perplexity with O(N log N) time and O(N) memory complexity.

Generated by thinkingmachines/Inkling-Small

This paper proposes a message-passing mechanism to address language modelling. A new layer type is introduced that aims to substitute self-attention for unidirectional sequence generation tasks. The system is shown to be competitive with existing methods: Given N tokens, the computational complexity is O(N logN) and the memory complexity is O(N) under reasonable assumptions. In the end, the Dispatcher layer is seen to achieve comparable perplexity to prior results while being more efficient.

View arXiv page View PDF Add to collection

Community

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2105.03994
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 1

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2105.03994 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2105.03994 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.