Walker

Walker is a decoder-only language model that adds trajectory modeling to the Transformer.

The main idea is simple:

Attention looks at other tokens. Walker looks at how representations move.

How does attention work?

A Transformer represents each token as a vector. For every token, self-attention creates:

• Query (Q): what the token is looking for
• Key (K): what each token contains
• Value (V): the information that can be retrieved

The query is compared with other keys to determine which tokens are relevant:

$$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^T}{\sqrt{d}}\right)V\backslash text\{Attention\}(Q,K,V)\; =\; \backslash text\{softmax\}\backslash left(\backslash frac\{QK^T\}\{\backslash sqrt\{d\}\}\backslash right)V$$

In simple terms:

Attention asks: "What should I look at?"

A different approach: trajectory

Walker keeps attention, but asks another question:

"Where is my representation going?"

Consider the hidden states:

$$x_1,x_2,x_3,x_4,\dots x\_1,\backslash \; x\_2,\backslash \; x\_3,\backslash \; x\_4,\backslash ldots$$

Instead of only using the states themselves, Walker measures their changes:

$$d_t=x_t-x_{t-1}d\_t\; =\; x\_t-x\_\{t-1\}$$

This is a discrete derivative of the hidden representation.

Now the model can observe a trajectory rather than just a sequence of points:

x₁ → x₂ → x₃ → x₄ → ...

     ↓
 differences
     ↓
 trajectory

Walker uses multiple scales of these differences to capture both short- and longer-range motion, along with changes in direction.

Why "Walker"?

Imagine the hidden states forming a path through a high-dimensional space.

A Transformer can look around the path using attention.

Walker also tries to walk along the path.

Transformer
    │
    └── Attention
          └── "What should I look at?"

Walker
    │
    └── Trajectory
          └── "Where am I going?"

The two mechanisms are complementary. Walker is not designed to replace attention; it adds trajectory features as a residual pathway alongside the Transformer.

Architecture

                 Input Tokens
                      │
                 Token Embedding
                      │
          ┌───────────┴───────────┐
          │                       │
      Attention                Walker
          │                       │
   Token relationships        Trajectory
          │                       │
          └───────────┬───────────┘
                      │
                 Residual Update
                      │
                      ▼
                 Next Block

The Walker pathway is initialized as an exact no-op, allowing the model to begin as a normal Transformer and gradually learn how much trajectory information to use.

Summary

$$\boxed{{\displaystyle \text{Walker}=\text{Attention}+\text{Trajectory Modeling}}}\backslash boxed\{\; \backslash text\{Walker\}\; =\; \backslash text\{Attention\}\; +\; \backslash text\{Trajectory\; Modeling\}\; \}$$

Attention: What information is relevant?

Walker: How is the representation moving?

Walker is a Transformer that not only looks at the sequence, but also walks through the trajectory of its own representations.