Point 2 and point 3 are doing more work together than either does alone, and the size of that win turned out to be language dependent in a way I did not expect.

The sequence tax for going byte-level is just bytes per token. So I measured it on a parallel corpus instead of guessing: crosslingual-rule-following/model-inference-responses, 4,680 prompts rendered in en, de and ru, identical ids in all three, Llama-3.1-8B responses, one sample each.

Same text, two tokenizers:

             en     de     ru     bytes per token
Qwen3-8B    4.94   3.91   6.12
GPT-2       4.87   2.64   1.69

The byte-level tax is not a property of bytes. It is a measurement of how badly your tokenizer fits the language. GPT-2 on Cyrillic is already most of the way to a byte model at 1.69, so going byte-level there costs 1.7x the positions. Qwen on the exact same Russian text costs 6.1x.

Which is why your point 2 has to fall before point 3 can pay. In a transformer, byte-level multiplies attention work by B squared: 2.9x for GPT-2 on that Russian, 37x for Qwen. Linear cost turns that back into a factor of B.

The uncomfortable part is the direction. The tokenizer that handles Russian best is the one with the most to lose by dropping tokens. The bad tokenizer already paid the tax years ago.

So is your six-point case for one shared byte model, or does the crossover sit in a different place per language?

"Efficient enough to absorb it" turns out not to be the axis. It resolves to a context length, and I can put a number on it.

Give the byte SSM every advantage. Linear cost, no attention term at all, and only the non-embedding parameters of Qwen3-8B (36 layers, d 4096, GQA 8 of 32, inter 12288): 6.946B against the token model's full 8.190B. Then the transformer's L squared term is the only thing that can pay for the byte model's B extra positions, and break-even is one division.

Driven by the same bytes-per-token I measured upthread:

                     B    prefill xover   decode xover   vs a 40,960 window
GPT-2 / ru        1.69           24,061         48,122     0.59x /  1.17x
GPT-2 / de        2.64           68,810        137,619     1.68x /  3.36x
Qwen3-8B / de     3.91          128,632        257,264     3.14x /  6.28x
Qwen3-8B / en     4.94          177,149        354,298     4.32x /  8.65x
Qwen3-8B / ru     6.12          232,732        465,463     5.68x / 11.36x

Your direction is right. The size is the part I would not have guessed. On the same Russian text, the crossover is 24,061 tokens against one tokenizer and 232,732 against the other. 9.7x, from nothing but whose vocabulary you are replacing.

That puts GPT-2 Russian inside an ordinary context window and Qwen Russian almost six windows past it. Which is not a question about the SSM. It is a question about how long your prompts are.

One caveat that cuts against me. I counted attention at 2n_lL^2*d, the conservative half of the QK and AV pair. Count both and every crossover halves: GPT-2 Russian to 12,030, Qwen Russian to 116,366. That shifts the level and not the 9.7x, because the ratio is set by B alone.

Which is the actual finding. The efficiency of the SSM cancels out of the ratio entirely. I handed it free parameters and a free quadratic term and Qwen-tokenized Russian still wants 233k tokens. What moves the answer is a smaller B, not a faster model.

So would you point a byte SSM at long-context serving, or at the languages the tokenizers already failed? Those look like two different models to me.