HSTU 1B Generative Recommendation Model Test Configuration

config.json in this directory is the test configuration for the HSTU (Hierarchical Sequential Transduction Unit) generative recommendation model. This configuration describes the approximately 1B-parameter HSTU test model and declares HSTUForCausalLM as its model entry point.

Note: use_random_model is set to true. The model therefore uses randomly initialized weights for inference-pipeline, operator, and performance tests. This configuration does not indicate that a trained model checkpoint has been loaded.

1. Configuration Summary

Configuration	Value	Description
`model_type`	`hstu`	HSTU model type
`architectures`	`HSTUForCausalLM`	Model loading entry point. `CausalLM` is a framework-compatible interface name; the actual task is recommendation candidate scoring.
Intended model size	Approximately `1B`	Target size of the test model represented by this configuration
`num_hidden_layers` / `hstu_config.num_layers`	`12`	Number of HSTU layers
`hidden_size`	`4096`	Hidden-state dimension at each token or feature position
`num_attention_heads`	`16`	Number of attention heads per layer
`head_dim`	`256`	Dimension of each attention head; `16 × 256 = 4096`
`max_seq_len`	`8832`	Maximum sequence length allowed for one request. The exact number of history and candidate positions depends on the input-packing scheme.
`torch_dtype` / `hstu_config.dtype`	`float32`	Data type used by this test configuration
`is_causal`	`true`	Enables causal masking, so a position cannot read future positions
`residual`	`true`	Enables residual connections in HSTU layers
`has_ffn`	`false`	No separate feed-forward network is included in each HSTU layer
`dropout_ratio`	`0`	Dropout is disabled for inference
`norm_epsilon`	`1e-5`	Numerical-stability epsilon used by normalization layers

2. Input Features and Embedding Tables

task_config.embedding_configs defines the recommendation features used by the model:

Feature	Table	Vocabulary Size	Embedding Dimension	Dynamic Embedding	Purpose
`action_weights`	`act`	`1024`	`4096`	No	User behavior/action type or behavior weight
`video_id`	`item`	`100000`	`4096`	Yes	Video/item ID used as a candidate-item feature

The configuration also declares:

item_feature_name = "video_id"
action_feature_name = "action_weights"
Both input embedding dimensions match hidden_size = 4096.

In a typical HSTU inference implementation, input features first go through embedding lookup. Multiple features at the same position are combined through concatenation and/or a linear projection to form a 4096-dimensional input representation. Positional information is then added before the representation is passed to the HSTU backbone.

3. HSTU Model Architecture

The logical path from model inputs to recommendation scores is:

User-history features + candidate-item features
        │
        ├─ action_weights embedding: 4096 → 4096
        └─ video_id embedding:       4096 → 4096
        │
        └─ Feature combination / linear projection + positional embedding
                         │
                         ▼
              ┌──────────────────────┐
              │ HSTU Layer 1         │
              │ …                    │
              │ HSTU Layer 12        │
              └──────────────────────┘
                         │
                         ▼
              Hidden states at candidate positions
                         │
                         ▼
                    Dense/MLP score head
                         │
                         ▼
              One scalar score for each candidate

A single HSTU layer can be summarized as follows:

Input x
 │
 ├─ Input normalization (epsilon = 1e-5)
 ├─ Fused UVQK linear projection
 │    └─ Split into U, V, Q, and K representations
 ├─ SiLU activation
 ├─ Causal attention (16 heads, head_dim = 256)
 ├─ Attention-output normalization
 ├─ U gating / element-wise modulation of the attention output
 ├─ Linear projection back to 4096 dimensions
 └─ Residual connection (residual = true)
 │
Output x'

Because has_ffn = false, this 1B configuration does not add a separate SwiGLU or other feed-forward sub-layer after the HSTU attention projection. The residual path is still enabled. The core per-layer transformation is therefore the normalized UVQK projection, causal attention, U-gated attention output, output projection, and residual addition.

This is not a conventional language model that generates text tokens. It uses a ForCausalLM-style interface to apply causal sequence modeling to the joint representation of user behavior history and candidate items, and ultimately produces a ranking score for each candidate. The current config.json does not explicitly specify the intermediate architecture of the prediction head under task_config; the concrete head configuration is determined by the model implementation or loader defaults. Its task semantics are to output one score per candidate.

4. Inference Paradigm: Generative Recommendation vs. LLM

Both model types may expose a CausalLM-style interface, but their decode-stage objectives are fundamentally different.

4.1 Generative Recommendation: Parallel Candidate Scoring

Given user history (h) and a candidate set (C = {c_1, \ldots, c_M}), HSTU computes a score for every candidate:

[ s_i = f_\theta(h, c_i), \qquad i = 1, \ldots, M ]

During decode, multiple candidates are arranged as positions or packed sequences that can be evaluated in parallel. A forward pass produces the score vector:

[ \mathbf{s} = [s_1, s_2, \ldots, s_M] ]

The candidates are then sorted by score and truncated to Top-K. Thus, the primary parallel dimension during decode is the candidate dimension. As the candidate count grows, candidate batching, candidate parallelism, and memory usage become important. The model does not need to generate (c_1) first and then generate (c_2) conditioned on (c_1).

4.2 LLM: Autoregressive Token-by-Token Generation

Given a prompt prefix (x_{<t}), an LLM computes the distribution of the next token at step (t):

[ p(x_t \mid x_{<t}) = \operatorname{softmax}(W h_t) ]

After selecting or sampling a token, the token is appended to the sequence and the model proceeds to step (t+1), continuing until a complete sequence is generated or an EOS token is reached. The logits for the whole vocabulary can be computed in parallel at each step, but the time dimension remains autoregressive and sequential. The KV cache is primarily used to reuse the already generated prefix states.

4.3 Comparison

Dimension	HSTU Generative Recommendation	Conventional LLM
Decode objective	Evaluate a set of candidate items simultaneously	Generate the next token
Computation unit	Candidate / candidate position	Token / time step
Typical output	`M` candidate scores, followed by sorting and Top-K selection	One token per step, repeated to form a complete sequence
Dependency pattern	Candidates generally do not require autoregressive dependencies on one another	The current token depends on previously generated tokens
Main parallel dimension	Candidate dimension; candidates can be batched and evaluated in parallel	Vocabulary logits are parallel within a step, but time steps are sequential
Stopping condition	Candidate scoring completes, followed by ranking/truncation	EOS, maximum generation length, or another stopping rule
Cache purpose	Reuse the user-history representation to reduce candidate-scoring cost	Reuse historical token K/V states to reduce incremental-generation cost

In simplified form:

HSTU decode:  history + [candidate_1 ... candidate_M]
              └──────── one parallel forward pass ────────┘
              → [score_1 ... score_M] → Top-K

LLM decode:   prompt → token_1 → token_2 → … → token_T
              Each step determines or samples only the next token

5. Configuration Consistency Check

The top-level vocab_size is 1,001,024, while the two explicitly configured embedding-table vocabulary sizes sum to 1024 + 100000 = 101,024. If vocab_size represents the actual number of rows in a merged embedding table, verify whether the additional 900,000 IDs correspond to reserved IDs, dynamic-embedding capacity, or another offset. If they have no additional purpose, this field should be checked before loading real weights and tokenizer/ID mappings to avoid embedding out-of-range errors or unnecessary memory allocation.