# HSTU 1B Generative Recommendation Model Test Configuration [`config.json`](D:/tmp/hstu_1b/config.json) in this directory is the test configuration for the **HSTU** (Hierarchical Sequential Transduction Unit) generative recommendation model. This configuration describes the approximately **1B-parameter** HSTU test model and declares `HSTUForCausalLM` as its model entry point. > **Note:** `use_random_model` is set to `true`. The model therefore uses randomly initialized weights for inference-pipeline, operator, and performance tests. This configuration does not indicate that a trained model checkpoint has been loaded. ## 1. Configuration Summary | Configuration | Value | Description | | --- | ---: | --- | | `model_type` | `hstu` | HSTU model type | | `architectures` | `HSTUForCausalLM` | Model loading entry point. `CausalLM` is a framework-compatible interface name; the actual task is recommendation candidate scoring. | | Intended model size | Approximately `1B` | Target size of the test model represented by this configuration | | `num_hidden_layers` / `hstu_config.num_layers` | `12` | Number of HSTU layers | | `hidden_size` | `4096` | Hidden-state dimension at each token or feature position | | `num_attention_heads` | `16` | Number of attention heads per layer | | `head_dim` | `256` | Dimension of each attention head; `16 × 256 = 4096` | | `max_seq_len` | `8832` | Maximum sequence length allowed for one request. The exact number of history and candidate positions depends on the input-packing scheme. | | `torch_dtype` / `hstu_config.dtype` | `float32` | Data type used by this test configuration | | `is_causal` | `true` | Enables causal masking, so a position cannot read future positions | | `residual` | `true` | Enables residual connections in HSTU layers | | `has_ffn` | `false` | No separate feed-forward network is included in each HSTU layer | | `dropout_ratio` | `0` | Dropout is disabled for inference | | `norm_epsilon` | `1e-5` | Numerical-stability epsilon used by normalization layers | ## 2. Input Features and Embedding Tables `task_config.embedding_configs` defines the recommendation features used by the model: | Feature | Table | Vocabulary Size | Embedding Dimension | Dynamic Embedding | Purpose | | --- | --- | ---: | ---: | --- | --- | | `action_weights` | `act` | `1024` | `4096` | No | User behavior/action type or behavior weight | | `video_id` | `item` | `100000` | `4096` | Yes | Video/item ID used as a candidate-item feature | The configuration also declares: - `item_feature_name = "video_id"` - `action_feature_name = "action_weights"` - Both input embedding dimensions match `hidden_size = 4096`. In a typical HSTU inference implementation, input features first go through embedding lookup. Multiple features at the same position are combined through concatenation and/or a linear projection to form a 4096-dimensional input representation. Positional information is then added before the representation is passed to the HSTU backbone. ## 3. HSTU Model Architecture The logical path from model inputs to recommendation scores is: ```text User-history features + candidate-item features │ ├─ action_weights embedding: 4096 → 4096 └─ video_id embedding: 4096 → 4096 │ └─ Feature combination / linear projection + positional embedding │ ▼ ┌──────────────────────┐ │ HSTU Layer 1 │ │ … │ │ HSTU Layer 12 │ └──────────────────────┘ │ ▼ Hidden states at candidate positions │ ▼ Dense/MLP score head │ ▼ One scalar score for each candidate ``` A single HSTU layer can be summarized as follows: ```text Input x │ ├─ Input normalization (epsilon = 1e-5) ├─ Fused UVQK linear projection │ └─ Split into U, V, Q, and K representations ├─ SiLU activation ├─ Causal attention (16 heads, head_dim = 256) ├─ Attention-output normalization ├─ U gating / element-wise modulation of the attention output ├─ Linear projection back to 4096 dimensions └─ Residual connection (residual = true) │ Output x' ``` Because `has_ffn = false`, this 1B configuration does **not** add a separate SwiGLU or other feed-forward sub-layer after the HSTU attention projection. The residual path is still enabled. The core per-layer transformation is therefore the normalized UVQK projection, causal attention, U-gated attention output, output projection, and residual addition. This is not a conventional language model that generates text tokens. It uses a `ForCausalLM`-style interface to apply causal sequence modeling to the joint representation of user behavior history and candidate items, and ultimately produces a ranking score for each candidate. The current `config.json` does not explicitly specify the intermediate architecture of the prediction head under `task_config`; the concrete head configuration is determined by the model implementation or loader defaults. Its task semantics are to output one score per candidate. ## 4. Inference Paradigm: Generative Recommendation vs. LLM Both model types may expose a `CausalLM`-style interface, but their decode-stage objectives are fundamentally different. ### 4.1 Generative Recommendation: Parallel Candidate Scoring Given user history \(h\) and a candidate set \(C = \{c_1, \ldots, c_M\}\), HSTU computes a score for every candidate: \[ s_i = f_\theta(h, c_i), \qquad i = 1, \ldots, M \] During decode, multiple candidates are arranged as positions or packed sequences that can be evaluated in parallel. A forward pass produces the score vector: \[ \mathbf{s} = [s_1, s_2, \ldots, s_M] \] The candidates are then sorted by score and truncated to Top-K. Thus, the primary parallel dimension during decode is the **candidate dimension**. As the candidate count grows, candidate batching, candidate parallelism, and memory usage become important. The model does not need to generate \(c_1\) first and then generate \(c_2\) conditioned on \(c_1\). ### 4.2 LLM: Autoregressive Token-by-Token Generation Given a prompt prefix \(x_{