File size: 3,689 Bytes
7aefd9e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
# Phase One — Source API & Execution Path Audit

## 1. Canonical Research API (`src/`)

### 1.1 `src.config.ModelConfig`
- **File:** `src/config.py`
- **Fields:**
  - `d_model: int = 128`
  - `n_heads: int = 4`
  - `n_layers: int = 2`
  - `ff_multiplier: int = 4`
  - `max_seq_len: int = 128`
  - `vocab_size: int = 10000`
  - `dropout: float = 0.1`
  - `tt_rank: int = 8`
  - `tt_min_rank: int = 2`
  - `use_tensor_ffn: bool = True`
  - `n_qubits: int = 4`
  - `n_quantum_layers: int = 2`
  - `quantum_sparsity: float = 0.3`
  - `use_quantum: bool = True`
  - `rank_alpha: float = 2.0`
  - `rank_smoothing: float = 0.9`
- **Validation:** `ModelConfig.validate()` asserts `d_model % n_heads == 0`, `1 <= tt_min_rank <= tt_rank`, `n_qubits <= 8`, and `0 <= quantum_sparsity <= 1`. `validate_model_config(config, model_class)` checks all required fields without inventing defaults.

### 1.2 `src.models.QTensorFormer`
- **File:** `src/models.py`
- **Constructor:** `QTensorFormer(config: ModelConfig, preset: str = "QTF_BALANCED")`
- **Supported Presets:** `QTF_FULL`, `QTF_BALANCED`, `QTF_LATENCY`, `QTF_MEMORY`, `QTF_ENERGY`, `QTF_EDGE`, `QTF_CLASSICAL_ONLY`.
- **Execution Path (`forward`):**
  1. Embeds `input_ids` (`nn.Embedding`) + sinusoidal `PositionalEncoding`.
  2. Iterates over `self.blocks` (`nn.ModuleList` of `HybridBlock`):
     - `HybridBlock` calls `self.info_engine(hidden_states=x)` $\rightarrow z_t \in [0, 1]^{B \times T \times 8}$.
     - `HybridBlock` calls `self.allocator(z_t, budget=budget, preset=preset, force_classical=...)` $\rightarrow$ selects `rank`, `token_ranks`, `attn_mode_idx`, `depth_mode`, `kv_precision`, `is_quantum_token`.
     - If `depth_mode == "skip"`, returns residual `x` immediately.
     - Updates `kv_cache.set_precision(kv_precision)` if `kv_cache` is supplied.
     - Runs `self.attention(normed_x, mask=mask, kv_cache=kv_cache, use_quantum=has_quantum_attn and not force_classical)`.
     - Checks optional `early_exit_head` if `enable_early_exit=True`.
     - If `depth_mode == "partial"`, skips FFN and returns `x`.
     - Sets `self.tt_ffn.set_rank(chosen_rank)` and executes `self.tt_ffn(normed_ffn_in, token_ranks=token_ranks)`.
     - If `is_quantum_token.any()` and not `force_classical`, applies `self.quantum_backend(normed_ffn_in)` masked by `is_quantum_token`.
  3. Applies final `self.ln_f` and weight-tied `self.lm_head`.

---

## 2. Canonical Hugging Face API (`configuration_qtensorformer.py` & `modeling_qtensorformer.py`)

### 2.1 `configuration_qtensorformer.QTensorFormerConfig`
- **Inherits from:** `transformers.PretrainedConfig` (`model_type = "qtensorformer"`)
- **Strict Loading Policy:** `keys_to_ignore_at_loading = []` (repaired in Phase Two so no missing weights are ignored).
- **Bridge Properties:** Exposes `.d_model`, `.n_layers`, `.n_heads`, `.n_kv_heads`, `.max_seq_len` and accepts legacy keyword arguments (`d_model`, `n_layers`, `n_heads`, `ff_multiplier`, `tt_rank`) in `__init__`.

### 2.2 `modeling_qtensorformer.QTensorFormerForCausalLM`
- **Inherits from:** `transformers.PreTrainedModel`
- **Submodules:**
  - `model.embed_tokens` (`nn.Embedding`)
  - `model.layers` (`nn.ModuleList` of `QTensorFormerDecoderLayer`, each containing `QTensorFormerAttention` with `TensorTrainLinear` projections, `QTensorFormerMLP` with `TensorTrainLinear` `gate_proj`/`up_proj`/`down_proj`, `QTensorFormerRMSNorm`, and `PIDResourceController`)
  - `model.norm` (`QTensorFormerRMSNorm`)
  - `lm_head` (`nn.Linear`)
- **Execution Path (`forward` & `generate`):**
  - Fully compatible with Hugging Face `CausalLMOutputWithPast` and `model.generate()`.
  - Uses `AdaptiveKVCache(Cache)` when `use_cache=True`.