KenT-300M: 8 layers, d_x=1024, N=32, D=240

README.md

@@ -22,18 +22,19 @@ tags:
 
 # KenT
 
-KenT is a recurrent language model from the SSM family. KenT concentrates its capacity in depth: many compact layers, each with a single transformation block, and a light learned-forgetting memory.
+KenT is a recurrent language model from the SSM family. KenT concentrates its capacity in a very wide spectral representation: few layers, each with a single transformation block, and a light learned-forgetting memory.
 
 > This repo contains `trust_remote_code` code; weights are stored in **FP16**.
 
 ## Architectural Philosophy
 
-- **Deep stack of compact layers.** KenT stacks many identical recurrent layers (24 in this ~300M model), each containing exactly
+- **Shallow stack, wide spectrum.** KenT stacks a few identical recurrent layers (8 in this ~300M model), each containing exactly
   one transformation block. The layers are connected in sequence through the representation stream, and the last layer produces
-  the final output. Depth is prioritized: the deeper, the better, while width is kept at a level that stays balanced with it.
+  the final output. The representation stream feeding `lm_head` is wide (1024), and each layer's Bilinear projection expands it
+  into a large resolution axis (N = 32 DCT positions) with enough features per position (D = 240) before the DCT.
 - **Invest in the transformation core.** Capacity is concentrated in the "hidden" part, where features are actually transformed,
-  rather than in the vocabulary table. Parameters were scaled from ~100M to ~300M mainly by adding depth, with width unchanged,
-  so that both axes stay balanced (width still matters, but is not where the extra capacity went).
+  rather than in the vocabulary table. Parameters were scaled from ~100M to ~300M by widening the representation stream and the
+  spectral axes (N and D together), not by stacking more layers.
 - **Two streams, two roles.** One stream is a representation carrying information about the current token; the other is a memory
   stream that aggregates history. The two streams meet only through learned projections and are never mixed directly.
 - **Frequency as a fixed foundation, learning where it's needed.** Features are processed in the cosine-coefficient space using a
@@ -42,8 +43,8 @@ KenT is a recurrent language model from the SSM family. KenT concentrates its ca
 - **Memory with learned forgetting.** History is summarized with an exponential moving average
   `H_t = (1 - a) H_{t-1} + a u_t`, where `a` is a single learned scalar per layer. Its initial value decreases with depth
   (shallow layers forget fast, deep layers remember longer). The decoding state has **constant** size regardless of context length.
-- **Depth is the main axis.** There is no inner chain of blocks inside a layer any more: feature depth and memory timescale both
-  grow with the number of layers.
+- **Width is the main axis.** There is no inner chain of blocks inside a layer: each layer applies one DCT/SwiGLU block over a
+  large N x D grid, and memory timescale still varies across the 8 layers.
 - **Chunk-parallelizable.** The moving average is a linear recurrence, so a long document can be split into blocks/chunks,
   processed concurrently, and merged with an affine scan, yielding the same result as sequential processing.
 

config.json

@@ -9,11 +9,11 @@
     "AutoModelForCausalLM": "model.KentForCausalLM"
   },
   "vocab_size": 50257,
-  "hidden_size": 512,
-  "num_hidden_layers": 24,
-  "resolution_size": 8,
-  "feature_dim": 512,
-  "intermediate_size": 2048,
+  "hidden_size": 1024,
+  "num_hidden_layers": 8,
+  "resolution_size": 32,
+  "feature_dim": 240,
+  "intermediate_size": 960,
   "rms_norm_eps": 1e-05,
   "use_bias": true,
   "initializer_range": 0.02,