AmauryLC

/

ys

Like 0

Hi Amaury,

Pretraining a clean causal model from scratch on French corpora (FineWeb-2, French Wikipedia, French-PD-Books) using the LLMs-from-scratch foundation is a really neat initiative for the first model of the Ys series.

Looking at your exact parameter accounting (49,470,720 parameters) and architecture configuration:

1. The 78.1% vocabulary parameter tax:
   With a 50,304 vocabulary at hidden size 384 and untied embeddings, your input embedding and lm_head consume 19.32M parameters each, totaling 38.63M parameters.
   That means 78.1% of your entire 49.5M parameter budget is locked in static lookup tables, leaving only ~10.6M parameters (less than 22%) for actual sequence modeling across your 6 transformer blocks.
   Each transformer block costs roughly 1.77M parameters. The static lookup tables consume the parameter equivalent of nearly 22 full layers of compute.
   Decoupling the vocabulary via low-rank projection (50,304 -> 64 -> 384 = ~3.24M params) or tying and factorizing reclaims over 30M parameters. Reallocating those weights into the backbone allows scaling from 6 layers to 18-20+ layers within the exact same 49.5M footprint.

2. French syntactic agreement and grammatical depth:
   French morphology features extensive gender/number agreements, verb conjugations, and elisions that require deep non-linear routing across tokens. A 6-layer stack has limited compositional depth to resolve multi-clause dependencies.
   In an open architecture project called Maba (101M reference model: https://huggingface.co/AndrewThompson1233/maba-v1-architecture), we address depth ceilings in small models using deterministic 2-pass block recycling:
   Routing hidden states through physical blocks twice with Split RMSNorm (distinct scale vectors for pass 0 and pass 1) expands depth from 6 to 12 effective layers at zero parameter overhead. This extra depth gives the model the representational capacity needed to maintain grammatical coherence across longer generations.

3. GPT-2 tokenizer fertility on French text:
   The standard GPT-2 BPE tokenizer has high fertility on French due to accent fragmentation (splitting characters like e, a, and c into multiple byte tokens). Training a custom 16k-32k French BPE tokenizer for the next Ys iteration would both cut sequence length and drastically reduce the raw embedding dimension.

Are you planning to test tied embeddings or a custom French tokenizer for the next iteration in the Ys series?

Best,
Andrew