Sparse Participation Fine-Tuning: Training Small Models Without Moving the Whole Beast

Author: Roy Shawn Colca Jr.
Organization: Convergent Intelligence LLC
Method: CIxOpt + targeted upper-layer sparse adaptation
Models tested: LFM-family checkpoints, Gemma 3 270M, GPT-X2-125M derivatives
Task type: causal language modeling / reasoning-style supervised fine-tuning

Abstract

Fine-tuning small language models is usually treated as a choice between two familiar paths: full-model training or adapter-based tuning. Full-model tuning gives broad control but can be expensive, unstable, and destructive to a compact model’s pretrained structure. Adapter methods are efficient but sometimes feel bolted onto the side of the model rather than integrated into its internal dynamics.

This article describes a third experimental path: sparse participation fine-tuning with heterogeneous optimizer routing.

The core idea is simple:

text Do not train everything. Do not treat every tensor the same. Move only the surfaces that matter. Route those surfaces according to what they are.

Using this method, I trained compact and hybrid language models by freezing most of the pretrained backbone while selectively updating upper-layer attention projections, convolutional projection surfaces, normalization parameters, or model-specific adaptation bands. These trainable surfaces were then optimized using CIxOpt, a custom optimizer that routes different parameter classes through different update strategies.

The goal is not to claim a final benchmark victory. The goal is to establish a practical, testable training pattern for small-model adaptation under limited compute.

Why Sparse Participation?

Small models are fragile.

When a 270M or 350M parameter model is pushed too aggressively, the update does not merely “teach” the model. It can overwrite the compressed structure that made the model useful in the first place. The model starts to drift, repeat, flatten, or become a strange soup of the dataset’s surface patterns.

The intuition behind sparse participation is that not all layers have the same role.

Lower layers often act as representational bedrock. They encode token patterns, syntax, local structure, and general language mechanics. Upper layers are often more involved in response formation, reasoning style, task behavior, and instruction alignment.

So instead of updating the entire model, sparse participation does this:

text lower layers -> preserve middle layers -> mostly preserve upper layers -> selectively adapt sensitive params -> update carefully large matrices -> update efficiently

This creates a fine-tuning pattern that is lightweight, controlled, and less likely to destroy the pretrained substrate.

The Method in One Sentence

Sparse participation fine-tuning trains only selected model surfaces while CIxOpt routes each trainable parameter through an optimizer strategy suited to its tensor type, size, and architectural role.

CIxOpt: Heterogeneous Optimizer Routing

CIxOpt is a custom optimizer framework designed around the fact that language model parameters are not all the same species of object.

A large projection matrix, a normalization vector, an embedding table, and an experimental positional governor should not necessarily receive identical optimizer behavior.

CIxOpt supports:

• AdamW-style adaptive updates
• Lion-style sign momentum
• AdaMax-compatible routing
• ASGD-style averaging
• Optional low-rank projected momentum
• Gradient centralization
• Decoupled weight decay
• fp32 optimizer state for bf16/fp16 safety
• Discrepancy-aware caution filtering for sign updates
• Parameter-name-aware routing

The default routing idea is:

text large projection matrices -> Lion-style sign momentum normalization / sensitive params -> AdamW-style updates embedding / lm-head surfaces -> conservative adaptive routing experimental control modules -> precise adaptive routing

This is not just optimization seasoning. It is part of the training philosophy. The optimizer becomes architecture-aware instead of acting like a single hammer looking for nails.

Sparse Participation Pattern

The general training flow looks like this:

text 1. Load pretrained model 2. Freeze all parameters 3. Select trainable adaptation surfaces 4. Register named parameters 5. Route parameters by module type 6. Train with causal LM loss 7. Clip gradients 8. Save full model or adapted checkpoint

The selection step is where the method becomes model-specific.

For an LFM-style hybrid model, the target surfaces may include:

text upper attention q/k/v/out projections upper short-conv in/out projections q/k normalization surfaces final normalization

For a Gemma-style compact transformer, the target surfaces may include:

text upper attention projections upper MLP projections RMSNorm surfaces possibly final norm

For GPT-X2-style custom models, the target surfaces may include:

text upper attention projections upper MLP projections normalization surfaces optional Symplectic Metric-RoPE governor modules

The principle is the same across architectures:

text Preserve the lower substrate. Adapt the behavior-shaping surfaces. Route each trainable tensor intelligently.

Example: Freezing the Backbone

A simplified version of the sparse selection pattern:

python def freeze_model(model): for _, p in model.named_parameters(): p.requires_grad = False def select_upper_layers( model, train_from_layer: int, target_keywords, ): trainable = [] for name, p in model.named_parameters(): layer_id = None parts = name.split(".") for i, part in enumerate(parts): if part == "layers" and i + 1 < len(parts): if parts[i + 1].isdigit(): layer_id = int(parts[i + 1]) break if layer_id is None: continue if layer_id >= train_from_layer and any(k in name for k in target_keywords): p.requires_grad = True trainable.append((name, p)) return trainable

Example target list:

python target_keywords = [ "q_proj", "k_proj", "v_proj", "o_proj", "out_proj", "gate_proj", "up_proj", "down_proj", "input_layernorm", "post_attention_layernorm", "q_layernorm", "k_layernorm", "norm", ]

This gives a compact adaptation band without waking the entire model.

Example: CIxOpt Routing

After selecting trainable parameters, CIxOpt can route by name:

python routing_rules = { "q_layernorm": "adamw", "k_layernorm": "adamw", "norm": "adamw", "q_proj": "lion", "k_proj": "lion", "v_proj": "lion", "o_proj": "lion", "out_proj": "lion", "gate_proj": "lion", "up_proj": "lion", "down_proj": "lion", "embed": "adamax", "lm_head": "adamax", }

The optimizer is then initialized over only the trainable parameter set:

python optimizer = CIxOpt( trainable_named_params, lr=8e-5, betas=(0.9, 0.99), weight_decay=0.01, strategy="auto", grad_centralize=True, caution=True, state_dtype=torch.float32, foreach=True, ) optimizer.register_param_names(model.named_parameters()) optimizer.autofind_and_route_params( trainable_named_params, custom_rules=routing_rules, )

This is the key split:

text selection decides what can move routing decides how it moves

Training Loss

The setup uses standard causal language modeling loss.

Padding labels are masked with -100:

python labels = input_ids.clone() labels[attention_mask == 0] = -100

For instruction-style datasets, assistant-only masking is usually preferable when the dataset structure supports it. In simpler continued-pretraining or mixed text settings, full causal loss over formatted text is acceptable.

Why This Worked Well Enough to Continue

The method is attractive because it provides a practical compromise:

Sparse participation is not magic. It is a controlled intervention.

It gives the model enough movement to adapt while reducing the risk of catastrophic drift.

Models Tested

This method has been applied experimentally to:

LFM-family models

The LFM setup is especially interesting because hybrid attention/convolution backbones have distinct adaptation surfaces. Rather than training every block, the method targeted upper attention projections and liquid short-convolution in/out projections.

Gemma 3 270M

Gemma 270M is a strong test case for small-model adaptation because its size makes overtraining easy. Sparse participation provides a way to shape behavior without treating the entire compact model as disposable clay.

GPT-X2-125M derivatives

GPT-X2 experiments included custom architecture work, long-context behavior, grouped-query attention, and optional Symplectic Metric-RoPE modules. This makes GPT-X2 useful for testing not only sparse fine-tuning, but also optimizer behavior around experimental architectural components.

Design Philosophy

This method follows a simple rule:

text The pretrained model is not raw material. It is structure.

The point of fine-tuning is not to flood the model with gradients until it remembers the dataset. The point is to identify where behavior can be changed with the least unnecessary damage.

That means treating the model less like a block of stone and more like a living circuit board. Some traces carry language structure. Some gates shape response behavior. Some surfaces are delicate. Some can absorb motion.

Sparse participation is a way of respecting that topology.

Practical Recommendations

For small models, I recommend starting with:

text learning rate: 5e-5 to 1.2e-4 batch size: as memory allows gradient accumulation: 8 to 16 max grad norm: 1.0 optimizer state: fp32 model dtype: bf16 if supported lower layers: frozen upper layers: selectively trainable lm_head: frozen at first embeddings: frozen at first

I also recommend avoiding full-logit finite checks inside every training step. Checking loss stability is usually enough during normal training. Full-logit checks are better used as a probe before training begins.

What This Is Not

This is not a claim that sparse participation beats LoRA, QLoRA, AdamW, or full fine-tuning across all settings.

It is not a benchmark paper yet.

It is not a universal recipe.

It is a working experimental method for constrained fine-tuning of compact and hybrid causal language models.

The next step is comparative evaluation.

Evaluation Plan

Future testing should compare:

text base model vs sparse-tuned model CIxOpt vs AdamW upper-layer sparse tuning vs full fine-tuning sparse participation vs LoRA assistant-only masking vs full causal masking different train-from-layer thresholds different projection targets

Useful benchmark directions include:

• Held-out perplexity
• Instruction following
• Repetition rate
• Short-form reasoning tasks
• Long-context recall probes
• Human side-by-side preference testing
• Stability under temperature changes
• Drift from base model behavior

Conclusion

Sparse participation fine-tuning is a practical method for adapting small language models without moving the whole architecture at once.

The key idea is:

text freeze broadly train selectively route intelligently measure honestly

CIxOpt adds the second half of the method by making optimizer behavior depend on parameter role instead of treating the entire model as one undifferentiated mass.

For small models, this matters. They do not have endless redundancy. Every update has weight. Every bad gradient leaves fingerprints.

Sparse participation is a way to adapt compact models with a lighter hand, a sharper blade, and less collateral damage.

Suggested Citation

bibtex @misc{colca_sparse_participation_cixopt_2026, title = {Sparse Participation Fine-Tuning with Heterogeneous Optimizer Routing}, author = {Roy Shawn Colca Jr.}, year = {2026}, publisher = {Hugging Face Articles}, note = {Convergent Intelligence LLC research note} }

Disclaimer

This method is experimental. Models trained with this approach should be evaluated carefully before deployment. Outputs from fine-tuned models may still hallucinate, repeat, fail safety expectations, or produce incorrect reasoning. Human review remains necessary for factual, technical, legal, medical, financial, operational, or safety-critical use.