Root cause of train_loss=0.0 and eval_loss=nan:
- no_init_weights() makes nn.Linear.reset_parameters() a no-op
- trans_input nn.Linear weights stay as torch.empty() (uninitialized)
- if weights=0: trans_input output=0, cal_attention does x/x.norm()=0/0=NaN
- NaN loss filtered by logging_nan_inf_filter -> reported as train_loss=0.0
Fixes:
- Re-initialize trans_input Linear layers with kaiming_uniform_ after from_pretrained
- Re-initialize prompt_key with uniform(-1,1)
- Add epsilon (1e-12) to norm division in cal_attention to prevent 0/0
- Use gradient_checkpointing_enable(use_reentrant=False) for T5-large compat
- Add forward+backward gradient test in sanity check
- Add trans_input weight diagnostics in sanity check
fix: pass attention_mask directly to model.generate(), not via GenerationConfig
transformers 4.40 removed Trainer._pad_across_processes().
Replace all 32 call sites (2 per trainer × 16 trainers across both repos)
with self.accelerator.pad_across_processes() which is the underlying
implementation. Behavior is identical: single-GPU returns tensor unchanged,
multi-GPU pads to max length across processes.
Affected files in root_gainlora/src/ and improve_gainlora/src/:
cl_trainer_gainlora_inflora.py, cl_trainer_inflora.py,
cl_trainer_inflora_llama.py, cl_trainer_olora.py, cl_trainer_olora_llama.py,
cl_trainer_gainlora_olora.py, cl_trainer_gainlora_olora_llama.py,
cl_trainer_gainlora_inflora_llama.py
fix: synced_gpus must be passed to generate(), not GenerationConfig
transformers 4.40+ raises ValueError if synced_gpus is included in
GenerationConfig(**gen_kwargs). Pop it out before constructing the
config and pass it directly to every model.generate() call instead.
Affects all cl_trainer_*.py files in both root and improve codebases:
- cl_trainer_gainlora_inflora.py
- cl_trainer_inflora.py
- cl_trainer_olora.py
- cl_trainer_gainlora_olora_llama.py
- cl_trainer_gainlora_inflora_llama.py
- cl_trainer_inflora_llama.py
- cl_trainer_gainlora_olora.py
Does not affect T5-large scripts (same fix, backward compatible).
fix: LoRALayer.forward reshape incompatible with T5-small
The reshape result.reshape(x.shape[0], -1, x.shape[-1]) assumed
out_features (inner_dim) == in_features (d_model), which holds for
T5-large (1024==1024) but fails for T5-small (384 != 512).
Fix: remove the no-op reshape and return result directly.
Shape (batch, seq, out_features=inner_dim) is correct for all model sizes.
Does not change T5-large behavior.
fix: restore _set_gradient_checkpointing + enable_input_require_grads for gradient checkpointing
