v10

improve_gainlora/IDEA_Overall.md

@@ -566,6 +566,8 @@ $A_t$ này đảm bảo capture **variance task-relevant tối đa** trong null-
 | C4: Preconditioner | `precompute_preconditioners()` → eigendecomposition | `cl_trainer_specroute.py` |
 | **C5: Data-informed init** | **`pre_task_data_collection()` → `eigh(Q@C@Q)` → set `lora_A.data`** | **`cl_trainer_specroute.py`** |
 | C5: Fallback | max eigval < 1e-6 → skip C5, keep Kaiming + InfLoRA projection | `cl_trainer_specroute.py` |
+| **V10a: Learned Routing** | **`Trans_input` + `prompt_key` gating with exact post-step GPM constraints** | **`t5_specroute.py` & `cl_trainer_specroute.py`** |
+| **V10b: Grassmann Routing** | **Geometry-based routing via Grassmannian distance on batch principal subspaces** | **`t5_specroute.py`** |
 
 ---
 

improve_gainlora/SPECROUTE_IDEA.md

@@ -433,5 +433,7 @@ The Routing–Protection Duality Theorem (Theorem 1) assumes $h \in \mathrm{span
 | **Adaptive GPM threshold** | ⬜ Pending | Relax constraint for later tasks to preserve capacity |
 | **Same-domain routing** | ⬜ Research | Geometry-based (no labels, no data) task similarity for routing |
 | **Rank expansion** | ⬜ Pending | Increase r for later tasks to compensate null-space shrinkage |
+| **V10a Learned Routing** | ✅ Implemented | Relax parameter-free constraint; use ROOT's MLP & prompt keys with strict GPM |
+| **V10b Grassmann Routing** | ✅ Implemented | Geometry-based routing using Grassmannian distance on batch principal subspaces |
 
-**Key constraint**: Any direction must keep zero-replay AND maintain Routing–Protection Duality narrative (SpecRoute's core theoretical contribution). Oracle routing during training is valid; inference routing must remain parameter-free for the claim to hold.
+**Key constraint**: Any direction must keep zero-replay AND maintain Routing–Protection Duality narrative (SpecRoute's core theoretical contribution). Oracle routing during training is valid; inference routing must remain parameter-free for the claim to hold (V10b achieves this, V10a relaxes it for empirical bounding).

improve_gainlora/T5_small/gen_script_long_order3_t5_small_specroute_v10a.sh

@@ -0,0 +1,906 @@
+#!/bin/bash
+#SBATCH -J cl
+#SBATCH -o cl-%j.out
+#SBATCH -p compute
+#SBATCH -N 1
+#SBATCH -t 20:00:00
+#SBATCH --mem 128G
+#SBATCH --gres=gpu:2
+
+export CUDA_DEVICE_ORDER="PCI_BUS_ID"
+
+port=$(shuf -i25000-30000 -n1)
+
+# ============================================================
+# Auto-detect GPU count and type for optimal parallelism
+# ============================================================
+NUM_GPUS=$(nvidia-smi -L 2>/dev/null | wc -l)
+GPU_MEM=$(nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits 2>/dev/null | head -1)
+
+if [ -z "$GPU_MEM" ]; then
+    echo "ERROR: No GPU detected!"
+    exit 1
+fi
+
+# Determine GPU type
+if [ "$GPU_MEM" -lt 20000 ]; then
+    IS_T4=1
+    echo "[GPU] Detected T4 GPUs (${GPU_MEM}MB VRAM each)"
+else
+    IS_T4=0
+    echo "[GPU] Detected high-memory GPUs (${GPU_MEM}MB VRAM each)"
+fi
+
+# Determine parallelism strategy
+if [ "$IS_T4" -eq 1 ] && [ "$NUM_GPUS" -ge 2 ]; then
+    GPU_MODE="t4_2gpu"
+    GPU_IDS="0,1"
+    FP16_FLAG=""
+    echo "[GPU] Strategy: 2x T4 DataParallel + fp32 + gradient_checkpointing"
+elif [ "$IS_T4" -eq 1 ]; then
+    GPU_MODE="t4_1gpu"
+    GPU_IDS="${1:-0}"
+    FP16_FLAG=""
+    echo "[GPU] Strategy: 1x T4 + fp32 + gradient_checkpointing"
+else
+    GPU_MODE="a100"
+    GPU_IDS="${1:-0}"
+    FP16_FLAG=""
+    echo "[GPU] Strategy: A100 (single GPU, fp32)"
+fi
+
+echo "[GPU] Using CUDA_VISIBLE_DEVICES=$GPU_IDS"
+echo "============================================================"
+echo ""
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/yelp \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/1-yelp \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10a \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode learned \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/1-yelp/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --load_checkpoint_from logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/1-yelp/saved_weights/trans_input.pt \
+   --previous_prompt_key_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/1-yelp/saved_weights/prompts_keys_till_now.pt \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/1-yelp/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/amazon \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/2-amazon \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10a \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_amazon \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode learned \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/2-amazon/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --load_checkpoint_from logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/2-amazon/saved_weights/trans_input.pt \
+   --previous_prompt_key_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/2-amazon/saved_weights/prompts_keys_till_now.pt \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/2-amazon/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/mnli \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/3-mnli \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10a \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_mnli \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode learned \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/3-mnli/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --load_checkpoint_from logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/3-mnli/saved_weights/trans_input.pt \
+   --previous_prompt_key_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/3-mnli/saved_weights/prompts_keys_till_now.pt \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/3-mnli/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/cb \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/4-cb \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10a \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_cb \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode learned \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/4-cb/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --load_checkpoint_from logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/4-cb/saved_weights/trans_input.pt \
+   --previous_prompt_key_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/4-cb/saved_weights/prompts_keys_till_now.pt \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/4-cb/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/copa \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/5-copa \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10a \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_copa \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode learned \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/5-copa/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --load_checkpoint_from logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/5-copa/saved_weights/trans_input.pt \
+   --previous_prompt_key_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/5-copa/saved_weights/prompts_keys_till_now.pt \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/5-copa/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/qqp \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/6-qqp \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10a \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_qqp \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode learned \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/6-qqp/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --load_checkpoint_from logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/6-qqp/saved_weights/trans_input.pt \
+   --previous_prompt_key_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/6-qqp/saved_weights/prompts_keys_till_now.pt \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/6-qqp/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/rte \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/7-rte \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10a \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_rte \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode learned \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/7-rte/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --load_checkpoint_from logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/7-rte/saved_weights/trans_input.pt \
+   --previous_prompt_key_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/7-rte/saved_weights/prompts_keys_till_now.pt \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/7-rte/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/imdb \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/8-imdb \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10a \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_imdb \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode learned \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/8-imdb/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --load_checkpoint_from logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/8-imdb/saved_weights/trans_input.pt \
+   --previous_prompt_key_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/8-imdb/saved_weights/prompts_keys_till_now.pt \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/7-rte/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/8-imdb/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/sst2 \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/9-sst2 \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10a \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_sst2 \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode learned \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/9-sst2/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --load_checkpoint_from logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/9-sst2/saved_weights/trans_input.pt \
+   --previous_prompt_key_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/9-sst2/saved_weights/prompts_keys_till_now.pt \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/7-rte/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/8-imdb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/9-sst2/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/dbpedia \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/10-dbpedia \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10a \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_dbpedia \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode learned \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/10-dbpedia/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --load_checkpoint_from logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/10-dbpedia/saved_weights/trans_input.pt \
+   --previous_prompt_key_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/10-dbpedia/saved_weights/prompts_keys_till_now.pt \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/7-rte/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/8-imdb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/9-sst2/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/10-dbpedia/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/agnews \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/11-agnews \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10a \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_agnews \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode learned \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/11-agnews/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --load_checkpoint_from logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/11-agnews/saved_weights/trans_input.pt \
+   --previous_prompt_key_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/11-agnews/saved_weights/prompts_keys_till_now.pt \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/7-rte/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/8-imdb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/9-sst2/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/10-dbpedia/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/11-agnews/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/yahoo \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/12-yahoo \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10a \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_yahoo \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode learned \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/12-yahoo/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --load_checkpoint_from logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/12-yahoo/saved_weights/trans_input.pt \
+   --previous_prompt_key_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/12-yahoo/saved_weights/prompts_keys_till_now.pt \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/7-rte/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/8-imdb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/9-sst2/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/10-dbpedia/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/11-agnews/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/12-yahoo/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/multirc \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/13-multirc \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10a \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_multirc \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode learned \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/13-multirc/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --load_checkpoint_from logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/13-multirc/saved_weights/trans_input.pt \
+   --previous_prompt_key_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/13-multirc/saved_weights/prompts_keys_till_now.pt \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/7-rte/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/8-imdb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/9-sst2/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/10-dbpedia/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/11-agnews/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/12-yahoo/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/13-multirc/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/boolq \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/14-boolq \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10a \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_boolq \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode learned \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/14-boolq/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --load_checkpoint_from logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/14-boolq/saved_weights/trans_input.pt \
+   --previous_prompt_key_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/14-boolq/saved_weights/prompts_keys_till_now.pt \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/7-rte/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/8-imdb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/9-sst2/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/10-dbpedia/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/11-agnews/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/12-yahoo/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/13-multirc/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/14-boolq/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/wic \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/15-wic \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10a \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_wic \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode learned \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10a/outputs/15-wic/checkpoint*
+
+sleep 5

improve_gainlora/T5_small/gen_script_long_order3_t5_small_specroute_v10b.sh

@@ -0,0 +1,878 @@
+#!/bin/bash
+#SBATCH -J cl
+#SBATCH -o cl-%j.out
+#SBATCH -p compute
+#SBATCH -N 1
+#SBATCH -t 20:00:00
+#SBATCH --mem 128G
+#SBATCH --gres=gpu:2
+
+export CUDA_DEVICE_ORDER="PCI_BUS_ID"
+
+port=$(shuf -i25000-30000 -n1)
+
+# ============================================================
+# Auto-detect GPU count and type for optimal parallelism
+# ============================================================
+NUM_GPUS=$(nvidia-smi -L 2>/dev/null | wc -l)
+GPU_MEM=$(nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits 2>/dev/null | head -1)
+
+if [ -z "$GPU_MEM" ]; then
+    echo "ERROR: No GPU detected!"
+    exit 1
+fi
+
+# Determine GPU type
+if [ "$GPU_MEM" -lt 20000 ]; then
+    IS_T4=1
+    echo "[GPU] Detected T4 GPUs (${GPU_MEM}MB VRAM each)"
+else
+    IS_T4=0
+    echo "[GPU] Detected high-memory GPUs (${GPU_MEM}MB VRAM each)"
+fi
+
+# Determine parallelism strategy
+if [ "$IS_T4" -eq 1 ] && [ "$NUM_GPUS" -ge 2 ]; then
+    GPU_MODE="t4_2gpu"
+    GPU_IDS="0,1"
+    FP16_FLAG=""
+    echo "[GPU] Strategy: 2x T4 DataParallel + fp32 + gradient_checkpointing"
+elif [ "$IS_T4" -eq 1 ]; then
+    GPU_MODE="t4_1gpu"
+    GPU_IDS="${1:-0}"
+    FP16_FLAG=""
+    echo "[GPU] Strategy: 1x T4 + fp32 + gradient_checkpointing"
+else
+    GPU_MODE="a100"
+    GPU_IDS="${1:-0}"
+    FP16_FLAG=""
+    echo "[GPU] Strategy: A100 (single GPU, fp32)"
+fi
+
+echo "[GPU] Using CUDA_VISIBLE_DEVICES=$GPU_IDS"
+echo "============================================================"
+echo ""
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/yelp \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/1-yelp \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10b \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode grassmann \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/1-yelp/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/1-yelp/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/amazon \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/2-amazon \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10b \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_amazon \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode grassmann \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/2-amazon/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/2-amazon/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/mnli \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/3-mnli \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10b \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_mnli \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode grassmann \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/3-mnli/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/3-mnli/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/cb \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/4-cb \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10b \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_cb \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode grassmann \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/4-cb/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/4-cb/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/copa \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/5-copa \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10b \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_copa \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode grassmann \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/5-copa/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/5-copa/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/qqp \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/6-qqp \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10b \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_qqp \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode grassmann \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/6-qqp/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/6-qqp/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/rte \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/7-rte \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10b \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_rte \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode grassmann \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/7-rte/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/7-rte/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/imdb \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/8-imdb \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10b \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_imdb \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode grassmann \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/8-imdb/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/7-rte/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/8-imdb/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/sst2 \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/9-sst2 \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10b \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_sst2 \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode grassmann \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/9-sst2/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/7-rte/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/8-imdb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/9-sst2/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/dbpedia \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/10-dbpedia \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10b \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_dbpedia \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode grassmann \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/10-dbpedia/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/7-rte/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/8-imdb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/9-sst2/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/10-dbpedia/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/agnews \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/11-agnews \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10b \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_agnews \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode grassmann \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/11-agnews/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/7-rte/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/8-imdb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/9-sst2/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/10-dbpedia/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/11-agnews/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/yahoo \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/12-yahoo \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10b \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_yahoo \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode grassmann \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/12-yahoo/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/7-rte/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/8-imdb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/9-sst2/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/10-dbpedia/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/11-agnews/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/12-yahoo/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/multirc \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/13-multirc \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10b \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_multirc \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode grassmann \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/13-multirc/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/7-rte/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/8-imdb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/9-sst2/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/10-dbpedia/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/11-agnews/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/12-yahoo/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/13-multirc/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/boolq \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/14-boolq \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10b \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_boolq \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode grassmann \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/14-boolq/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/7-rte/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/8-imdb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/9-sst2/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/10-dbpedia/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/11-agnews/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/12-yahoo/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/13-multirc/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/14-boolq/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/wic \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/15-wic \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v10b \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_wic \
+   --evaluation_strategy epoch \
+   --save_strategy epoch \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq -1 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --routing_mode grassmann \
+   --threshold 0.995 \
+   --transthreshold 0.995 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v10b/outputs/15-wic/checkpoint*
+
+sleep 5

improve_gainlora/discuss_AI.txt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6d90f4755ebe7ef6c899e0eef85a1866bc5629ae25ce0ec3fd56616fa92644c4
-size 3934
+oid sha256:199005383467e5e167f68250698a393b3ab431cad10597bb6377b1cb52019985
+size 20466

improve_gainlora/generate_v10_scripts.py

@@ -0,0 +1,45 @@
+import re
+import os
+
+with open("T5_small/gen_script_long_order3_t5_small_gainlora_inflora.sh", "r") as f:
+    gainlora_content = f.read()
+
+with open("T5_small/gen_script_long_order3_t5_small_specroute.sh", "r") as f:
+    specroute_content = f.read()
+
+def create_script(mode, suffix):
+    new_content = specroute_content.replace("gen_script_long_order3_t5_small_specroute", f"gen_script_long_order3_t5_small_specroute_{suffix}")
+    new_content = new_content.replace("--model_name specroute \\", f"--model_name specroute \\\n   --routing_mode {mode} \\")
+    
+    if mode == "learned":
+        # Extract previous_prompt_key_path and load_checkpoint_from from gainlora
+        blocks = new_content.split("python src/run_t5.py")
+        final_content = blocks[0]
+        
+        gainlora_blocks = gainlora_content.split("python src/run_t5.py")
+        
+        for i in range(1, len(blocks)):
+            block = blocks[i]
+            gainlora_block = gainlora_blocks[i]
+            
+            m1 = re.search(r'--load_checkpoint_from\s+([^\s\\]+)', gainlora_block)
+            m2 = re.search(r'--previous_prompt_key_path\s+([^\s\\]+)', gainlora_block)
+            
+            args_to_add = ""
+            if m1:
+                path1 = m1.group(1).replace("gen_script_long_order3_t5_small_gainlora_inflora", "gen_script_long_order3_t5_small_specroute_v10a")
+                args_to_add += f"   --load_checkpoint_from {path1} \\\n"
+            if m2:
+                path2 = m2.group(1).replace("gen_script_long_order3_t5_small_gainlora_inflora", "gen_script_long_order3_t5_small_specroute_v10a")
+                args_to_add += f"   --previous_prompt_key_path {path2} \\\n"
+                
+            final_content += "python src/run_t5.py" + block.replace("   --do_train \\\n", f"   --do_train \\\n{args_to_add}")
+            
+        new_content = final_content
+
+    with open(f"T5_small/gen_script_long_order3_t5_small_specroute_{suffix}.sh", "w") as f:
+        f.write(new_content)
+    print(f"Created T5_small/gen_script_long_order3_t5_small_specroute_{suffix}.sh")
+
+create_script("learned", "v10a")
+create_script("grassmann", "v10b")

improve_gainlora/src/cl_trainer_specroute.py

@@ -81,6 +81,51 @@ class PeriodicGCCallback(TrainerCallback):
         return control
 
 
+class TransInputGPMCallback(TrainerCallback):
+    """V10a: Apply GPM projection to trans_input and prompt_key after optimizer step."""
+    def __init__(self, trainer):
+        self.trainer = trainer
+
+    def on_step_begin(self, args: TrainingArguments, state: TrainerState, control: TrainerControl, **kwargs):
+        if getattr(self.trainer, "cur_task_id", 0) > 1 and getattr(self.trainer.model.encoder, "routing_mode", "") == "learned":
+            from copy import deepcopy
+            self.trainer._old_trans_input_0 = deepcopy(self.trainer.model.encoder.trans_input[0].weight.detach())
+            self.trainer._old_trans_input_1 = deepcopy(self.trainer.model.encoder.trans_input[2].weight.detach())
+            self.trainer._old_prompt_key = deepcopy(self.trainer.model.encoder.prompt_key.detach())
+
+    def on_step_end(self, args: TrainingArguments, state: TrainerState, control: TrainerControl, **kwargs):
+        if getattr(self.trainer, "cur_task_id", 0) > 1 and getattr(self.trainer.model.encoder, "routing_mode", "") == "learned":
+            if not hasattr(self.trainer, "feature_trans_mat") or not self.trainer.feature_trans_mat:
+                return
+            
+            from copy import deepcopy
+            new_trans_input_0 = deepcopy(self.trainer.model.encoder.trans_input[0].weight.detach())
+            new_trans_input_1 = deepcopy(self.trainer.model.encoder.trans_input[2].weight.detach())
+            new_trans_input_0norm = new_trans_input_0.norm(dim=1, keepdim=True)
+            new_trans_input_1norm = new_trans_input_1.norm(dim=1, keepdim=True)
+
+            new_prompt_key = deepcopy(self.trainer.model.encoder.prompt_key.detach())
+            new_prompt_key_norm = new_prompt_key.norm(dim=1, keepdim=True)
+            
+            old_trans_input_0 = self.trainer._old_trans_input_0
+            old_trans_input_1 = self.trainer._old_trans_input_1
+            old_prompt_key = self.trainer._old_prompt_key
+            
+            for index in self.trainer.feature_trans_mat[0].keys():
+                new_trans_input_0[:,index*self.trainer.model.encoder.step:(index+1)*self.trainer.model.encoder.step] = self.trainer.model.encoder.trans_input[0].weight.detach()[:,index*self.trainer.model.encoder.step:(index+1)*self.trainer.model.encoder.step] - torch.mm(self.trainer.model.encoder.trans_input[0].weight.detach()[:,index*self.trainer.model.encoder.step:(index+1)*self.trainer.model.encoder.step]-old_trans_input_0[:,index*self.trainer.model.encoder.step:(index+1)*self.trainer.model.encoder.step], self.trainer.feature_trans_mat[0][index])
+                new_prompt_key[:,index*self.trainer.model.encoder.step:(index+1)*self.trainer.model.encoder.step] = self.trainer.model.encoder.prompt_key.detach()[:,index*self.trainer.model.encoder.step:(index+1)*self.trainer.model.encoder.step] - torch.mm(self.trainer.model.encoder.prompt_key.detach()[:,index*self.trainer.model.encoder.step:(index+1)*self.trainer.model.encoder.step]-old_prompt_key[:,index*self.trainer.model.encoder.step:(index+1)*self.trainer.model.encoder.step], self.trainer.feature_trans_mat[2][index])
+            new_trans_input_1 = self.trainer.model.encoder.trans_input[2].weight.detach() - torch.mm(self.trainer.model.encoder.trans_input[2].weight.detach()-old_trans_input_1, self.trainer.feature_trans_mat[1])
+
+            new_trans_input_0 = new_trans_input_0*new_trans_input_0norm / new_trans_input_0.norm(dim=1, keepdim=True).clamp(min=1e-12)
+            new_trans_input_1 = new_trans_input_1*new_trans_input_1norm / new_trans_input_1.norm(dim=1, keepdim=True).clamp(min=1e-12)
+            new_prompt_key = new_prompt_key*new_prompt_key_norm / new_prompt_key.norm(dim=1, keepdim=True).clamp(min=1e-12)
+
+            self.trainer.model.encoder.trans_input[0].weight.data.copy_(new_trans_input_0)
+            self.trainer.model.encoder.trans_input[2].weight.data.copy_(new_trans_input_1)
+            self.trainer.model.encoder.prompt_key.data.copy_(new_prompt_key)
+        return control
+
+
 class SpecRoute_Trainer(Seq2SeqTrainer):
 
     def __init__(self, model, args, train_dataset, cur_task_id, task_order,
@@ -89,6 +134,9 @@ class SpecRoute_Trainer(Seq2SeqTrainer):
                  lambda_entropy=0.0, use_preconditioning=False,
                  precond_eps=1e-6, entropy_warmup_ratio=0.1,
                  n_batches_c5=100):
+        if callbacks is None:
+            callbacks = []
+        callbacks.append(TransInputGPMCallback(self))
         super().__init__(
             model=model, args=args, train_dataset=train_dataset,
             eval_dataset=eval_dataset, tokenizer=tokenizer,
@@ -259,13 +307,14 @@ class SpecRoute_Trainer(Seq2SeqTrainer):
         print(f'[C5] Covariance collected for {len(self._task_covariance)} layers.')
 
     def load_previous_reg_matrix(self):
-        """Load LoRA GPM bases from previous task. No trans_input GPM needed."""
+        """Load LoRA GPM bases from previous task. Also load trans_input GPM if learned routing."""
         log_path = os.path.dirname(self.args.output_dir)
         local_dir = os.path.basename(self.args.output_dir)
         print(log_path)
 
         all_dirs = os.listdir(log_path)
         reg_matrix = []
+        reg_trans_matrix = []
         for all_dir in all_dirs:
             if not os.path.isdir(os.path.join(log_path, all_dir)):
                 continue
@@ -277,22 +326,38 @@ class SpecRoute_Trainer(Seq2SeqTrainer):
                             os.path.join(os.path.join(log_path, all_dir), "reg_{}.pt".format(i))
                         ))
                         i += 1
+                if getattr(self.model.encoder, "routing_mode", "") == "learned":
+                    reg_trans_matrix.append(torch.load(os.path.join(os.path.join(log_path, all_dir, 'trans_input'), "reg_0.pt"), weights_only=True))
+                    reg_trans_matrix.append(torch.load(os.path.join(os.path.join(log_path, all_dir, 'trans_input'), "reg_1.pt"), weights_only=True))
+                    reg_trans_matrix.append(torch.load(os.path.join(os.path.join(log_path, all_dir, 'trans_input'), "reg_2.pt"), weights_only=True))
+
                 print(os.path.join(log_path, all_dir))
                 print(len(reg_matrix))
                 break
-        return reg_matrix, eval(local_dir.split('-')[0]) - 1
+        return reg_matrix, reg_trans_matrix, eval(local_dir.split('-')[0]) - 1
 
     def get_reg_matrix(self):
         """
         Project current LoRA A into null-space of old tasks' GPM bases.
         No prompt_key/trans_input operations.
         """
-        self.feature_list, self._cur_task = self.load_previous_reg_matrix()
+        self.feature_list, self.feature_trans_list, self._cur_task = self.load_previous_reg_matrix()
 
         if len(self.feature_list) == 0:
             # First task: no constraints
             return
 
+        if getattr(self.model.encoder, "routing_mode", "") == "learned":
+            self.feature_trans_mat = []
+            for i in range(len(self.feature_trans_list)):
+                if i == 1:
+                    self.feature_trans_mat.append(torch.mm(self.feature_trans_list[i], self.feature_trans_list[i].T).to("cuda:0"))
+                else:
+                    feature_trans_mat = {}
+                    for index in self.feature_trans_list[i].keys():
+                        feature_trans_mat[index] = torch.mm(self.feature_trans_list[i][index], self.feature_trans_list[i][index].T).to("cuda:0")
+                    self.feature_trans_mat.append(feature_trans_mat)
+
         # Compute projection matrices for LoRA GPM
         self.feature_mat, i = [], 0
         for name, module in self.model.named_modules():
@@ -366,10 +431,9 @@ class SpecRoute_Trainer(Seq2SeqTrainer):
     def get_repsentation(self):
         """
         Collect LoRA input covariance and compute GPM bases via SVD.
-        ESA: Use constant threshold (no increasing schedule).
-        No trans_input features collected.
+        For V10a (learned routing), also collect trans_input covariance.
         """
-        self.feature_list, self._cur_task = self.load_previous_reg_matrix()
+        self.feature_list, self.feature_trans_list, self._cur_task = self.load_previous_reg_matrix()
 
         train_dataloader = self.get_train_dataloader()
         if isinstance(train_dataloader, DataLoader) and isinstance(train_dataloader.sampler, DistributedSampler):
@@ -382,6 +446,11 @@ class SpecRoute_Trainer(Seq2SeqTrainer):
                 module.get_feature = True
                 module.stage = 0
 
+        # V10a: enable trans_input covariance collection
+        if getattr(self.model.encoder, "routing_mode", "") == "learned":
+            self.model.encoder.get_chunk(self.args.chunk)
+            self.model.encoder.get_trans_feature = True
+
         print('begin get representation')
         with torch.no_grad():
             for step, inputs in enumerate(train_dataloader):
@@ -395,6 +464,10 @@ class SpecRoute_Trainer(Seq2SeqTrainer):
                     break
         print('end get representation')
 
+        # V10a: disable trans_input collection after forward pass
+        if getattr(self.model.encoder, "routing_mode", "") == "learned":
+            self.model.encoder.get_trans_feature = False
+
         # Collect LoRA covariance matrices
         mat_list = []
         for name, module in self.model.named_modules():
@@ -469,6 +542,32 @@ class SpecRoute_Trainer(Seq2SeqTrainer):
                     else:
                         self.feature_list[i][index] = from_dlpack(Ui.toDlpack())
 
+        # Collect trans_input GPM bases if learned routing
+        if getattr(self.model.encoder, "routing_mode", "") == "learned":
+            mat_trans_list = []
+            if self.model.encoder.matrix_trans_2.sum() != 0:
+                mat_trans_list.append(self.model.encoder.matrix_trans_1)
+                mat_trans_list.append(self.model.encoder.matrix_trans_2)
+                mat_trans_list.append(self.model.encoder.matrix_trans_3)
+                
+                self.feature_trans_list, self.feature_trans_mat = [], []
+                for i in range(len(mat_trans_list)):
+                    if i == 1:
+                        U, S, Vh = torch.linalg.svd(mat_trans_list[i].data, full_matrices=False)
+                        sval_total = (S**2).sum()
+                        sval_ratio = (S**2)/sval_total
+                        r = np.sum(np.cumsum(sval_ratio.cpu().numpy()) < self.args.transthreshold) + 1
+                        self.feature_trans_list.append(U[:,0:r].float())
+                    else:
+                        feature_trans_list, feature_trans_mat = {}, {}
+                        for index in mat_trans_list[i].keys():
+                            U, S, Vh = torch.linalg.svd(mat_trans_list[i][index].data, full_matrices=False)
+                            sval_total = (S**2).sum()
+                            sval_ratio = (S**2)/sval_total
+                            r = np.sum(np.cumsum(sval_ratio.cpu().numpy()) < self.args.transthreshold) + 1
+                            feature_trans_list[index] = U[:,0:r].float()
+                        self.feature_trans_list.append(feature_trans_list)
+
         print('-' * 40)
         print('Gradient Constraints Summary')
         print('-' * 40)
@@ -485,8 +584,12 @@ class SpecRoute_Trainer(Seq2SeqTrainer):
         for i in range(len(self.feature_list)):
             torch.save(self.feature_list[i], os.path.join(self.args.output_dir, 'reg_{}.pt'.format(i)))
 
-        # No trans_input GPM to save
-
+        # Save trans_input GPM bases
+        if getattr(self.model.encoder, "routing_mode", "") == "learned" and hasattr(self, "feature_trans_list"):
+            os.makedirs(os.path.join(self.args.output_dir, 'trans_input'), exist_ok=True)
+            for i in range(len(self.feature_trans_list)):
+                torch.save(self.feature_trans_list[i], os.path.join(self.args.output_dir, 'trans_input', 'reg_{}.pt'.format(i)))
+                
     # training_step: removed — base Seq2SeqTrainer handles it correctly.
     # SpecRoute has no memory replay or custom training_step logic.
 

improve_gainlora/src/run_t5.py

@@ -172,6 +172,10 @@ class ModelArguments:
                     "Adaptive bias = T*ln(alpha*n_old/(1-alpha)). Set 0 to use fixed training_bias."
         },
     )
+    previous_prompt_key_path: Optional[str] = field(
+        default=None,
+        metadata={"help": "Path to the previous key prompt layer."}
+    )
 
     # C4: Spectrally-Conditioned LoRA Training
     lambda_entropy: Optional[float] = field(
@@ -367,6 +371,10 @@ class TrainingArguments(Seq2SeqTrainingArguments):
         default='SAPT',
         metadata={"help": "models' name"}
     )
+    routing_mode: Optional[str] = field(
+        default='spectral',
+        metadata={"help": "Routing mode for SpecRoute"}
+    )
     chunk: Optional[int] = field(
         default=1,
         metadata={"help": "models' name"}
@@ -512,6 +520,7 @@ def main():
         'seq_len': data_args.max_source_length,
         'mlp_hidden_dim': model_args.mlp_hidden_dim,
         'attn_temperature': model_args.attn_temperature,
+        'routing_mode': training_args.routing_mode,
         'previous_lora_path': model_args.previous_lora_path,
         'previous_prompt_key_path': model_args.previous_prompt_key_path,
         'task_id': cur_task_id,
@@ -566,13 +575,13 @@ def main():
         device = torch.device(f"cuda:{local_rank}")
     except:
         device = torch.device(f"cuda:0")
-    if model_args.load_checkpoint_from and training_args.model_name != 'specroute':
+    if model_args.load_checkpoint_from and (training_args.model_name != 'specroute' or getattr(training_args, "routing_mode", "") == "learned"):
         if not os.path.exists(model_args.load_checkpoint_from):
             logger.warning(f"load_checkpoint_from not found: {model_args.load_checkpoint_from}, skipping load")
         else:
             print("----------Loading Previous Query Projection Layer----------")
             model.encoder.trans_input.load_state_dict(torch.load(model_args.load_checkpoint_from, map_location=device))
-            if training_args.model_name in ['gainlora_inflora', 'gainlora_olora']:
+            if training_args.model_name in ['gainlora_inflora', 'gainlora_olora'] or (training_args.model_name == 'specroute' and getattr(training_args, "routing_mode", "") == "learned"):
                 model.encoder.previous_trans_input.input_linear[0].data.copy_(torch.load(model_args.load_checkpoint_from, map_location=device)['0.weight'])
                 model.encoder.previous_trans_input.output_linear[0].data.copy_(torch.load(model_args.load_checkpoint_from, map_location=device)['2.weight'])
                 model.encoder.previous_trans_input.state_dict()
@@ -661,6 +670,9 @@ def main():
             param.requires_grad = False
             if "lora_B" in name and "previous_lora_weights" not in name:
                 param.requires_grad = True
+            if getattr(training_args, "routing_mode", "") == "learned":
+                if ("trans_input" in name and "previous_trans_input" not in name) or "prompt_key" in name:
+                    param.requires_grad = True
 
     total_params, params = 0, 0
     for n, p in model.named_parameters():
@@ -1029,6 +1041,17 @@ def main():
             signatures = compute_spectral_signatures(trainer.model, config)
             torch.save(signatures, os.path.join(save_path, 'spectral_signatures.pt'))
             print("----------Saved spectral signatures----------")
+
+            if getattr(training_args, "routing_mode", "") == "learned":
+                from copy import deepcopy
+                if not prompt_config["run_single"]:
+                    if prompt_config["previous_prompt_key_path"] is not None:
+                        previous_trans_input = deepcopy(trainer.model.encoder.previous_trans_input.state_dict())
+                        torch.save(previous_trans_input, os.path.join(save_path, 'previous_trans_input.pt'))
+                        torch.save(torch.cat([trainer.model.encoder.prompt_key, trainer.model.encoder.previous_prompts_keys], dim=0).data, os.path.join(save_path, 'prompts_keys_till_now.pt'))
+                    else:
+                        torch.save(trainer.model.encoder.prompt_key.data, os.path.join(save_path, 'prompts_keys_till_now.pt'))
+                    torch.save(trainer.model.encoder.trans_input.state_dict(), os.path.join(save_path, 'trans_input.pt'))
         # Only save tokenizer for non-specroute (specroute never reloads it)
         if training_args.model_name != 'specroute':
             tokenizer.save_pretrained(save_path)

improve_gainlora/src/t5_specroute.py

@@ -57,6 +57,7 @@ from t5_gainlora_inflora import (
     T5LayerCrossAttention,
     T5Block,
     T5PreTrainedModel,
+    Trans_input,
 )
 
 logger = logging.get_logger(__name__)
@@ -145,16 +146,43 @@ class T5Stack(T5PreTrainedModel):
         self.prompt_config = prompt_config
 
         if not self.is_decoder and not prompt_config["run_single"]:
-            # ===== Spectral routing: NO learned parameters for routing =====
-            # Spectral signatures loaded from previous tasks' saved weights
+            self.routing_mode = prompt_config.get("routing_mode", "spectral")
+            
+            # Common for all spectral/grassmann modes
             self.spectral_signatures = []  # List[dict] — one dict per old task
-            self.routing_temperature = prompt_config.get('attn_temperature', 1.0)
+            
+            if self.routing_mode == "learned":
+                # V10a: Learned routing matching GainLoRA ROOT exactly
+                self.prompt_key = nn.Parameter(torch.randn((1, config.d_model)))
+                nn.init.uniform_(self.prompt_key, -1, 1)
+
+                self.trans_input = nn.Sequential(
+                    nn.Linear(config.d_model, prompt_config["mlp_hidden_dim"], bias=False),
+                    nn.SiLU(),
+                    nn.Linear(prompt_config["mlp_hidden_dim"], config.d_model, bias=False),
+                    nn.SiLU(),
+                )
 
-            # Adaptive training bias: beta = T * ln(alpha * n_old / (1 - alpha))
-            # Ensures current task gets consistent routing weight ~alpha regardless
-            # of total number of tasks (fixes softmax dilution with constant bias).
-            # At inference, same A-row formula without bias (V8 symmetric routing).
-            self._target_routing_alpha = prompt_config.get('target_routing_alpha', 0.8)
+                self.get_trans_feature = False
+                self.stage_trans = 0
+                self.matrix_trans_1 = torch.zeros(config.d_model, config.d_model)
+                self.matrix_trans_2 = torch.zeros(prompt_config["mlp_hidden_dim"], prompt_config["mlp_hidden_dim"])
+                self.n_trans_matrix = 0
+
+                self.previous_prompts_keys = None
+                if prompt_config.get("previous_prompt_key_path") is not None and prompt_config.get("task_id", 0):
+                    print("----------Loading Previous Keys----------")
+                    self.previous_prompts_keys = nn.Parameter(torch.randn((prompt_config["task_id"], config.d_model)))
+                    self.previous_prompts_keys.data = torch.load(prompt_config["previous_prompt_key_path"], weights_only=True)
+                    self.previous_prompts_keys.requires_grad = False
+                    
+                    self.previous_trans_input = Trans_input(config.d_model, prompt_config["mlp_hidden_dim"], prompt_config["task_id"])
+                    for param in self.previous_trans_input.parameters():
+                        param.requires_grad = False
+            else:
+                # V8/V9/V10b: Spectral routing parameters
+                self.routing_temperature = prompt_config.get('attn_temperature', 1.0)
+                self._target_routing_alpha = prompt_config.get('target_routing_alpha', 0.8)
 
             # For inference logging
             self.all_attn_weights = []
@@ -174,7 +202,139 @@ class T5Stack(T5PreTrainedModel):
     # The old format (with 'value' param) causes transformers to silently ignore
     # gradient_checkpointing_kwargs (including use_reentrant=False).
 
-    def compute_spectral_routing(self, avg_inputs_embeds):
+    def get_chunk(self, chunk):
+        if self.routing_mode == "learned":
+            self.chunk_trans = chunk
+            self.index_trans, self.step_trans = chunk, self.config.d_model // chunk
+            self.step, self.index = self.step_trans, self.index_trans
+            self.matrix_trans_1, self.matrix_trans_3, self.n_trans_matrix = {}, {}, {}
+            for idx in range(self.index_trans):
+                self.matrix_trans_1[idx] = torch.zeros(self.step_trans, self.step_trans).cuda()
+                self.matrix_trans_3[idx] = torch.zeros(self.step_trans, self.step_trans).cuda()
+                self.n_trans_matrix[idx] = 0
+            self.matrix_trans_2 = self.matrix_trans_2.cuda()
+
+    def get_matrix3(self, x, medium, x_final):
+        if self.routing_mode == "learned":
+            for idx in range(self.index_trans):
+                m1_curr = torch.bmm(x[:,:,idx*self.step_trans:(idx+1)*self.step_trans].detach().permute(0, 2, 1), x[:,:,idx*self.step_trans:(idx+1)*self.step_trans].detach()).sum(dim=0).float()/(x.shape[0]*x.shape[1])
+                m3_curr = torch.bmm(x_final[:,:,idx*self.step_trans:(idx+1)*self.step_trans].detach().permute(0, 2, 1), x_final[:,:,idx*self.step_trans:(idx+1)*self.step_trans].detach()).sum(dim=0).float()/(x_final.shape[0]*x_final.shape[1])
+                
+                if len(self.matrix_trans_1) > 0 and isinstance(self.matrix_trans_1.get(idx), torch.Tensor) and self.matrix_trans_1.get(idx).sum() != 0:
+                    self.matrix_trans_1[idx] = (self.matrix_trans_1[idx]*self.n_trans_matrix[idx] + m1_curr)/(self.n_trans_matrix[idx] + x.shape[0]*x.shape[1])
+                    self.matrix_trans_3[idx] = (self.matrix_trans_3[idx]*self.n_trans_matrix[idx] + m3_curr)/(self.n_trans_matrix[idx] + x_final.shape[0]*x_final.shape[1])
+                else:
+                    self.matrix_trans_1[idx] = m1_curr
+                    self.matrix_trans_3[idx] = m3_curr
+                self.n_trans_matrix[idx] += x.shape[0]*x.shape[1]
+
+            if self.matrix_trans_2.sum() == 0:
+                self.matrix_trans_2 = torch.bmm(medium.detach().permute(0, 2, 1), medium.detach()).sum(dim=0).float()/(medium.shape[0]*medium.shape[1])
+            else:
+                self.matrix_trans_2 = (self.matrix_trans_2*self.n_trans_matrix[0] + torch.bmm(medium.detach().permute(0, 2, 1), medium.detach()).sum(dim=0).float())/(self.n_trans_matrix[0] + medium.shape[0]*medium.shape[1])
+
+    def cal_attention(self, prompt_key, x, return_logits=False):
+        # ROOT-style routing similarity
+        x = x/(x.norm(dim=-1,keepdim=True) + 1e-12)
+        prompt_key = prompt_key/(prompt_key.norm(dim=-1,keepdim=True) + 1e-12)
+        attn_scores = (x*prompt_key).sum(dim=-1, keepdim=True)
+        weights = torch.abs(torch.nn.functional.sigmoid(attn_scores*4)*2-1)
+        if not return_logits:
+            return weights
+        else:
+            return attn_scores
+
+    def compute_learned_routing(self, avg_inputs_embeds, batch_size):
+        """V10a: Learned MLP Routing copying GainLoRA exactly"""
+        prompt_key = self.prompt_key
+        if self.previous_prompts_keys is not None:
+            prompt_key = self.prompt_key.to(prompt_key.device)
+            past_prompt_key = torch.cat([prompt_key.repeat(batch_size, 1, 1), self.previous_prompts_keys.repeat(batch_size, 1, 1)], dim=1)
+
+            medium = self.trans_input[1](self.trans_input[0](avg_inputs_embeds))
+            x = self.trans_input[3](self.trans_input[2](medium))
+            if getattr(self, "get_trans_feature", False):
+                self.get_matrix3(avg_inputs_embeds, medium, x)
+
+            past_x = torch.cat([x, self.previous_trans_input(avg_inputs_embeds)], dim=1)
+            key_attention_weights = self.cal_attention(past_prompt_key, past_x)
+        else:
+            medium = self.trans_input[1](self.trans_input[0](avg_inputs_embeds))
+            x = self.trans_input[3](self.trans_input[2](medium))
+            if getattr(self, "get_trans_feature", False):
+                self.get_matrix3(avg_inputs_embeds, medium, x)
+
+            key_attention_weights = self.cal_attention(prompt_key.repeat(batch_size, 1, 1), x)
+        return key_attention_weights
+
+    def compute_grassmann_routing(self, h, h_norm_sq):
+        """V10b: Grassmann Distance Routing
+        Calculates principal angles between batch local subspace and candidate A_t subspaces.
+        """
+        B, _, d_model = h.shape
+        if self.training or B < 8:
+            # Fallback to A-row fit for very small batches or training (oracle handles training)
+            return self.compute_spectral_routing(h, h_norm_sq)
+            
+        fits = []
+        r = self.block[0].layer[0].SelfAttention.lora_q.r
+        
+        # Batch PCA to get local subspace U_batch (using SVD)
+        # h is (B, 1, d_model) -> reshape to (B, d_model)
+        h_flat = h.squeeze(1)
+        # torch.linalg.svd returns (U, S, Vh) where Vh = V^T
+        # We want right singular vectors V: h_flat = U @ diag(S) @ Vh, so V = Vh.T
+        _, _, Vh_batch = torch.linalg.svd(h_flat - h_flat.mean(dim=0, keepdim=True), full_matrices=False)
+        U_batch = Vh_batch[:r, :]  # Vh is (min(B,d), d), so first r rows = top-r right sing. vectors, shape (r, d_model)
+        
+        # Current task Grassmann dist
+        current_layer_dists = []
+        for block in self.block:
+            attn = block.layer[0].SelfAttention
+            for lora in [attn.lora_q, attn.lora_v]:
+                A = lora.lora_A.data.float().to(h.device) # (r, d_model)
+                # SVD of A^T: A^T = U_A @ diag(S_A) @ Vh_A => columns of U_A are right sing vecs of A
+                _, _, Vh_A = torch.linalg.svd(A, full_matrices=False)  # A is (r, d_model), Vh_A is (r, d_model)
+                U_A = Vh_A[:r, :]  # (r, d_model) — top-r right singular vectors of A, forming the subspace
+                
+                # Grassmann distance via principal angles
+                # cos(theta_i) = singular values of U_batch @ U_A^T
+                M = torch.matmul(U_batch, U_A.T)  # (r, r)
+                angles = torch.linalg.svdvals(M).clamp(-1.0, 1.0)
+                principal_angles = torch.acos(angles)
+                dist = torch.sqrt(torch.sum(principal_angles**2))
+                current_layer_dists.append(dist)
+        
+        current_dist = torch.stack(current_layer_dists).mean(dim=0).item()
+        fits.append(1.0 / (current_dist + 1e-4)) # Inverse dist as affinity
+        
+        # Old tasks
+        for sig_dict in self.spectral_signatures:
+            task_dists = []
+            for key, sig_data in sig_dict.items():
+                if not key.startswith('enc.'):
+                    continue
+                A = sig_data['A'].to(h.device, dtype=torch.float32)  # (r, d_model)
+                _, _, Vh_A = torch.linalg.svd(A, full_matrices=False)
+                U_A = Vh_A[:r, :]
+                
+                M = torch.matmul(U_batch, U_A.T)
+                angles = torch.linalg.svdvals(M).clamp(-1.0, 1.0)
+                dist = torch.sqrt(torch.sum(torch.acos(angles)**2))
+                task_dists.append(dist)
+            
+            if task_dists:
+                task_dist = torch.stack(task_dists).mean(dim=0).item()
+                fits.append(1.0 / (task_dist + 1e-4))
+            else:
+                fits.append(0.0)
+                
+        fit_scores = torch.tensor(fits, device=h.device).unsqueeze(0).repeat(B, 1) # (B, n_tasks)
+        max_idx = fit_scores.argmax(dim=1, keepdim=True)
+        weights = torch.zeros_like(fit_scores).scatter_(1, max_idx, 1.0)
+        return weights.unsqueeze(2)
+
+    def compute_spectral_routing(self, h, h_norm_sq):
         """
         V9: Routing with oracle-training / spectral-inference split + calibration.
 
@@ -202,9 +362,6 @@ class T5Stack(T5PreTrainedModel):
         Returns:
             (B, n_tasks, 1) routing weights: oracle one-hot (training) or top-1 (inference)
         """
-        h = avg_inputs_embeds  # (B, 1, d_model)
-        h_norm_sq = (h ** 2).sum(dim=-1) + 1e-8  # (B, 1)
-
         fits = []
 
         # === CURRENT TASK: A-row fit ===
@@ -380,29 +537,36 @@ class T5Stack(T5PreTrainedModel):
             avg_inputs_embeds = (attention_mask.unsqueeze(-1) * inputs_embeds).sum(dim=1, keepdim=True) / _mask_count
 
         if not self.is_decoder and not self.prompt_config["run_single"]:
-            if len(self.spectral_signatures) > 0:
-                # Multi-task: compute routing
-                key_attention_weights = self.compute_spectral_routing(avg_inputs_embeds)
-                # Detach: routing weights are shared across all gradient-checkpointed
-                # blocks via closure. Without detach, the second block's backward
-                # fails with "backward through graph a second time" because the
-                # first block already freed the shared graph (inputs_embeds -> routing).
-                # Safe because routing uses lora_A.data (detached) and frozen signatures.
-                key_attention_weights = key_attention_weights.detach()
-
-                if self.is_inference:
-                    self.all_attn_weights.append(
-                        key_attention_weights.squeeze().mean(dim=0, keepdim=True).detach().to(torch.float).cpu().numpy()
-                    )
+            if self.routing_mode == "learned":
+                key_attention_weights = self.compute_learned_routing(avg_inputs_embeds, batch_size)
+                
+                if self.is_inference and self.previous_prompts_keys is not None:
+                    self.all_attn_weights.append(key_attention_weights.squeeze().mean(dim=0, keepdim=True).detach().to(torch.float).cpu().numpy())
+                elif self.is_inference:
+                    self.all_attn_weights.append(key_attention_weights.squeeze(2).mean(dim=0, keepdim=True).detach().to(torch.float).cpu().numpy())
             else:
-                # First task or no previous info: single LoRA, weight = 1
-                key_attention_weights = torch.ones(
-                    batch_size, 1, 1, device=inputs_embeds.device, dtype=inputs_embeds.dtype
-                )
-                if self.is_inference:
-                    self.all_attn_weights.append(
-                        key_attention_weights.squeeze(2).mean(dim=0, keepdim=True).detach().to(torch.float).cpu().numpy()
+                if len(self.spectral_signatures) > 0:
+                    h_norm_sq = (avg_inputs_embeds ** 2).sum(dim=-1) + 1e-8  # (B, 1)
+                    if self.routing_mode == "grassmann":
+                        key_attention_weights = self.compute_grassmann_routing(avg_inputs_embeds, h_norm_sq)
+                    else:
+                        key_attention_weights = self.compute_spectral_routing(avg_inputs_embeds, h_norm_sq)
+                        
+                    key_attention_weights = key_attention_weights.detach()
+
+                    if self.is_inference:
+                        self.all_attn_weights.append(
+                            key_attention_weights.squeeze().mean(dim=0, keepdim=True).detach().to(torch.float).cpu().numpy()
+                        )
+                else:
+                    # First task or no previous info: single LoRA, weight = 1
+                    key_attention_weights = torch.ones(
+                        batch_size, 1, 1, device=inputs_embeds.device, dtype=inputs_embeds.dtype
                     )
+                    if self.is_inference:
+                        self.all_attn_weights.append(
+                            key_attention_weights.squeeze(2).mean(dim=0, keepdim=True).detach().to(torch.float).cpu().numpy()
+                        )
             self.key_attention_weights = key_attention_weights
         else:
             # Decoder or run_single: use whatever was passed (from encoder)

results/experiment_versions.md

@@ -360,4 +360,20 @@ V8 fail imdb/sst2/yahoo do B_t không học (gradient bị block). V9 oracle rou
 | - | V5 | **59.55** | **62.19** | Prototype routing + entropy + preconditioning |
 | - | V6 | ~27.4 | ~35.5 | SVD + C4 only (no prototypes) — **FAILED** |
 | - | V8 | 35.78 | 43.73 | C5 Data-Informed Init + C4 precond + A-row routing (no β) — PARTIAL |
-| - | V9 | (pending) | (pending) | Oracle routing (training) + calibrated Top-1 (inference) — bug fix |
+| - | V9 | 43.14 | 51.55 | Oracle routing (training) + calibrated Top-1 (inference) — bug fix |
+| - | V10a | (pending) | (pending) | Learned Routing + GPM + C5 + C4 |
+| - | V10b | (pending) | (pending) | Grassmannian Distance Routing + C5 + C4 |
+
+---
+
+## V10 — Duality of Routing Mechanisms
+
+**Motivation**: V9 showed that Top-1 A-row routing struggles to isolate orthogonal subspaces despite C4+C5. V10 explores two distinct modes to address routing precision while preserving C5's benefits.
+
+### V10a (Learned Routing - The Practical Baseline)
+- **Method**: Reintroduces ROOT's `Trans_input` MLP and `prompt_key` gating, with exact GPM constraints applied to their weights post-optimizer step.
+- **Why**: Proves that C5 initialization and C4 preconditioning can synergize with explicit function approximation for routing. Sacrifices the "parameter-free" claim but serves as a strong upper-bound baseline.
+
+### V10b (Grassmannian Distance Routing - The Zero-Replay Ideal)
+- **Method**: Evaluates similarity by computing the Grassmannian distance (principal angles) between the batch's local principal subspace $U_{batch}$ and expert orthogonal projection $U_A$.
+- **Why**: Directly measures subset geometric alignment, entirely bypassing scale-based similarity issues (GPM-Routing paradox). Batch-level SVD aggregates representations properly. Valid for batched inference ($B \ge 8$), falling back to A-row for small batches.