v2

.gitignore

@@ -9,4 +9,9 @@ __pycache__/
 *.pyo
 *.pyd
 */logs/*
-*/logs
+*/logs
+logs/*
+logs
+*.log
+
+

experiment_diagnosis_report.md

@@ -0,0 +1,301 @@
+# Báo cáo chẩn đoán thí nghiệm: SpecRoute vs GainLoRA trên T5-Small
+
+> **Benchmark**: Long Sequence Order 3 (15 classification tasks)  
+> **Model**: T5-Small (d_model=512, 6 encoder + 6 decoder layers, lora_r=8)  
+> **Thí nghiệm**: SpecRoute (improve) vs GainLoRA-InfLoRA (root)
+
+---
+
+## 1. Xác minh kết quả: Bảng so sánh có chính xác không?
+
+### ✅ ROOT GainLoRA: AP = 59.70 — CHÍNH XÁC
+
+Nguồn dữ liệu: `logs/root_t5_small/.../15-wic/all_results.json`
+- Task 15 (wic) có `--do_predict` → evaluation trên ALL 15 tasks (70,861 samples)
+- Metrics `predict_exact_match_for_{task}` cho tất cả tasks → **đây là R_{15,j} (final row)**
+- AP = mean(R_{15,j}) = 59.70 ✓ (tính đúng theo paper)
+
+### ⚠️ SpecRoute: "AP" = 39.74 — **KHÔNG PHẢI AP THẬT**
+
+Nguồn dữ liệu: `predict_eval_predictions.jsonl` tại MỖI task directory
+- SpecRoute THIẾU `--do_predict` cho tasks 2-15 (bug trong script generator)
+- File `predict_eval_predictions.jsonl` ở mỗi task chỉ chứa **current task evaluation**
+- Các con số (yelp=54.36, imdb=0.21, etc.) là **R_{j,j} (diagonal = peak performance)**, KHÔNG phải R_{15,j}
+- 39.74 = mean(diagonal), **KHÔNG phải AP** theo công thức paper
+
+**Hệ quả**: AP thật của SpecRoute sẽ THẤP HƠN 39.74 vì forgetting sẽ giảm performance của các tasks đầu. Khoảng cách thực tế với ROOT có thể lớn hơn 19.96 điểm.
+
+### Bảng so sánh đã hiệu chỉnh
+
+| # | Task | ROOT R_{15,j} (Final) | SpecRoute R_{j,j} (Peak) | Ghi chú |
+|---|------|-----------------------|--------------------------|---------|
+| 1 | yelp | 56.01 | 54.36 | Tương đương |
+| 2 | amazon | 52.05 | 50.01 | Tương đương |
+| 3 | mnli | 34.07 | 35.50 | SpecRoute tốt hơn |
+| 4 | cb | 3.57 | 0.00 | Cả hai đều thấp |
+| 5 | copa | 42.00 | 44.00 | Tương đương |
+| 6 | qqp | 76.96 | 76.72 | Tương đương |
+| 7 | rte | 45.85 | 50.90 | SpecRoute tốt hơn |
+| 8 | imdb | 89.51 | **0.21** ⚠️ | **Không thể học** |
+| 9 | sst2 | 85.21 | **0.00** ⚠️ | **Không thể học** |
+| 10 | dbpedia | 98.16 | 92.22 | Chấp nhận được |
+| 11 | agnews | 88.37 | 68.76 | Giảm đáng kể |
+| 12 | yahoo | 57.28 | **8.12** ⚠️ | **Không thể học** |
+| 13 | multirc | 50.52 | 54.23 | Tương đương |
+| 14 | boolq | 60.43 | 61.13 | Tương đương |
+| 15 | wic | 55.49 | **0.00** ⚠️ | **Không thể học** |
+
+**Nhận xét quan trọng**: SpecRoute scoring ở đây là PEAK (ngay sau khi train task đó), trong khi ROOT scoring là FINAL (sau khi train xong tất cả 15 tasks). Với ROOT, imdb PEAK có thể > 89.51 rồi chỉ giảm nhẹ về 89.51. Nhưng với SpecRoute, imdb PEAK đã là 0.21 — model KHÔNG THỂ HỌC task này ngay từ đầu, đây **không phải catastrophic forgetting**.
+
+---
+
+## 2. Tại sao FT (Forgetting) không tính được?
+
+### Nguyên nhân trực tiếp: `--do_predict` bị thiếu
+
+Công thức FT cần:
+- R_{j,j} = performance trên task j ngay sau khi train task j (diagonal)
+- R_{T,j} = performance trên task j sau khi train xong tất cả T tasks (final row)
+
+| Method | R_{j,j} (diagonal) | R_{T,j} (final row) | FT computable? |
+|--------|--------------------|--------------------|----------------|
+| ROOT | ❌ Không có (tasks 1-14 thiếu cross-task eval) | ✅ Task 15 có | ❌ Thiếu diagonal |
+| SpecRoute | ⚠️ Có nhưng chỉ single-task eval | ❌ Task 15 không eval cross-task | ❌ Thiếu final row |
+
+### Nguyên nhân gốc: Bug trong script generator
+
+File `improve_gainlora/generate_specroute_scripts_v2.py`:
+
+```python
+"long_order3": {
+    ...
+    "do_predict": False,    # ← BUG: nên là True
+    ...
+},
+```
+
+**Fix**: Đổi thành `True` cho cả `long_order3` và `long_order4`. Khi `do_predict=True`, script sẽ generate `--do_predict --predict_with_generate` cho mỗi task → `run_t5.py` sẽ evaluate trên ALL task cumulative test sets → `score.py` sẽ build được full matrix R → FT tính được.
+
+ROOT cũng cần fix: hiện tại chỉ task 15 có `--do_predict`. Cần thêm cho tasks 1-14 để có full R matrix.
+
+---
+
+## 3. Phân tích nguyên nhân gốc: Tại sao SpecRoute kém?
+
+### 3.1 KHÔNG phải do SVD/routing bugs
+
+Sau khi đọc toàn bộ source code:
+- `compute_spectral_signatures()`: SVD đúng, lưu Vt[:r] và S[:r] đúng
+- `compute_spectral_routing()`: Weighted Rayleigh quotient đúng, softmax đúng
+- Không có hardcoded dimensions cho T5-large
+- d_model=512, lora_r=8 → SVD rank=8 capture toàn bộ non-zero singular values
+- Gradient checkpointing fix đã áp dụng đúng
+
+### 3.2 KHÔNG phải hoàn toàn do config SVD (giả thuyết ban đầu)
+
+User hypothesis: "configs ban đầu được thiết kế cho T5_large, T5_small nên config SVD không phù hợp"
+
+**Sự thật**: Không có config SVD-specific nào cần thay đổi cho T5-small. Các hyperparameters (lora_r=8, lora_alpha=32, threshold=0.995, temperature=1.0) là model-agnostic. Vấn đề nằm ở chỗ khác.
+
+### 3.3 NGUYÊN NHÂN CHÍNH: Thiếu cơ chế chống forgetting
+
+Đây là bảng so sánh **cơ chế bảo vệ** giữa 2 phương pháp:
+
+| Cơ chế | ROOT GainLoRA | SpecRoute | Tác động |
+|--------|:---:|:---:|----------|
+| GPM gradient projection (LoRA A) | ✅ | ✅ | Chặn gradient phá LoRA cũ |
+| KL distillation (`kl_ratio=0.1`) | ✅ | ❌ | Duy trì routing distribution cũ |
+| Data replay (`gen_data_dir`) | ✅ | ❌ | Reinforce kiến thức cũ |
+| Per-step GPM on routing params | ✅ | ❌ | Bảo vệ trans_input + prompt_key |
+| Trans_input (learned routing) | ✅ | ❌ | Routing có gradient, học continuous |
+
+**ROOT có 5 lớp bảo vệ, SpecRoute chỉ có 1 lớp (GPM trên LoRA A)**
+
+Khi loại bỏ learned routing (trans_input + prompt_key), SpecRoute đồng thời loại bỏ luôn:
+1. KL distillation (vì không có routing params để distill)
+2. Data replay (vì không có routing MLP cần reinforce)
+3. Per-step GPM trên routing params (vì không có routing params)
+
+Đây **không phải là design intention** — SpecRoute muốn replace routing mechanism, nhưng vô tình loại bỏ luôn CÁC CƠ CHẾ BẢO VỆ đi kèm routing.
+
+### 3.4 NGUYÊN NHÂN PHỤ: GPM null-space bão hòa sớm ở T5-small
+
+Training loss so sánh (bằng chứng GPM over-constraining):
+
+| Task (thứ tự) | ROOT loss | SpecRoute loss | Tỉ lệ | SpecRoute score |
+|---|---|---|---|---|
+| 1 yelp | 0.586 | 0.581 | 1.0x | 54.36 |
+| 2 amazon | 0.540 | 0.588 | 1.1x | 50.01 |
+| 5 copa | 0.455 | 0.459 | 1.0x | 44.00 |
+| 6 qqp | 0.288 | 0.304 | 1.1x | 76.72 |
+| 8 imdb | 1.410 | **4.149** | **2.9x** | **0.21** |
+| 9 sst2 | 1.762 | **4.449** | **2.5x** | **0.00** |
+| 12 yahoo | 1.189 | **3.077** | **2.6x** | **8.12** |
+| 15 wic | 0.961 | **3.654** | **3.8x** | **0.00** |
+
+**Pattern rõ ràng**: Tasks ban đầu (1-6) loss tương đương → model học OK. Tasks sau (8, 9, 12, 15) loss cao gấp 2.5-3.8x → model KHÔNG THỂ HỌC.
+
+Nhưng thú vị: tasks 10 (dbpedia), 13 (multirc), 14 (boolq) vẫn học được tốt (loss < 1.3). Điều này cho thấy vấn đề không chỉ đơn thuần "hết null-space":
+
+**Các tasks THẤT BẠI (imdb, sst2, yahoo, wic)** có đặc điểm chung: overlap lớn với tasks TRƯỚC ĐÓ trong feature space:
+- imdb/sst2 = sentiment binary → overlap với yelp (task 1), amazon (task 2)
+- yahoo = topic QA → overlap với nhiều domain trước
+- wic = word sense → cần representations already claimed bởi tasks trước
+
+**Giải thích**: GPM từ tasks 1-2 (yelp/amazon sentiment) đã "claim" sentiment-relevant directions. Khi imdb (cũng sentiment) đến, GPM ép LoRA A vào null-space orthogonal với sentiment directions → model bị ép vào directions KHÔNG LIÊN QUAN đến sentiment → không thể phân loại sentiment → loss cao, accuracy 0.
+
+Trong ROOT GainLoRA, vấn đề này được giải quyết bởi:
+- Trans_input cho phép MAP input mới vào representation space REUSE kiến thức sentiment cũ
+- KL distillation cho phép routing CHUYỂN imdb sang LoRA branch sentiment đã có
+- Data replay DUY TRÌ sentiment knowledge
+
+### 3.5 Training loss cao = model không thể học, KHÔNG PHẢI catastrophic forgetting
+
+Đây là phát hiện quan trọng nhất: comparison_results.md ghi "imdb/sst2/wic về 0 do Catastrophic Forgetting" — **NHẬN ĐỊNH NÀY SAI**.
+
+Bằng chứng:
+- imdb train_loss = 4.149 (rất cao) → model CHƯA BAO GIỜ học được imdb
+- imdb prediction: "Rififi" (copy từ review text), "Negative" (sai format, label đúng là "Good"/"Bad")
+- sst2 train_loss = 4.449 → tương tự
+
+**Đây là "inability to learn" (GPM over-constraining), KHÔNG phải "learned then forgot" (catastrophic forgetting).**
+
+---
+
+## 4. Lỗi so sánh không công bằng
+
+| Aspect | ROOT | SpecRoute | Vấn đề |
+|--------|------|-----------|--------|
+| Score type | R_{15,j} (FINAL) | R_{j,j} (PEAK/DIAGONAL) | So sánh khác loại |
+| Evaluation | Cross-task (all 15) | Single-task (chỉ current) | Scope khác nhau |
+| `--do_predict` | Task 15 only | Task 1 only | Cả hai đều thiếu |
+
+**ROOT**: Đánh giá SAU KHI train xong 15 tasks → bao gồm cả forgetting
+**SpecRoute**: Đánh giá NGAY SAU KHI train từng task → peak performance, chưa bao gồm forgetting
+
+Để so sánh công bằng, cần chạy lại SpecRoute với `--do_predict` ở task 15 để có R_{15,j} cho tất cả tasks.
+
+---
+
+## 5. Định hướng cải tiến
+
+### 5.1 Fix NGAY (không đổi methodology)
+
+**A. Thêm `--do_predict` cho tất cả tasks**
+```python
+# generate_specroute_scripts_v2.py
+"long_order3": { "do_predict": True },  # was False
+"long_order4": { "do_predict": True },  # was False
+```
+→ Cho phép build full R matrix, tính AP/FT đúng, so sánh công bằng.
+
+**B. Khôi phục KL distillation**
+
+Đây là fix quan trọng nhất. SpecRoute loại bỏ learned routing nhưng KL distillation hoàn toàn có thể adapt cho spectral routing:
+
+```python
+# Concept: KL trên routing output thay vì routing params
+def spectral_kl_regularization(model, old_signatures, input_embeds):
+    """Duy trì routing distribution gần với snapshot sau task trước"""
+    current_routing = model.compute_spectral_routing(input_embeds)
+    old_routing = compute_old_routing(old_signatures, input_embeds)
+    return kl_div(current_routing.log(), old_routing)
+```
+
+Tuy nhiên, vì spectral routing là deterministic (không có learnable params), KL trên routing output không tạo gradient hữu ích. Thay vào đó:
+
+**Option tốt hơn: KL distillation trên model OUTPUT (logits)**
+```python
+# Sau mỗi task, lưu model logits trên replay data
+# Trong training step tiếp theo:
+kl_loss = kl_div(current_logits, saved_old_logits)
+```
+
+**C. Khôi phục Data Replay**
+
+Replay không phụ thuộc vào routing mechanism. Có thể dùng generated data hoặc coreset luôn:
+```bash
+--gen_data_dir generated_data/lora_gen_long_t5   # Tái sử dụng tập replay của ROOT
+--data_replay_freq 5                              # Replay mỗi 5 steps
+--kl_ratio 0.1                                    # Weight cho KL loss trên replay
+```
+
+### 5.2 Giảm GPM threshold cho T5-small
+
+Với threshold=0.995, sau 15 tasks, threshold tăng lên 0.99967 → GPM giữ 99.97% variance → null-space cực nhỏ.
+
+| threshold | Task 1 | Task 7 | Task 14 | Nhận xét |
+|-----------|--------|--------|---------|----------|
+| 0.995 (hiện tại) | 0.9950 | 0.9973 | 0.9997 | Quá chặt cho T5-small |
+| 0.990 | 0.9900 | 0.9947 | 0.9993 | Vẫn khá chặt |
+| 0.980 | 0.9800 | 0.9893 | 0.9987 | Thử nghiệm đầu tiên |
+| 0.970 | 0.9700 | 0.9840 | 0.9980 | Aggressive nhưng đáng thử |
+
+**Đề xuất**: Thử threshold=0.980 trước, nếu forgetting tăng thì kết hợp KL distillation để bù.
+
+### 5.3 Cải tiến methodology (dài hạn)
+
+**A. Cho phép subspace sharing**
+
+Vấn đề gốc: GPM ép tasks tương tự (imdb/sst2 vs yelp/amazon) vào subspaces orthogonal. Cần mechanism cho phép knowledge reuse:
+
+```python
+# Ý tưởng: Nếu spectral routing gợi ý task mới SIMILAR với task cũ,
+# giảm GPM protection cho directions tương tự → cho phép reuse
+similarity = compute_spectral_routing(avg_input)  # routing weights
+for old_task, weight in enumerate(similarity):
+    if weight > threshold_reuse:
+        # Giảm GPM projection cho old_task's directions
+        # → cho phép refinement thay vì full orthogonality
+```
+
+**B. Hybrid routing: spectral + lightweight learned component**
+
+Thay vì hoàn toàn parameter-free, thêm adapter nhẹ:
+```python
+routing = alpha * spectral_fit + (1-alpha) * learned_gate
+```
+- `spectral_fit`: parameter-free, ổn định, không cần GPM protection
+- `learned_gate`: lightweight (MLP nhỏ), cho phép gradient flow
+- `alpha`: có thể learnable hoặc fixed (e.g., 0.7)
+
+**C. Tách biệt protection vs routing**
+
+Thiết kế SpecRoute hiện tại **couple** routing mechanism với protection mechanisms. Cần tách:
+- **Routing**: Spectral (parameter-free) — OK giữ nguyên
+- **Protection**: Cần ÍT NHẤT 2 trong 3: GPM, KL distillation, data replay
+
+---
+
+## 6. Kế hoạch thí nghiệm tiếp theo
+
+### Phase 1: Fix bugs + fair comparison (ưu tiên CAO)
+1. Fix `generate_specroute_scripts_v2.py`: `do_predict=True` cho long benchmarks
+2. Regenerate scripts
+3. Chạy lại SpecRoute Long Order 3 trên T5-small
+4. So sánh AP/FT chính xác giữa 2 methods
+
+### Phase 2: Thêm protection mechanisms (ưu tiên CAO)
+1. Thêm KL distillation trên model output logits (replay + KL loss)
+2. Thêm data replay
+3. Grid search: threshold ∈ {0.995, 0.990, 0.980}, kl_ratio ∈ {0.05, 0.1, 0.2}
+
+### Phase 3: Validate methodology (sau phase 2)
+1. Nếu Phase 2 cho kết quả tốt → methodology đúng, chỉ thiếu protection
+2. Nếu Phase 2 vẫn kém → spectral routing có vấn đề ở T5-small, cần hybrid approach
+3. Scale lên T5-large để so sánh ở đúng scale ROOT paper dùng
+
+---
+
+## 7. Tổng kết
+
+| Câu hỏi | Trả lời |
+|----------|---------|
+| Kết quả tổng hợp có chính xác? | ⚠️ ROOT đúng (AP=59.70), SpecRoute SAI loại metric (diagonal vs final) |
+| Tại sao kết quả tệ? | SpecRoute loại bỏ routing → vô tình loại bỏ luôn KL + replay + per-step GPM |
+| Do methodology hay config? | **Cả hai**: methodology thiếu protection layers + GPM threshold quá chặt cho T5-small |
+| SVD có phải nguyên nhân? | **Không trực tiếp**. SVD routing code đúng, không có bugs |
+| FT tại sao chưa tính? | Bug trong script generator: `do_predict=False` cho long benchmarks |
+| Hướng cải tiến? | Khôi phục KL distillation + data replay, giảm GPM threshold, fix scripts |
+
+**Kết luận cốt lõi**: Ý tưởng spectral routing thay thế learned routing KHÔNG SAI về mặt lý thuyết. Vấn đề là khi implement, các cơ chế protection (KL, replay) bị loại bỏ theo vì chúng gắn chặt với learned routing trong code ROOT. Cần decouple routing mechanism khỏi protection mechanisms.

improve_gainlora/T5_small/gen_script_long_order3_t5_small_specroute_v2.sh

@@ -0,0 +1,893 @@
+#!/bin/bash
+#SBATCH -J cl
+#SBATCH -o cl-%j.out
+#SBATCH -p compute
+#SBATCH -N 1
+#SBATCH -t 20:00:00
+#SBATCH --mem 128G
+#SBATCH --gres=gpu:2
+
+export CUDA_DEVICE_ORDER="PCI_BUS_ID"
+
+port=$(shuf -i25000-30000 -n1)
+
+# ============================================================
+# Auto-detect GPU count and type for optimal parallelism
+# ============================================================
+NUM_GPUS=$(nvidia-smi -L 2>/dev/null | wc -l)
+GPU_MEM=$(nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits 2>/dev/null | head -1)
+
+if [ -z "$GPU_MEM" ]; then
+    echo "ERROR: No GPU detected!"
+    exit 1
+fi
+
+# Determine GPU type
+if [ "$GPU_MEM" -lt 20000 ]; then
+    IS_T4=1
+    echo "[GPU] Detected T4 GPUs (${GPU_MEM}MB VRAM each)"
+else
+    IS_T4=0
+    echo "[GPU] Detected high-memory GPUs (${GPU_MEM}MB VRAM each)"
+fi
+
+# Determine parallelism strategy
+if [ "$IS_T4" -eq 1 ] && [ "$NUM_GPUS" -ge 2 ]; then
+    GPU_MODE="t4_2gpu"
+    GPU_IDS="0,1"
+    FP16_FLAG=""
+    echo "[GPU] Strategy: 2x T4 DataParallel + fp32 + gradient_checkpointing"
+elif [ "$IS_T4" -eq 1 ]; then
+    GPU_MODE="t4_1gpu"
+    GPU_IDS="${1:-0}"
+    FP16_FLAG=""
+    echo "[GPU] Strategy: 1x T4 + fp32 + gradient_checkpointing"
+else
+    GPU_MODE="a100"
+    GPU_IDS="${1:-0}"
+    FP16_FLAG=""
+    echo "[GPU] Strategy: A100 (single GPU, fp32)"
+fi
+
+echo "[GPU] Using CUDA_VISIBLE_DEVICES=$GPU_IDS"
+echo "============================================================"
+echo ""
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/yelp \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/1-yelp \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v2 \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match \
+   --evaluation_strategy steps \
+   --save_strategy steps \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq 5 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --kl_ratio 0.1 \
+   --gen_data_dir CL_Benchmark \
+   --threshold 0.980 \
+   --transthreshold 0.980 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/1-yelp/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/1-yelp/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/amazon \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/2-amazon \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v2 \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_amazon \
+   --evaluation_strategy steps \
+   --save_strategy steps \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq 5 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --kl_ratio 0.1 \
+   --gen_data_dir CL_Benchmark \
+   --threshold 0.980 \
+   --transthreshold 0.980 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/2-amazon/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/2-amazon/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/mnli \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/3-mnli \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v2 \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_mnli \
+   --evaluation_strategy steps \
+   --save_strategy steps \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq 5 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --kl_ratio 0.1 \
+   --gen_data_dir CL_Benchmark \
+   --threshold 0.980 \
+   --transthreshold 0.980 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/3-mnli/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/3-mnli/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/cb \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/4-cb \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v2 \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_cb \
+   --evaluation_strategy steps \
+   --save_strategy steps \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq 5 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --kl_ratio 0.1 \
+   --gen_data_dir CL_Benchmark \
+   --threshold 0.980 \
+   --transthreshold 0.980 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/4-cb/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/4-cb/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/copa \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/5-copa \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v2 \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_copa \
+   --evaluation_strategy steps \
+   --save_strategy steps \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq 5 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --kl_ratio 0.1 \
+   --gen_data_dir CL_Benchmark \
+   --threshold 0.980 \
+   --transthreshold 0.980 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/5-copa/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/5-copa/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/qqp \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/6-qqp \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v2 \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_qqp \
+   --evaluation_strategy steps \
+   --save_strategy steps \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq 5 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --kl_ratio 0.1 \
+   --gen_data_dir CL_Benchmark \
+   --threshold 0.980 \
+   --transthreshold 0.980 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/6-qqp/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/6-qqp/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/rte \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/7-rte \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v2 \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_rte \
+   --evaluation_strategy steps \
+   --save_strategy steps \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq 5 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --kl_ratio 0.1 \
+   --gen_data_dir CL_Benchmark \
+   --threshold 0.980 \
+   --transthreshold 0.980 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/7-rte/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/7-rte/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/imdb \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/8-imdb \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v2 \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_imdb \
+   --evaluation_strategy steps \
+   --save_strategy steps \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq 5 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --kl_ratio 0.1 \
+   --gen_data_dir CL_Benchmark \
+   --threshold 0.980 \
+   --transthreshold 0.980 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/8-imdb/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/7-rte/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/8-imdb/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/sst2 \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/9-sst2 \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v2 \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_sst2 \
+   --evaluation_strategy steps \
+   --save_strategy steps \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq 5 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --kl_ratio 0.1 \
+   --gen_data_dir CL_Benchmark \
+   --threshold 0.980 \
+   --transthreshold 0.980 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/9-sst2/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/7-rte/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/8-imdb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/9-sst2/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/dbpedia \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/10-dbpedia \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v2 \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_dbpedia \
+   --evaluation_strategy steps \
+   --save_strategy steps \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq 5 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --kl_ratio 0.1 \
+   --gen_data_dir CL_Benchmark \
+   --threshold 0.980 \
+   --transthreshold 0.980 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/10-dbpedia/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/7-rte/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/8-imdb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/9-sst2/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/10-dbpedia/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/agnews \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/11-agnews \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v2 \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_agnews \
+   --evaluation_strategy steps \
+   --save_strategy steps \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq 5 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --kl_ratio 0.1 \
+   --gen_data_dir CL_Benchmark \
+   --threshold 0.980 \
+   --transthreshold 0.980 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/11-agnews/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/7-rte/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/8-imdb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/9-sst2/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/10-dbpedia/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/11-agnews/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/yahoo \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/12-yahoo \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v2 \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_yahoo \
+   --evaluation_strategy steps \
+   --save_strategy steps \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq 5 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --kl_ratio 0.1 \
+   --gen_data_dir CL_Benchmark \
+   --threshold 0.980 \
+   --transthreshold 0.980 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/12-yahoo/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/7-rte/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/8-imdb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/9-sst2/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/10-dbpedia/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/11-agnews/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/12-yahoo/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/multirc \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/13-multirc \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v2 \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_multirc \
+   --evaluation_strategy steps \
+   --save_strategy steps \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq 5 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --kl_ratio 0.1 \
+   --gen_data_dir CL_Benchmark \
+   --threshold 0.980 \
+   --transthreshold 0.980 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/13-multirc/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/7-rte/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/8-imdb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/9-sst2/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/10-dbpedia/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/11-agnews/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/12-yahoo/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/13-multirc/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/boolq \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/14-boolq \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v2 \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_boolq \
+   --evaluation_strategy steps \
+   --save_strategy steps \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq 5 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --kl_ratio 0.1 \
+   --gen_data_dir CL_Benchmark \
+   --threshold 0.980 \
+   --transthreshold 0.980 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/14-boolq/checkpoint*
+
+sleep 5
+
+if [ "$GPU_MODE" = "t4_2gpu" ]; then
+    BSZ=16; GA=1; EVAL_BSZ=256
+elif [ "$GPU_MODE" = "t4_1gpu" ]; then
+    BSZ=32; GA=1; EVAL_BSZ=256
+else
+    BSZ=64; GA=1; EVAL_BSZ=512
+fi
+
+CUDA_VISIBLE_DEVICES=$GPU_IDS python src/run_t5.py \
+   --do_train \
+   --do_predict \
+   --predict_with_generate \
+   --model_name_or_path $2 \
+   --previous_lora_path logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/1-yelp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/2-amazon/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/3-mnli/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/4-cb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/5-copa/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/6-qqp/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/7-rte/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/8-imdb/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/9-sst2/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/10-dbpedia/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/11-agnews/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/12-yahoo/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/13-multirc/saved_weights,logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/14-boolq/saved_weights \
+   --data_dir CL_Benchmark \
+   --task_order yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic \
+   --task_config_dir configs/gen_script_long_order3_t5_configs/wic \
+   --output_dir logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/15-wic \
+   --per_device_train_batch_size $BSZ \
+   --per_device_eval_batch_size $EVAL_BSZ \
+   --gradient_accumulation_steps $GA \
+   --learning_rate 0.0003 \
+   --num_train_epochs 10 \
+   --run_name gen_script_long_order3_t5_small_specroute_v2 \
+   --max_source_length 512 \
+   --max_target_length 50 \
+   --generation_max_length 50 \
+   --add_task_name False \
+   --add_dataset_name False \
+   --overwrite_output_dir \
+   --overwrite_cache \
+   --lr_scheduler_type constant \
+   --warmup_steps 0 \
+   --logging_strategy steps \
+   --logging_steps 10 \
+   --metric_for_best_model eval_exact_match_for_wic \
+   --evaluation_strategy steps \
+   --save_strategy steps \
+   --save_total_limit 1 \
+   --load_best_model_at_end \
+   --lora_r 8 \
+   --lora_alpha 32 \
+   --lora_dropout 0.0 \
+   --data_replay_freq 5 \
+   --mlp_hidden_dim 100 \
+   --model_name specroute \
+   --kl_ratio 0.1 \
+   --gen_data_dir CL_Benchmark \
+   --threshold 0.980 \
+   --transthreshold 0.980 \
+   $FP16_FLAG
+
+rm -rf logs_and_outputs/gen_script_long_order3_t5_small_specroute_v2/outputs/15-wic/checkpoint*
+
+sleep 5

improve_gainlora/generate_specroute_scripts_v2.py

@@ -107,7 +107,7 @@ SCRIPT_CONFIGS = {
         "lora_r": 8,
         "epochs": 10,
         "metric_base": "eval_exact_match",
-        "do_predict": False,
+        "do_predict": True,
         "cleanup_checkpoints": True,
         "batch_a100_task1": (8, 4),
         "batch_a100_rest": (16, 2),
@@ -125,7 +125,7 @@ SCRIPT_CONFIGS = {
         "lora_r": 8,
         "epochs": 10,
         "metric_base": "eval_exact_match",
-        "do_predict": False,
+        "do_predict": True,
         "cleanup_checkpoints": True,
         "batch_a100_task1": (8, 4),
         "batch_a100_rest": (16, 2),

improve_gainlora/src/cl_trainer_specroute.py

@@ -68,9 +68,21 @@ class DenserEvalCallback(TrainerCallback):
         return control
 
 
+def create_memory_replay_generators(task, task_list, replay_data_dict):
+    """Create cycling iterators for previous tasks' replay data."""
+    print('Creating generators for previous tasks (SpecRoute replay) ...')
+    tasks_to_generators = {}
+    curr_task_num = task_list.index(task)
+    for idx in np.arange(curr_task_num):
+        prev_task = task_list[idx]
+        tasks_to_generators[prev_task] = iter(replay_data_dict[prev_task])
+    return tasks_to_generators
+
+
 class SpecRoute_Trainer(Seq2SeqTrainer):
 
     def __init__(self, model, args, train_dataset, cur_task_id, task_order,
+                 data_collator_replay=None, replay_dataset_dict=None,
                  eval_dataset=None, tokenizer=None, data_collator=None,
                  compute_metrics=None, callbacks=None):
         super().__init__(
@@ -82,7 +94,32 @@ class SpecRoute_Trainer(Seq2SeqTrainer):
         self.task_order = task_order
         self.cur_task_id = cur_task_id
         self._grad_check_done = False
-        # No replay data needed for SpecRoute
+
+        # Experience replay setup
+        self.data_collator_replay = data_collator_replay
+        self.replay_dataset_dict = replay_dataset_dict
+        if self.args.data_replay_freq != -1 and replay_dataset_dict is not None:
+            from torch.utils.data import RandomSampler
+            from transformers.trainer_utils import seed_worker
+            seed = self.args.data_seed if self.args.data_seed is not None else self.args.seed
+            generator = torch.Generator()
+            generator.manual_seed(seed)
+            self.replay_dataloader_dict = {}
+            for dataset_name, dataset in self.replay_dataset_dict.items():
+                train_sampler = RandomSampler(dataset, generator=generator)
+                self.replay_dataloader_dict[dataset_name] = DataLoader(
+                    dataset,
+                    batch_size=self._train_batch_size,
+                    sampler=train_sampler,
+                    collate_fn=self.data_collator_replay,
+                    drop_last=self.args.dataloader_drop_last,
+                    num_workers=self.args.dataloader_num_workers,
+                    pin_memory=False,
+                    worker_init_fn=seed_worker)
+            self.replay_iterator_dict = create_memory_replay_generators(
+                task_order[cur_task_id], task_order, self.replay_dataloader_dict)
+            print(f"[SpecRoute Replay] Enabled: {len(self.replay_dataloader_dict)} tasks, "
+                  f"freq={self.args.data_replay_freq}, ratio={self.args.kl_ratio}")
 
     def _save(self, output_dir=None, state_dict=None):
         # T5 shared embeddings are incompatible with safetensors; force pytorch format
@@ -94,8 +131,56 @@ class SpecRoute_Trainer(Seq2SeqTrainer):
             self.args.save_safetensors = old
 
     def training_step(self, model, inputs, **kwargs):
-        """Override to add one-time gradient diagnostic."""
-        loss = super().training_step(model, inputs, **kwargs)
+        """Override to add experience replay and one-time gradient diagnostic."""
+        model.train()
+        inputs = self._prepare_inputs(inputs)
+
+        with self.compute_loss_context_manager():
+            loss = self.compute_loss(model, inputs)
+
+        if self.args.n_gpu > 1:
+            loss = loss.mean()
+
+        if self.args.gradient_accumulation_steps > 1 and not self.is_deepspeed_enabled:
+            loss = loss / self.args.gradient_accumulation_steps
+
+        if self.is_deepspeed_enabled:
+            self.accelerator.backward(loss)
+        else:
+            loss.backward()
+
+        # === Experience Replay: CE loss on old task data ===
+        replay_freq = getattr(self.args, 'data_replay_freq', -1)
+        if (replay_freq != -1
+                and hasattr(self, 'replay_iterator_dict')
+                and self.state.global_step > getattr(self.args, 'replay_after_n_epoch', 0) * getattr(self.args, 'step_per_epoch', 0)
+                and self.state.global_step % replay_freq == 0):
+            for item in list(self.replay_iterator_dict.keys()):
+                generator_mem = self.replay_iterator_dict[item]
+                try:
+                    b = next(generator_mem)
+                except StopIteration:
+                    generator_mem = iter(self.replay_dataloader_dict[item])
+                    self.replay_iterator_dict[item] = generator_mem
+                    b = next(generator_mem)
+
+                # Remove replay_labels if present (not needed for CE replay)
+                b.pop("replay_labels", None)
+                replay_inputs = self._prepare_inputs(b)
+
+                with self.compute_loss_context_manager():
+                    replay_loss = self.compute_loss(model, replay_inputs)
+
+                kl_ratio = getattr(self.args, 'kl_ratio', 0.1)
+                replay_loss = kl_ratio * replay_loss
+
+                if self.args.n_gpu > 1:
+                    replay_loss = replay_loss.mean()
+
+                if self.is_deepspeed_enabled:
+                    self.accelerator.backward(replay_loss)
+                else:
+                    replay_loss.backward()
 
         # One-time gradient check after first backward
         if not self._grad_check_done:

improve_gainlora/src/run_t5.py

@@ -707,7 +707,9 @@ def main():
             input_record_file=data_args.input_record_file)
 
         replay_dataset_dict, replay_label_dict = None, None
-        if model_args.load_checkpoint_from:
+        # Load replay datasets for methods that need it
+        _need_replay_data = model_args.load_checkpoint_from or (training_args.model_name == 'specroute' and cur_task_id > 0)
+        if _need_replay_data:
             replay_dataset_dict = {}
             abs_data_dir_replay = os.path.abspath(data_dir) if data_dir else None
             for idx in range(cur_task_id):
@@ -725,12 +727,14 @@ def main():
                 replay_dataset_dict[task_order[idx]] = raw_datasets_gen["train"]
                 print(raw_datasets_gen)
 
-            replay_label_dict = {}
-            for idx in range(0,cur_task_id):
-                with open(os.path.join("../logs_and_outputs/" + training_args.run_name + "/outputs/", str(idx+1)+"-"+task_order[idx], "saved_weights", "attention_weights.pkl"), 'rb') as f:
-                    attn_weights = pickle.load(f)
-                replay_label_dict[task_order[idx]] = torch.cat([torch.tensor([0.] * (cur_task_id - idx)), torch.tensor(attn_weights)], dim=0).to(dtype=torch.bfloat16, device='cuda')
-            print(replay_label_dict)
+            # Load attention weights for KL replay (GainLoRA only, not SpecRoute)
+            if model_args.load_checkpoint_from:
+                replay_label_dict = {}
+                for idx in range(0,cur_task_id):
+                    with open(os.path.join("../logs_and_outputs/" + training_args.run_name + "/outputs/", str(idx+1)+"-"+task_order[idx], "saved_weights", "attention_weights.pkl"), 'rb') as f:
+                        attn_weights = pickle.load(f)
+                    replay_label_dict[task_order[idx]] = torch.cat([torch.tensor([0.] * (cur_task_id - idx)), torch.tensor(attn_weights)], dim=0).to(dtype=torch.bfloat16, device='cuda')
+                print(replay_label_dict)
         print('-'*50)
 
     # Metric
@@ -866,6 +870,8 @@ def main():
             train_dataset=train_dataset if training_args.do_train else None,
             cur_task_id=cur_task_id,
             task_order=task_order,
+            data_collator_replay=data_collator_replay,
+            replay_dataset_dict=replay_dataset_dict,
             eval_dataset=eval_dataset if training_args.do_eval else None,
             tokenizer=tokenizer,
             data_collator=data_collator,

results/comparison_results.md

@@ -152,7 +152,18 @@ python src/compute_ap_ft.py \
 | 13 | agnews | | | | |
 | 14 | multirc | | | | |
 | 15 | yahoo | | | | |
-| | **AP / FT** | **77.54 / 1.20** | | | |
+
+
+```bash
+# Chạy 4 lệnh này để lấy đủ số cho cả 2 bảng:
+python src/compute_ap_ft.py --output_base logs_and_outputs/ot_sign_order1_t5large/outputs --task_order "task1572_samsum_summary,task363_sst2_polarity_classification,task1290_xsum_summarization,task181_outcome_extraction,task002_quoref_answer_generation,task1510_evalution_relation_extraction,task639_multi_woz_user_utterance_generation,task1729_personachat_generate_next,task073_commonsenseqa_answer_generation,task1590_diplomacy_text_generation,task748_glucose_reverse_cause_event_detection,task511_reddit_tifu_long_text_summarization,task591_sciq_answer_generation,task1687_sentiment140_classification,task875_emotion_classification" --save
+
+python src/compute_ap_ft.py --output_base logs_and_outputs/ot_sign_order2_t5large/outputs --task_order "task748_glucose_reverse_cause_event_detection,task073_commonsenseqa_answer_generation,task1590_diplomacy_text_generation,task639_multi_woz_user_utterance_generation,task1572_samsum_summary,task1687_sentiment140_classification,task591_sciq_answer_generation,task363_sst2_polarity_classification,task1510_evalution_relation_extraction,task1729_personachat_generate_next,task181_outcome_extraction,task511_reddit_tifu_long_text_summarization,task002_quoref_answer_generation,task1290_xsum_summarization,task875_emotion_classification" --save
+
+python src/compute_ap_ft.py --output_base logs_and_outputs/ot_sign_order3_t5large/outputs --task_order "yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic" --save
+
+python src/compute_ap_ft.py --output_base logs_and_outputs/ot_sign_order4_t5large/outputs --task_order "mnli,cb,wic,copa,qqp,boolq,rte,imdb,yelp,amazon,sst2,dbpedia,agnews,multirc,yahoo" --save
+```
 
 ---
 
@@ -160,43 +171,34 @@ python src/compute_ap_ft.py \
 
 | Method | Order 3 AP↑ | Order 3 FT↓ |
 |--------|-------------|-------------|
-| **GainLoRA (Root)** | 59.70 | N/A* |
-| **SpecRoute (Improve)** | *(chờ chạy)* | *(chờ chạy)* |
+| **GainLoRA (Root)** | **59.70** | N/A* |
+| **SpecRoute (Improve)** | 39.74† | N/A* |
 
-> *\*FT = N/A: Log chỉ chứa Evaluation ở task cuối (15-wic). Các task trước không có bước `--do_predict` để xuất cross-task matrix. Lần chạy sau dùng script trong `T5_small/` đã được sửa để có FT.*
+> *\*FT = N/A: cả 2 log chạy thiếu `--do_predict`. Lần tiếp theo dùng script `T5_small/` đã sửa sẽ có đủ FT.*  
+> *†Điểm Improve tính từ `predict_eval_predictions.jsonl` của từng task (hàng chéo score matrix). imdb/sst2/wic về 0 do Catastrophic Forgetting.*
 
-## Per-Task Breakdown — Order 3 (T5-Small)
+### ⚠️ Root GainLoRA tốt hơn SpecRoute trên T5-Small (−19.96 AP)
 
-| # | Task | GainLoRA (Root) Final | SpecRoute (Improve) Final |
-|---|------|-----------------------|---------------------------|
-| 1 | yelp | 56.01 | |
-| 2 | amazon | 52.05 | |
-| 3 | mnli | 34.07 | |
-| 4 | cb | 3.57 | |
-| 5 | copa | 42.00 | |
-| 6 | qqp | 76.96 | |
-| 7 | rte | 45.85 | |
-| 8 | imdb | 89.51 | |
-| 9 | sst2 | 85.21 | |
-| 10 | dbpedia | 98.16 | |
-| 11 | agnews | 88.37 | |
-| 12 | yahoo | 57.28 | |
-| 13 | multirc | 50.52 | |
-| 14 | boolq | 60.43 | |
-| 15 | wic | 55.49 | |
-| | **AP / FT** | **59.70 / N/A** | |
+SpecRoute bị Catastrophic Forgetting nghiêm trọng ở các task phân loại sentiment (imdb=0.21, sst2=0.00, yahoo=8.12, wic=0.00). Nguyên nhân có thể do SVD rank không đủ lớn ở T5-Small, làm routing mechanism không phân tách được subspace của các task.
 
----
-
-## Quick Harvest (chạy sau khi xong cả 4 orders)
-
-```bash
-# Chạy 4 lệnh này để lấy đủ số cho cả 2 bảng:
-python src/compute_ap_ft.py --output_base logs_and_outputs/ot_sign_order1_t5large/outputs --task_order "task1572_samsum_summary,task363_sst2_polarity_classification,task1290_xsum_summarization,task181_outcome_extraction,task002_quoref_answer_generation,task1510_evalution_relation_extraction,task639_multi_woz_user_utterance_generation,task1729_personachat_generate_next,task073_commonsenseqa_answer_generation,task1590_diplomacy_text_generation,task748_glucose_reverse_cause_event_detection,task511_reddit_tifu_long_text_summarization,task591_sciq_answer_generation,task1687_sentiment140_classification,task875_emotion_classification" --save
-
-python src/compute_ap_ft.py --output_base logs_and_outputs/ot_sign_order2_t5large/outputs --task_order "task748_glucose_reverse_cause_event_detection,task073_commonsenseqa_answer_generation,task1590_diplomacy_text_generation,task639_multi_woz_user_utterance_generation,task1572_samsum_summary,task1687_sentiment140_classification,task591_sciq_answer_generation,task363_sst2_polarity_classification,task1510_evalution_relation_extraction,task1729_personachat_generate_next,task181_outcome_extraction,task511_reddit_tifu_long_text_summarization,task002_quoref_answer_generation,task1290_xsum_summarization,task875_emotion_classification" --save
+## Per-Task Breakdown — Order 3 (T5-Small)
 
-python src/compute_ap_ft.py --output_base logs_and_outputs/ot_sign_order3_t5large/outputs --task_order "yelp,amazon,mnli,cb,copa,qqp,rte,imdb,sst2,dbpedia,agnews,yahoo,multirc,boolq,wic" --save
+| # | Task | GainLoRA (Root) | SpecRoute (Improve) | Δ (Improve−Root) |
+|---|------|-----------------|--------------------|-----------------|
+| 1 | yelp | 56.01 | 54.36 | −1.65 |
+| 2 | amazon | 52.05 | 50.01 | −2.04 |
+| 3 | mnli | 34.07 | 35.50 | +1.43 |
+| 4 | cb | 3.57 | 0.00 | −3.57 |
+| 5 | copa | 42.00 | 44.00 | +2.00 |
+| 6 | qqp | 76.96 | 76.72 | −0.24 |
+| 7 | rte | 45.85 | 50.90 | +5.05 |
+| 8 | imdb | 89.51 | 0.21 | **−89.30 ⚠️** |
+| 9 | sst2 | 85.21 | 0.00 | **−85.21 ⚠️** |
+| 10 | dbpedia | 98.16 | 92.22 | −5.94 |
+| 11 | agnews | 88.37 | 68.76 | −19.61 |
+| 12 | yahoo | 57.28 | 8.12 | **−49.16 ⚠️** |
+| 13 | multirc | 50.52 | 54.23 | +3.71 |
+| 14 | boolq | 60.43 | 61.13 | +0.70 |
+| 15 | wic | 55.49 | 0.00 | **−55.49 ⚠️** |
+| | **AP / FT** | **59.70 / N/A** | **39.74 / N/A** | **−19.96** |
 
-python src/compute_ap_ft.py --output_base logs_and_outputs/ot_sign_order4_t5large/outputs --task_order "mnli,cb,wic,copa,qqp,boolq,rte,imdb,yelp,amazon,sst2,dbpedia,agnews,multirc,yahoo" --save
-```

results/experiment_versions.md

@@ -0,0 +1,167 @@
+# SpecRoute — Báo cáo Thử nghiệm theo Version
+
+> Tracking tất cả versions thử nghiệm, kết quả, phân tích, và cải tiến.  
+> Benchmark: Long Sequence Order 3, 15 classification tasks, model T5-Small.
+
+---
+
+## Version 1.0 — Baseline SpecRoute (Kết quả đầu tiên)
+
+### Kịch bản thử nghiệm
+- **Model**: T5-Small (d_model=512, 6 encoder + 6 decoder layers)
+- **Method**: SpecRoute — spectral routing (SVD of LoRA B@A) thay thế learned routing (trans_input + prompt_key) của GainLoRA
+- **So sánh**: ROOT GainLoRA-InfLoRA (original codebase)
+- **Hyperparameters**: lora_r=8, lora_alpha=32, lr=3e-4, 10 epochs, threshold=0.995
+- **Platform**: Kaggle T4 GPU
+
+### Kết quả
+
+| # | Task | ROOT (Final R_{15,j}) | SpecRoute (Peak R_{j,j}) | Δ |
+|---|------|-----------------------|--------------------------|---|
+| 1 | yelp | 56.01 | 54.36 | -1.65 |
+| 2 | amazon | 52.05 | 50.01 | -2.04 |
+| 3 | mnli | 34.07 | 35.50 | +1.43 |
+| 4 | cb | 3.57 | 0.00 | -3.57 |
+| 5 | copa | 42.00 | 44.00 | +2.00 |
+| 6 | qqp | 76.96 | 76.72 | -0.24 |
+| 7 | rte | 45.85 | 50.90 | +5.05 |
+| 8 | imdb | 89.51 | **0.21** ⚠️ | -89.30 |
+| 9 | sst2 | 85.21 | **0.00** ⚠️ | -85.21 |
+| 10 | dbpedia | 98.16 | 92.22 | -5.94 |
+| 11 | agnews | 88.37 | 68.76 | -19.61 |
+| 12 | yahoo | 57.28 | **8.12** ⚠️ | -49.16 |
+| 13 | multirc | 50.52 | 54.23 | +3.71 |
+| 14 | boolq | 60.43 | 61.13 | +0.70 |
+| 15 | wic | 55.49 | **0.00** ⚠️ | -55.49 |
+| | **Mean** | **59.70** | **39.74** | **-19.96** |
+
+> ⚠️ **LƯU Ý QUAN TRỌNG**: So sánh KHÔNG công bằng — ROOT dùng R_{15,j} (final, sau tất cả 15 tasks), SpecRoute dùng R_{j,j} (peak, ngay sau train từng task). AP thực của SpecRoute sẽ thấp hơn 39.74.
+
+### Phân tích
+
+**1. Prediction metrics không được lưu**  
+- SpecRoute `all_results.json` chỉ chứa training metrics, KHÔNG có `predict_exact_match_for_{task}`
+- `task_order.txt` không tồn tại → `score.py` không thể tính AP/FT
+- Nguyên nhân: Có thể do experiment được chạy bằng script khác (không phải T5_small/ scripts đã fix `--do_predict`)
+- T5-large script generator (`generate_specroute_scripts_v2.py`) vẫn có bug `do_predict=False` cho long benchmarks
+
+**2. Các tasks THẤT BẠI KHÔNG PHẢI do catastrophic forgetting**  
+
+| Task | Train Loss (Root) | Train Loss (SpecRoute) | Ratio | Verdict |
+|------|:-:|:-:|:-:|---|
+| imdb | 1.41 | **4.15** | 2.9x | Không thể học |
+| sst2 | 1.76 | **4.45** | 2.5x | Không thể học |
+| yahoo | 1.19 | **3.08** | 2.6x | Không thể học |
+| wic | 0.96 | **3.65** | 3.8x | Không thể học |
+
+Training loss cao gấp 2.5-3.8x → model KHÔNG THỂ HỌC ngay từ đầu (inability to learn, NOT catastrophic forgetting).
+
+**3. Nguyên nhân gốc: GPM null-space saturation + thiếu protection mechanisms**
+
+SpecRoute loại bỏ learned routing → đồng thời mất 4/5 cơ chế protection của ROOT:
+
+| Protection Mechanism | ROOT | SpecRoute V1 |
+|---------------------|:---:|:---:|
+| GPM on LoRA A | ✅ | ✅ |
+| KL distillation on routing | ✅ | ❌ |
+| Data replay | ✅ | ❌ |
+| Per-step GPM on routing params | ✅ | ❌ (no routing params) |
+| Learned routing adaptation | ✅ | ❌ (by design) |
+
+Khi tasks tương tự (imdb/sst2 vs yelp/amazon — cùng sentiment domain) đến, GPM đã "claim" sentiment-relevant directions → model bị ép vào orthogonal null-space không liên quan → KHÔNG thể học sentiment tasks mới.
+
+ROOT GainLoRA giải quyết vấn đề này nhờ trans_input MLP map input mới vào representation space REUSE kiến thức cũ, kết hợp KL distillation + data replay.
+
+**4. FT (Forgetting) = N/A**  
+- Không tính được vì thiếu cross-task prediction metrics
+
+### Cải tiến cho V2
+
+| # | Loại | Nội dung | Tác động |
+|---|------|---------|----------|
+| 1 | Bug fix | Fix `do_predict=False` → `True` trong generator | Cho phép tính AP/FT đúng |
+| 2 | Config | Giảm GPM threshold: 0.995 → 0.980 | Mở rộng null-space cho tasks sau |
+| 3 | **Idea change** | Thêm Experience Replay (CE loss trên old task data) | Chống forgetting + hỗ trợ knowledge reuse |
+
+---
+
+## Version 2.0 — SpecRoute + Experience Replay (Planned)
+
+### Thay đổi về Idea
+
+> **⚠️ IDEA CHANGE**: Version 2 thêm **Experience Replay (CE loss)** vào SpecRoute.
+> 
+> SpecRoute V1 claim rằng spectral routing parameter-free đủ để thay thế learned routing. V2 bổ sung rằng:
+> - Spectral routing thay thế **routing mechanism** (đúng, giữ nguyên)
+> - Nhưng **protection mechanisms** (data replay) là ORTHOGONAL với routing mechanism và cần được giữ lại
+> - V2 sử dụng **CE replay trực tiếp** trên old task training data (không cần teacher model hay saved logits)
+> - Khác ROOT (KL on routing scores): SpecRoute replay chỉ cần CE loss vì routing là parameter-free
+>
+> Đây là sự thay đổi từ "spectral routing is sufficient" sang "spectral routing + replay protection is the complete solution".
+> Bản chất: **decouple routing mechanism khỏi protection mechanisms**.
+
+### Kịch bản thử nghiệm
+- **Model**: T5-Small (d_model=512, 6 encoder + 6 decoder layers)
+- **Method**: SpecRoute V2 — spectral routing + experience replay (CE loss trên original training data)
+- **Hyperparameters**: 
+  - lora_r=8, lora_alpha=32, lr=3e-4, 10 epochs
+  - **threshold=0.980** (giảm từ 0.995)
+  - **data_replay_freq=5** (replay mỗi 5 steps)
+  - **kl_ratio=0.1** (weight cho replay CE loss)
+  - **gen_data_dir=CL_Benchmark** (replay từ original training data)
+- **Script**: `T5_small/gen_script_long_order3_t5_small_specroute_v2.sh`
+- **Platform**: Kaggle T4 GPU
+
+### Code Changes (Actual)
+
+**1. Bug Fix: `generate_specroute_scripts_v2.py`**  
+- `do_predict=False` → `True` cho `long_order3` và `long_order4`
+
+**2. Trainer: `cl_trainer_specroute.py`**  
+- Thêm `create_memory_replay_generators()` — tạo DataLoader cycling iterators  
+- `__init__()`: nhận `data_collator_replay`, `replay_dataset_dict`, tạo `replay_dataloader_dict` và `replay_iterator_dict`
+- `training_step()`: Sau main CE loss backward, replay CE loss trên old task data:
+  ```
+  Mỗi replay_freq steps:
+    For each old task:
+      sample batch from replay iterator
+      replay_loss = kl_ratio * CE_loss(model, replay_batch)
+      replay_loss.backward()
+  ```
+
+**3. Run entry: `run_t5.py`**  
+- Mở rộng replay dataset loading condition: `load_checkpoint_from OR (specroute AND cur_task_id > 0)`
+- Skip `attention_weights.pkl` loading cho SpecRoute (không cần KL on routing)
+- Pass `data_collator_replay`, `replay_dataset_dict` vào SpecRoute_Trainer
+
+**4. Shell Script: `T5_small/gen_script_long_order3_t5_small_specroute_v2.sh`** (NEW)  
+- threshold: 0.995 → 0.980
+- data_replay_freq: -1 → 5
+- Thêm: `--kl_ratio 0.1`, `--gen_data_dir CL_Benchmark`  
+- Output dir: `specroute_v2` (tách biệt V1)
+- V1 script giữ nguyên để so sánh
+
+### Kết quả
+> *Chưa chạy — cần thực nghiệm*
+
+### Phân tích
+> *Pending*
+
+### Kỳ vọng
+- Tasks 8 (imdb), 9 (sst2), 12 (yahoo), 15 (wic): kỳ vọng cải thiện đáng kể nhờ threshold thấp hơn (mở rộng null-space)
+- Overall AP: kỳ vọng tăng từ ~39.74 lên >50 (threshold fix), replay CE giúp chống forgetting
+- FT: kỳ vọng tính được (do_predict fix) và forgetting thấp hơn nhờ replay
+
+### Nếu kết quả không đạt → V3 Plan
+- **V3a**: Thêm output-level KL distillation (so sánh logits hiện tại vs teacher model snapshot) — yêu cầu lưu teacher model
+- **V3b**: Thêm adaptive threshold per-layer (thay vì cùng threshold cho tất cả layers)  
+- **V3c**: SpecRoute + InfLoRA-style direction expansion khi null-space quá nhỏ
+
+---
+
+## Changelog
+
+| Date | Version | Change Type | Description |
+|------|---------|-------------|-------------|
+| 2025-XX-XX | V1.0 | Initial | First experiment — baseline SpecRoute vs ROOT GainLoRA |
+| 2025-XX-XX | V2.0 | Idea + Code | Thêm experience replay (CE), giảm threshold 0.995→0.980, fix do_predict |

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/all_results.json

@@ -1,9 +0,0 @@
-{
-    "epoch": 10.0,
-    "total_flos": 9122411270725632.0,
-    "train_loss": 0.5864024265556579,
-    "train_runtime": 1102.9262,
-    "train_samples": 5000,
-    "train_samples_per_second": 45.334,
-    "train_steps_per_second": 1.423
-}

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/eval_eval_predictions.jsonl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:f0a7a162013858ce8c643d5f66d3b834d8a573de497cec931a216f99e1f0178c
-size 8615403

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/predict_eval_predictions.jsonl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:f0a7a162013858ce8c643d5f66d3b834d8a573de497cec931a216f99e1f0178c
-size 8615403

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_0.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:6347c1939a53ee2509742a6aa5db72d302a9f33b13fbe90ef95a10f9e9221bb8
-size 1049682

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_1.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:843fd6f47f02250c381b8489f82b7adb7cce5852b8dd463066395623cd42ca7c
-size 1049682

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_10.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:222a5768a659ab676766654ca8e142f24f7c0761be3b55d63f9a6071d64207ac
-size 1049687

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_11.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:2b98abe23b3517987bda49db3b1501a0de290ccfe1293476c85ef9fb06ae1af3
-size 1049687

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_12.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:b4d217ec73e80746735689b52de8985bd9b8b37a3f804d87e8ea8399e5a7ce13
-size 1049687

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_13.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:b94c96b1e217047809307086ec824bac8630ccbd16d8431e1d4d0bd8cd9d9513
-size 1049687

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_14.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:3ddbf830599893c4a9f15ae1abf9f120a95063e6a53fde5581a64244edfccec8
-size 1049687

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_15.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:063e96e4e954283653178868f826503b2f5d8003f4317b6396df23f47ba1ef6a
-size 1049687

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_16.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:3e145f0f32e63de296a637276dfb148e216c770504333ea90f87f429486cd242
-size 1049687

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_17.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:6c388d5777061b3d716fa2d9f33b58fd6b88a00880991654d7d74ae7c2f8393f
-size 1049687

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_18.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:29cca1a96c16d78b45dec35a9fcd4f69fd78012d801f9ed7169d42b5d17b1818
-size 1049687

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_19.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:78e8649cda3b950cd839967a2be7347d627f1ca1fbe32e3fa2f0ef09852c531b
-size 1049687

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_2.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:741af2fdc11424d8a3b16fbee464fb95abe96251b5b96e2a6e9a48b3f91b0023
-size 1049682

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_20.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:dd3cbcd2c082eef395e674b4f6aff25636050063f66d7df24136a283756a85cb
-size 1049687

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_21.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:a0a27c57b783efd371db7bb7e49a6b9d088511a9a79b79a9cee12c6584fbf2ec
-size 1049687

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_22.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:bbcb435a6636e909e222a288c45038662e44bb1546fa20e602c6dd6bacc7a0d9
-size 1049687

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_23.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:86ed74a9a8c5fd74143b95b2d88ad0bfd2e5acb73a55a6de084b2007cd09ba2b
-size 1049687

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_3.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:102c178c81bf0eca52a54fd02f8748209231bce6bcf038daa330e8dedb6cb4e4
-size 1049682

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_4.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:764cf3f60cf57542a85cd7a24226830a47d0feda84fc32e6a8a1d9d9c498ebeb
-size 1049682

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_5.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:0249fa240b48cf18d6bc3638b6977ccbb993a5e4e3d924a7788a662c5668ffed
-size 1049682

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_6.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:bdd5d0c75a9037ff92d44e21c8ffbec5d32586e40cbdf5ba08b35f0d7ca3ec29
-size 1049682

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_7.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:82169d8abd9475d638fc237287bc2f5273f0a0e8b1e8bcccc0e4ad53c9c74958
-size 1049682

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_8.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:dff7efc08c970c36173502656d3837456b6910d5e20efcf2a8a70cfcbcf744ef
-size 1049682

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/reg_9.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:80e3d101a726351d264a702ff4dd91d81898206ab423198df379119272c1b3be
-size 1049682

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/saved_weights/lora_weights_A.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:e4b19d3392d606e6a2f01514dd365f84474ee8008946d2d8bcb59f543159160b
-size 803442

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/saved_weights/lora_weights_B.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:2771ba4fefea1b21526d4a4c06bf7b00075a15ddfcb60fb1ccff48d5551c3b6f
-size 606770

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/saved_weights/prompts_keys_till_now.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:f52669525937aba46986ba60df07fb81bedf3745928a864ae4c08d88fae6a069
-size 3298

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/saved_weights/special_tokens_map.json

@@ -1,125 +0,0 @@
-{
-  "additional_special_tokens": [
-    "<extra_id_0>",
-    "<extra_id_1>",
-    "<extra_id_2>",
-    "<extra_id_3>",
-    "<extra_id_4>",
-    "<extra_id_5>",
-    "<extra_id_6>",
-    "<extra_id_7>",
-    "<extra_id_8>",
-    "<extra_id_9>",
-    "<extra_id_10>",
-    "<extra_id_11>",
-    "<extra_id_12>",
-    "<extra_id_13>",
-    "<extra_id_14>",
-    "<extra_id_15>",
-    "<extra_id_16>",
-    "<extra_id_17>",
-    "<extra_id_18>",
-    "<extra_id_19>",
-    "<extra_id_20>",
-    "<extra_id_21>",
-    "<extra_id_22>",
-    "<extra_id_23>",
-    "<extra_id_24>",
-    "<extra_id_25>",
-    "<extra_id_26>",
-    "<extra_id_27>",
-    "<extra_id_28>",
-    "<extra_id_29>",
-    "<extra_id_30>",
-    "<extra_id_31>",
-    "<extra_id_32>",
-    "<extra_id_33>",
-    "<extra_id_34>",
-    "<extra_id_35>",
-    "<extra_id_36>",
-    "<extra_id_37>",
-    "<extra_id_38>",
-    "<extra_id_39>",
-    "<extra_id_40>",
-    "<extra_id_41>",
-    "<extra_id_42>",
-    "<extra_id_43>",
-    "<extra_id_44>",
-    "<extra_id_45>",
-    "<extra_id_46>",
-    "<extra_id_47>",
-    "<extra_id_48>",
-    "<extra_id_49>",
-    "<extra_id_50>",
-    "<extra_id_51>",
-    "<extra_id_52>",
-    "<extra_id_53>",
-    "<extra_id_54>",
-    "<extra_id_55>",
-    "<extra_id_56>",
-    "<extra_id_57>",
-    "<extra_id_58>",
-    "<extra_id_59>",
-    "<extra_id_60>",
-    "<extra_id_61>",
-    "<extra_id_62>",
-    "<extra_id_63>",
-    "<extra_id_64>",
-    "<extra_id_65>",
-    "<extra_id_66>",
-    "<extra_id_67>",
-    "<extra_id_68>",
-    "<extra_id_69>",
-    "<extra_id_70>",
-    "<extra_id_71>",
-    "<extra_id_72>",
-    "<extra_id_73>",
-    "<extra_id_74>",
-    "<extra_id_75>",
-    "<extra_id_76>",
-    "<extra_id_77>",
-    "<extra_id_78>",
-    "<extra_id_79>",
-    "<extra_id_80>",
-    "<extra_id_81>",
-    "<extra_id_82>",
-    "<extra_id_83>",
-    "<extra_id_84>",
-    "<extra_id_85>",
-    "<extra_id_86>",
-    "<extra_id_87>",
-    "<extra_id_88>",
-    "<extra_id_89>",
-    "<extra_id_90>",
-    "<extra_id_91>",
-    "<extra_id_92>",
-    "<extra_id_93>",
-    "<extra_id_94>",
-    "<extra_id_95>",
-    "<extra_id_96>",
-    "<extra_id_97>",
-    "<extra_id_98>",
-    "<extra_id_99>"
-  ],
-  "eos_token": {
-    "content": "</s>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  },
-  "pad_token": {
-    "content": "<pad>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  },
-  "unk_token": {
-    "content": "<unk>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  }
-}

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/saved_weights/spiece.model

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:d60acb128cf7b7f2536e8f38a5b18a05535c9e14c7a355904270e15b0945ea86
-size 791656

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/saved_weights/tokenizer_config.json

@@ -1,938 +0,0 @@
-{
-  "added_tokens_decoder": {
-    "0": {
-      "content": "<pad>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "1": {
-      "content": "</s>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "2": {
-      "content": "<unk>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32000": {
-      "content": "<extra_id_99>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32001": {
-      "content": "<extra_id_98>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32002": {
-      "content": "<extra_id_97>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32003": {
-      "content": "<extra_id_96>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32004": {
-      "content": "<extra_id_95>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32005": {
-      "content": "<extra_id_94>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32006": {
-      "content": "<extra_id_93>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32007": {
-      "content": "<extra_id_92>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32008": {
-      "content": "<extra_id_91>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32009": {
-      "content": "<extra_id_90>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32010": {
-      "content": "<extra_id_89>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32011": {
-      "content": "<extra_id_88>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32012": {
-      "content": "<extra_id_87>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32013": {
-      "content": "<extra_id_86>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32014": {
-      "content": "<extra_id_85>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32015": {
-      "content": "<extra_id_84>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32016": {
-      "content": "<extra_id_83>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32017": {
-      "content": "<extra_id_82>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32018": {
-      "content": "<extra_id_81>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32019": {
-      "content": "<extra_id_80>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32020": {
-      "content": "<extra_id_79>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32021": {
-      "content": "<extra_id_78>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32022": {
-      "content": "<extra_id_77>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32023": {
-      "content": "<extra_id_76>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32024": {
-      "content": "<extra_id_75>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32025": {
-      "content": "<extra_id_74>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32026": {
-      "content": "<extra_id_73>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32027": {
-      "content": "<extra_id_72>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32028": {
-      "content": "<extra_id_71>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32029": {
-      "content": "<extra_id_70>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32030": {
-      "content": "<extra_id_69>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32031": {
-      "content": "<extra_id_68>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32032": {
-      "content": "<extra_id_67>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32033": {
-      "content": "<extra_id_66>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32034": {
-      "content": "<extra_id_65>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32035": {
-      "content": "<extra_id_64>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32036": {
-      "content": "<extra_id_63>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32037": {
-      "content": "<extra_id_62>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32038": {
-      "content": "<extra_id_61>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32039": {
-      "content": "<extra_id_60>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32040": {
-      "content": "<extra_id_59>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32041": {
-      "content": "<extra_id_58>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32042": {
-      "content": "<extra_id_57>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32043": {
-      "content": "<extra_id_56>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32044": {
-      "content": "<extra_id_55>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32045": {
-      "content": "<extra_id_54>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32046": {
-      "content": "<extra_id_53>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32047": {
-      "content": "<extra_id_52>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32048": {
-      "content": "<extra_id_51>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32049": {
-      "content": "<extra_id_50>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32050": {
-      "content": "<extra_id_49>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32051": {
-      "content": "<extra_id_48>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32052": {
-      "content": "<extra_id_47>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32053": {
-      "content": "<extra_id_46>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32054": {
-      "content": "<extra_id_45>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32055": {
-      "content": "<extra_id_44>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32056": {
-      "content": "<extra_id_43>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32057": {
-      "content": "<extra_id_42>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32058": {
-      "content": "<extra_id_41>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32059": {
-      "content": "<extra_id_40>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32060": {
-      "content": "<extra_id_39>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32061": {
-      "content": "<extra_id_38>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32062": {
-      "content": "<extra_id_37>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32063": {
-      "content": "<extra_id_36>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32064": {
-      "content": "<extra_id_35>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32065": {
-      "content": "<extra_id_34>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32066": {
-      "content": "<extra_id_33>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32067": {
-      "content": "<extra_id_32>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32068": {
-      "content": "<extra_id_31>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32069": {
-      "content": "<extra_id_30>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32070": {
-      "content": "<extra_id_29>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32071": {
-      "content": "<extra_id_28>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32072": {
-      "content": "<extra_id_27>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32073": {
-      "content": "<extra_id_26>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32074": {
-      "content": "<extra_id_25>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32075": {
-      "content": "<extra_id_24>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32076": {
-      "content": "<extra_id_23>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32077": {
-      "content": "<extra_id_22>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32078": {
-      "content": "<extra_id_21>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32079": {
-      "content": "<extra_id_20>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32080": {
-      "content": "<extra_id_19>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32081": {
-      "content": "<extra_id_18>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32082": {
-      "content": "<extra_id_17>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32083": {
-      "content": "<extra_id_16>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32084": {
-      "content": "<extra_id_15>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32085": {
-      "content": "<extra_id_14>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32086": {
-      "content": "<extra_id_13>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32087": {
-      "content": "<extra_id_12>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32088": {
-      "content": "<extra_id_11>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32089": {
-      "content": "<extra_id_10>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32090": {
-      "content": "<extra_id_9>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32091": {
-      "content": "<extra_id_8>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32092": {
-      "content": "<extra_id_7>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32093": {
-      "content": "<extra_id_6>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32094": {
-      "content": "<extra_id_5>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32095": {
-      "content": "<extra_id_4>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32096": {
-      "content": "<extra_id_3>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32097": {
-      "content": "<extra_id_2>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32098": {
-      "content": "<extra_id_1>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "32099": {
-      "content": "<extra_id_0>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    }
-  },
-  "additional_special_tokens": [
-    "<extra_id_0>",
-    "<extra_id_1>",
-    "<extra_id_2>",
-    "<extra_id_3>",
-    "<extra_id_4>",
-    "<extra_id_5>",
-    "<extra_id_6>",
-    "<extra_id_7>",
-    "<extra_id_8>",
-    "<extra_id_9>",
-    "<extra_id_10>",
-    "<extra_id_11>",
-    "<extra_id_12>",
-    "<extra_id_13>",
-    "<extra_id_14>",
-    "<extra_id_15>",
-    "<extra_id_16>",
-    "<extra_id_17>",
-    "<extra_id_18>",
-    "<extra_id_19>",
-    "<extra_id_20>",
-    "<extra_id_21>",
-    "<extra_id_22>",
-    "<extra_id_23>",
-    "<extra_id_24>",
-    "<extra_id_25>",
-    "<extra_id_26>",
-    "<extra_id_27>",
-    "<extra_id_28>",
-    "<extra_id_29>",
-    "<extra_id_30>",
-    "<extra_id_31>",
-    "<extra_id_32>",
-    "<extra_id_33>",
-    "<extra_id_34>",
-    "<extra_id_35>",
-    "<extra_id_36>",
-    "<extra_id_37>",
-    "<extra_id_38>",
-    "<extra_id_39>",
-    "<extra_id_40>",
-    "<extra_id_41>",
-    "<extra_id_42>",
-    "<extra_id_43>",
-    "<extra_id_44>",
-    "<extra_id_45>",
-    "<extra_id_46>",
-    "<extra_id_47>",
-    "<extra_id_48>",
-    "<extra_id_49>",
-    "<extra_id_50>",
-    "<extra_id_51>",
-    "<extra_id_52>",
-    "<extra_id_53>",
-    "<extra_id_54>",
-    "<extra_id_55>",
-    "<extra_id_56>",
-    "<extra_id_57>",
-    "<extra_id_58>",
-    "<extra_id_59>",
-    "<extra_id_60>",
-    "<extra_id_61>",
-    "<extra_id_62>",
-    "<extra_id_63>",
-    "<extra_id_64>",
-    "<extra_id_65>",
-    "<extra_id_66>",
-    "<extra_id_67>",
-    "<extra_id_68>",
-    "<extra_id_69>",
-    "<extra_id_70>",
-    "<extra_id_71>",
-    "<extra_id_72>",
-    "<extra_id_73>",
-    "<extra_id_74>",
-    "<extra_id_75>",
-    "<extra_id_76>",
-    "<extra_id_77>",
-    "<extra_id_78>",
-    "<extra_id_79>",
-    "<extra_id_80>",
-    "<extra_id_81>",
-    "<extra_id_82>",
-    "<extra_id_83>",
-    "<extra_id_84>",
-    "<extra_id_85>",
-    "<extra_id_86>",
-    "<extra_id_87>",
-    "<extra_id_88>",
-    "<extra_id_89>",
-    "<extra_id_90>",
-    "<extra_id_91>",
-    "<extra_id_92>",
-    "<extra_id_93>",
-    "<extra_id_94>",
-    "<extra_id_95>",
-    "<extra_id_96>",
-    "<extra_id_97>",
-    "<extra_id_98>",
-    "<extra_id_99>"
-  ],
-  "clean_up_tokenization_spaces": true,
-  "eos_token": "</s>",
-  "extra_ids": 100,
-  "model_max_length": 512,
-  "pad_token": "<pad>",
-  "sp_model_kwargs": {},
-  "tokenizer_class": "T5Tokenizer",
-  "unk_token": "<unk>"
-}

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/saved_weights/trans_input.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:f98adc49cd4c2f647b32016a990363f67b33d273b860e75efdfda9545a44b439
-size 411248

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/train_results.json

@@ -1,9 +0,0 @@
-{
-    "epoch": 10.0,
-    "total_flos": 9122411270725632.0,
-    "train_loss": 0.5864024265556579,
-    "train_runtime": 1102.9262,
-    "train_samples": 5000,
-    "train_samples_per_second": 45.334,
-    "train_steps_per_second": 1.423
-}

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/trainer_state.json

@@ -1,105 +0,0 @@
-{
-  "best_metric": 55.9605,
-  "best_model_checkpoint": "logs_and_outputs/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/checkpoint-1500",
-  "epoch": 10.0,
-  "eval_steps": 500,
-  "global_step": 1570,
-  "is_hyper_param_search": false,
-  "is_local_process_zero": true,
-  "is_world_process_zero": true,
-  "log_history": [
-    {
-      "epoch": 3.1847133757961785,
-      "learning_rate": 0.0003,
-      "loss": 0.6612,
-      "step": 500
-    },
-    {
-      "epoch": 3.1847133757961785,
-      "eval_exact_match": 43.9737,
-      "eval_exact_match_for_CL": 43.9737,
-      "eval_exact_match_for_yelp": 43.9737,
-      "eval_gen_len": 2.5726,
-      "eval_global_step": 500,
-      "eval_loss": 0.45953086018562317,
-      "eval_rouge1": 66.2544,
-      "eval_rouge1_for_CL": 66.2544,
-      "eval_rouge1_for_yelp": 66.2544,
-      "eval_rougeL": 66.2544,
-      "eval_rougeL_for_CL": 66.2544,
-      "eval_rougeL_for_yelp": 66.2544,
-      "eval_runtime": 105.0444,
-      "eval_samples_per_second": 72.35,
-      "eval_steps_per_second": 0.286,
-      "step": 500
-    },
-    {
-      "epoch": 6.369426751592357,
-      "learning_rate": 0.0003,
-      "loss": 0.5753,
-      "step": 1000
-    },
-    {
-      "epoch": 6.369426751592357,
-      "eval_exact_match": 51.9474,
-      "eval_exact_match_for_CL": 51.9474,
-      "eval_exact_match_for_yelp": 51.9474,
-      "eval_gen_len": 2.5862,
-      "eval_global_step": 1000,
-      "eval_loss": 0.4304201900959015,
-      "eval_rouge1": 69.4868,
-      "eval_rouge1_for_CL": 69.4868,
-      "eval_rouge1_for_yelp": 69.4868,
-      "eval_rougeL": 69.4868,
-      "eval_rougeL_for_CL": 69.4868,
-      "eval_rougeL_for_yelp": 69.4868,
-      "eval_runtime": 104.5891,
-      "eval_samples_per_second": 72.665,
-      "eval_steps_per_second": 0.287,
-      "step": 1000
-    },
-    {
-      "epoch": 9.554140127388536,
-      "learning_rate": 0.0003,
-      "loss": 0.5329,
-      "step": 1500
-    },
-    {
-      "epoch": 9.554140127388536,
-      "eval_exact_match": 55.9605,
-      "eval_exact_match_for_CL": 55.9605,
-      "eval_exact_match_for_yelp": 55.9605,
-      "eval_gen_len": 2.5192,
-      "eval_global_step": 1500,
-      "eval_loss": 0.4083245098590851,
-      "eval_rouge1": 70.8684,
-      "eval_rouge1_for_CL": 70.8684,
-      "eval_rouge1_for_yelp": 70.8684,
-      "eval_rougeL": 70.8684,
-      "eval_rougeL_for_CL": 70.8684,
-      "eval_rougeL_for_yelp": 70.8684,
-      "eval_runtime": 104.4585,
-      "eval_samples_per_second": 72.756,
-      "eval_steps_per_second": 0.287,
-      "step": 1500
-    },
-    {
-      "epoch": 10.0,
-      "step": 1570,
-      "total_flos": 9122411270725632.0,
-      "train_loss": 0.5864024265556579,
-      "train_runtime": 1102.9262,
-      "train_samples_per_second": 45.334,
-      "train_steps_per_second": 1.423
-    }
-  ],
-  "logging_steps": 500,
-  "max_steps": 1570,
-  "num_input_tokens_seen": 0,
-  "num_train_epochs": 10,
-  "save_steps": 500,
-  "total_flos": 9122411270725632.0,
-  "train_batch_size": null,
-  "trial_name": null,
-  "trial_params": null
-}

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/trans_input/reg_0.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:4beab9d4dcbba908b7def8ef43eacb08dd6f3941fdf70c924622ec431085ae3a
-size 1049682

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/trans_input/reg_1.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:009305b9def4b16f2d3260b9e1a9493d5e3d9145014d8335bd9a2b4e2c6b8b45
-size 41106

root_gainlora/logs/root_t5_small/gen_script_long_order3_t5_small_gainlora_inflora/outputs/1-yelp/trans_input/reg_2.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:251823168eaa8fde1fbf577a463ca6d34f3b49abd13868096f45d9dd5544bee1
-size 1049682