midorin-Linux commited on
Commit
6d26e10
Β·
verified Β·
1 Parent(s): 939bd3a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +0 -29
README.md CHANGED
@@ -38,32 +38,3 @@ Traditional fine-tuning often suffers from:
38
  - **Catastrophic forgetting** when training on sequential datasets
39
  - **Imbalanced capabilities** from single-source training
40
  - **Style inconsistencies** across different task types
41
-
42
- Our multi-phase approach with strategic layer freezing, replay buffers, and EWC regularization addresses these challenges systematically.
43
-
44
- ## Architecture
45
- ```text
46
- GPT-OSS-20B Base Model
47
- β”‚
48
- β”œβ”€β”€β”€ Phase 1: Foundation Training
49
- β”‚ β”œβ”€ Data: GPT-5.2-codex-max (1000) + Claude 4.5 Opus (250) + Claude 4.5 Sonnet (250)
50
- β”‚ β”œβ”€ Layers: MLP + Attention
51
- β”‚ └─ Goal: Establish coding + reasoning foundation
52
- β”‚
53
- β”œβ”€β”€β”€ Phase 1.5: Knowledge Consolidation
54
- β”‚ β”œβ”€ Data: Mixed replay of Phase 1 data
55
- β”‚ β”œβ”€ Layers: MLP + Attention
56
- β”‚ └─ Goal: Prevent early forgetting
57
- β”‚
58
- β”œβ”€β”€β”€ Phase 2: Specialization Training
59
- β”‚ β”œβ”€ Data: Claude Sonnet (250) + GPT-5.2 high (250) + Replay (150)
60
- β”‚ β”œβ”€ Layers: MLP + Adapter
61
- β”‚ └─ Goal: Integrate balanced reasoning + maintain coding
62
- β”‚
63
- └─── Phase 2.5: Gradual Unfreezing
64
- β”œβ”€ Data: Full mixed dataset
65
- β”œβ”€ Layers: Upper Attention layers + MLP + Adapter
66
- └─ Goal: Fine-tune attention patterns if needed
67
- ```
68
-
69
-
 
38
  - **Catastrophic forgetting** when training on sequential datasets
39
  - **Imbalanced capabilities** from single-source training
40
  - **Style inconsistencies** across different task types