dexmac commited on
Commit
b0bd07d
Β·
verified Β·
1 Parent(s): 57ea641

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +48 -30
README.md CHANGED
@@ -5,6 +5,7 @@ library_name: peft
5
  base_model: Qwen/Qwen2.5-1.5B
6
  tags:
7
  - lora
 
8
  - baseline
9
  - flat-training
10
  - math
@@ -15,57 +16,74 @@ datasets:
15
  pipeline_tag: text-generation
16
  ---
17
 
18
- # Flat-LoRA Baseline (Qwen 2.5 1.5B) β€” Control Group
19
 
20
- **Baseline (control group) for the Progressive Cognitive Architecture experiment.**
21
 
22
- ## What is this?
23
 
24
- This is a standard LoRA adapter trained on the **same 6,000 math examples** as the Progressive model, but:
25
- - ❌ No 4-phase curriculum (all data mixed together)
26
- - ❌ No Dream Pruning
27
- - ❌ No progressive complexity
 
28
 
29
- This serves as the **control group** to demonstrate that the improvements come from the cognitive architecture, not simply from LoRA fine-tuning.
30
 
31
- ## πŸ“Š Results
32
 
33
- | Metric | Flat-LoRA (this) | Dream-LoRA | Base |
34
- |--------|-----------------|-----------|------|
35
- | Exact Accuracy | **60.6%** Β± 3.8 | 58.6% Β± 2.9 | 18.2% |
36
- | Number Sense | **0.0%** ❌ | 60.0% | 57.0% |
37
- | Metacognition | **0.0%** ❌ | 100.0% | 84.9% |
 
 
38
 
39
- **The Paradox of Accuracy**: Flat-LoRA achieves the highest raw accuracy but completely destroys the model's number sense and ability to delegate. It's an "idiot savant" β€” good at one thing, bad at everything else.
40
-
41
- ## πŸš€ Quick Start
42
 
43
  ```python
44
  from transformers import AutoModelForCausalLM, AutoTokenizer
45
  from peft import PeftModel
46
 
47
- base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B", device_map="auto")
 
 
48
  tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B")
49
 
50
- model = PeftModel.from_pretrained(base_model, "dexmac/progressive-cognitive-baseline-lora")
 
 
 
51
 
52
- inputs = tokenizer("Calculate: 347 + 891 =", return_tensors="pt").to(model.device)
53
- outputs = model.generate(**inputs, max_new_tokens=20)
 
 
54
  print(tokenizer.decode(outputs[0], skip_special_tokens=True))
55
  ```
56
 
57
- ## βš™οΈ Training Details
 
 
 
 
 
58
 
59
- - **Base model**: Qwen/Qwen2.5-1.5B
60
- - **LoRA config**: rank=16, alpha=32, targets=q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
61
- - **Training**: 3 epochs, 6,000 mixed samples, lr=1e-4
62
- - **Hardware**: NVIDIA T4
63
 
64
- ## πŸ“„ Related
 
 
 
 
 
 
 
 
65
 
66
- - **Progressive model (Dream-LoRA)**: [dexmac/progressive-cognitive-dream-lora](https://huggingface.co/dexmac/progressive-cognitive-dream-lora)
67
- - **GitHub**: [dexmac221/progressive-cognitive](https://github.com/dexmac221/progressive-cognitive)
68
 
69
- ## πŸ“œ License
70
 
71
  Apache 2.0
 
 
5
  base_model: Qwen/Qwen2.5-1.5B
6
  tags:
7
  - lora
8
+ - peft
9
  - baseline
10
  - flat-training
11
  - math
 
16
  pipeline_tag: text-generation
17
  ---
18
 
19
+ # Architettura Cognitiva Progressiva β€” Flat-LoRA (Italiano, Controllo)
20
 
21
+ **Modello di controllo** β€” Qwen2.5-1.5B addestrato con tutti i dati in un singolo passaggio (senza fasi, senza pruning). Serve come baseline per valutare l'addestramento progressivo.
22
 
23
+ ## πŸ“Š Risultati
24
 
25
+ | Metrica | Flat-LoRA (questo) | Dream-LoRA | Progressive-LoRA |
26
+ |---------|-------------------|-----------|-----------------|
27
+ | Accuratezza Esatta | **60.6%** | 58.6% Β± 2.9 | 37.0% Β± 0.5 |
28
+ | Number Sense | 0.0% | **60.0% Β± 0.8** | **57.7% Β± 0.5** |
29
+ | Metacognizione | 0.0% | **100.0%** | **98.5%** |
30
 
31
+ Il flat training raggiunge la migliore accuratezza esatta ma **non sviluppa nΓ© number sense nΓ© metacognizione** β€” dimostrando che queste sono capacitΓ  emergenti dell'architettura progressiva, non del dato.
32
 
33
+ ## πŸ”§ Configurazione
34
 
35
+ | Parametro | Valore |
36
+ |-----------|--------|
37
+ | Modello Base | Qwen/Qwen2.5-1.5B |
38
+ | LoRA Rank | 16 |
39
+ | LoRA Alpha | 32 |
40
+ | Target LoRA | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
41
+ | Lingua Dati | Italiano |
42
 
43
+ ## πŸš€ Uso Rapido
 
 
44
 
45
  ```python
46
  from transformers import AutoModelForCausalLM, AutoTokenizer
47
  from peft import PeftModel
48
 
49
+ base_model = AutoModelForCausalLM.from_pretrained(
50
+ "Qwen/Qwen2.5-1.5B", device_map="auto", torch_dtype="auto"
51
+ )
52
  tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B")
53
 
54
+ model = PeftModel.from_pretrained(
55
+ base_model,
56
+ "dexmac/progressive-cognitive-baseline-lora"
57
+ )
58
 
59
+ messages = [{"role": "user", "content": "Calcola: 342 * 67"}]
60
+ text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
61
+ inputs = tokenizer(text, return_tensors="pt").to(model.device)
62
+ outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.1)
63
  print(tokenizer.decode(outputs[0], skip_special_tokens=True))
64
  ```
65
 
66
+ ## πŸ”— Modelli Correlati
67
+
68
+ - [**Dream-LoRA (IT)**](https://huggingface.co/dexmac/progressive-cognitive-dream-lora) β€” Addestramento progressivo + Dream Pruning
69
+ - [Progressive-LoRA (IT)](https://huggingface.co/dexmac/progressive-cognitive-lora) β€” Primo prototipo con magnitude pruning
70
+ - [1.5B Flat (EN)](https://huggingface.co/dexmac/progressive-cognitive-baseline-lora-en) β€” Equivalente inglese
71
+ - [GitHub](https://github.com/dexmac221/progressive-cognitive) β€” Codice sorgente completo
72
 
73
+ ## πŸ“ Citation
 
 
 
74
 
75
+ ```bibtex
76
+ @software{progressive_cognitive_2026,
77
+ author = {Dex Mac},
78
+ title = {Progressive Cognitive Architecture for LLMs},
79
+ year = {2026},
80
+ url = {https://github.com/dexmac221/progressive-cognitive},
81
+ version = {1.0.0}
82
+ }
83
+ ```
84
 
 
 
85
 
86
+ ## πŸ“„ License
87
 
88
  Apache 2.0
89
+