dexmac commited on
Commit
d30506f
Β·
verified Β·
1 Parent(s): fda5b0a

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +58 -21
README.md CHANGED
@@ -5,6 +5,7 @@ library_name: peft
5
  base_model: Qwen/Qwen2.5-1.5B
6
  tags:
7
  - lora
 
8
  - cognitive-architecture
9
  - progressive-learning
10
  - magnitude-pruning
@@ -15,46 +16,82 @@ datasets:
15
  pipeline_tag: text-generation
16
  ---
17
 
18
- # Progressive Cognitive Architecture β€” Progressive-LoRA (Qwen 2.5 1.5B)
19
 
20
- **4-phase progressive training with magnitude pruning (the predecessor to Dream Pruning).**
21
 
22
- ## What is this?
23
 
24
- This is the original Progressive-LoRA model using **magnitude pruning** (zeroing small weights) instead of SVD Dream Pruning. It was the first version of the progressive cognitive architecture.
 
 
 
 
25
 
26
- ## πŸ“Š Results
27
 
28
- | Metric | Progressive-LoRA (this) | Dream-LoRA | Flat-LoRA |
29
- |--------|------------------------|-----------|-----------|
30
- | Exact Accuracy | 37.0% Β± 0.5 | 58.6% Β± 2.9 | 60.6% |
31
- | Number Sense | 57.7% Β± 0.5 | 60.0% Β± 0.8 | 0.0% |
32
- | Metacognition | 98.5% | 100.0% | 0.0% |
33
 
34
- Dream Pruning (SVD) significantly improved upon magnitude pruning by preserving more of the learned information during compression.
35
 
36
- ## πŸš€ Quick Start
 
 
 
 
 
 
 
 
 
37
 
38
  ```python
39
  from transformers import AutoModelForCausalLM, AutoTokenizer
40
  from peft import PeftModel
41
 
42
- base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B", device_map="auto")
 
 
43
  tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B")
44
 
45
- # Note: adapters are in lora_adapters/ subfolder
46
- model = PeftModel.from_pretrained(base_model, "dexmac/progressive-cognitive-lora", subfolder="lora_adapters")
 
 
 
47
 
48
- inputs = tokenizer("Calcola: 347 + 891 =", return_tensors="pt").to(model.device)
49
- outputs = model.generate(**inputs, max_new_tokens=20)
 
 
50
  print(tokenizer.decode(outputs[0], skip_special_tokens=True))
51
  ```
52
 
53
- ## πŸ“„ Related
 
 
 
 
 
54
 
55
- - **Dream-LoRA (improved)**: [dexmac/progressive-cognitive-dream-lora](https://huggingface.co/dexmac/progressive-cognitive-dream-lora)
56
- - **GitHub**: [dexmac221/progressive-cognitive](https://github.com/dexmac221/progressive-cognitive)
 
 
 
 
 
 
 
 
 
57
 
58
- ## πŸ“œ License
 
59
 
60
  Apache 2.0
 
 
5
  base_model: Qwen/Qwen2.5-1.5B
6
  tags:
7
  - lora
8
+ - peft
9
  - cognitive-architecture
10
  - progressive-learning
11
  - magnitude-pruning
 
16
  pipeline_tag: text-generation
17
  ---
18
 
19
+ # Architettura Cognitiva Progressiva β€” Progressive-LoRA con Magnitude Pruning (Italiano)
20
 
21
+ **Primo prototipo** β€” Qwen2.5-1.5B addestrato con architettura cognitiva progressiva a 4 fasi, usando **magnitude pruning** (azzeramento pesi piccoli). Successivamente sostituito da SVD Dream Pruning.
22
 
23
+ ## πŸ“Š Risultati
24
 
25
+ | Metrica | Progressive-LoRA (questo) | Dream-LoRA | Flat-LoRA |
26
+ |---------|--------------------------|-----------|-----------|
27
+ | Accuratezza Esatta | 37.0% Β± 0.5 | 58.6% Β± 2.9 | 60.6% |
28
+ | Number Sense | 57.7% Β± 0.5 | 60.0% Β± 0.8 | 0.0% |
29
+ | Metacognizione | 98.5% | 100.0% | 0.0% |
30
 
31
+ ## 🧠 Architettura
32
 
33
+ Training progressivo a 4 fasi su dati aritmetici italiani:
34
+ 1. **Fondamenta** β€” Aritmetica esatta
35
+ 2. **Consolidamento** β€” Magnitude pruning + fine-tuning su approssimazioni
36
+ 3. **Delega** β€” Routing complessitΓ : calcolo interno vs. strumento
37
+ 4. **Orchestrazione** β€” Pipeline completa: intuizione β†’ routing β†’ tool β†’ validazione
38
 
39
+ ## πŸ”§ Configurazione
40
 
41
+ | Parametro | Valore |
42
+ |-----------|--------|
43
+ | Modello Base | Qwen/Qwen2.5-1.5B |
44
+ | LoRA Rank | 16 |
45
+ | LoRA Alpha | 32 |
46
+ | Target LoRA | q_proj, k_proj, v_proj, o_proj |
47
+ | Tipo Pruning | Magnitude (azzeramento pesi piccoli) |
48
+ | Lingua Dati | Italiano |
49
+
50
+ ## πŸš€ Uso Rapido
51
 
52
  ```python
53
  from transformers import AutoModelForCausalLM, AutoTokenizer
54
  from peft import PeftModel
55
 
56
+ base_model = AutoModelForCausalLM.from_pretrained(
57
+ "Qwen/Qwen2.5-1.5B", device_map="auto", torch_dtype="auto"
58
+ )
59
  tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B")
60
 
61
+ model = PeftModel.from_pretrained(
62
+ base_model,
63
+ "dexmac/progressive-cognitive-lora",
64
+ subfolder="lora_adapters"
65
+ )
66
 
67
+ messages = [{"role": "user", "content": "Calcola: 342 * 67"}]
68
+ text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
69
+ inputs = tokenizer(text, return_tensors="pt").to(model.device)
70
+ outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.1)
71
  print(tokenizer.decode(outputs[0], skip_special_tokens=True))
72
  ```
73
 
74
+ ## πŸ”— Modelli Correlati
75
+
76
+ - [**Dream-LoRA (IT)**](https://huggingface.co/dexmac/progressive-cognitive-dream-lora) β€” Versione migliorata con SVD Dream Pruning
77
+ - [Flat-LoRA (IT)](https://huggingface.co/dexmac/progressive-cognitive-baseline-lora) β€” Controllo senza fasi
78
+ - [1.5B Dream (EN)](https://huggingface.co/dexmac/progressive-cognitive-dream-lora-en) β€” Miglior modello (inglese)
79
+ - [GitHub](https://github.com/dexmac221/progressive-cognitive) β€” Codice sorgente completo
80
 
81
+ ## πŸ“ Citation
82
+
83
+ ```bibtex
84
+ @software{progressive_cognitive_2026,
85
+ author = {Dex Mac},
86
+ title = {Progressive Cognitive Architecture for LLMs},
87
+ year = {2026},
88
+ url = {https://github.com/dexmac221/progressive-cognitive},
89
+ version = {1.0.0}
90
+ }
91
+ ```
92
 
93
+
94
+ ## πŸ“„ License
95
 
96
  Apache 2.0
97
+