dexmac commited on
Commit
fda5b0a
·
verified ·
1 Parent(s): ec4fec4

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +60 -0
README.md ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: it
3
+ license: apache-2.0
4
+ library_name: peft
5
+ base_model: Qwen/Qwen2.5-1.5B
6
+ tags:
7
+ - lora
8
+ - cognitive-architecture
9
+ - progressive-learning
10
+ - magnitude-pruning
11
+ - math
12
+ - arithmetic
13
+ datasets:
14
+ - custom
15
+ pipeline_tag: text-generation
16
+ ---
17
+
18
+ # Progressive Cognitive Architecture — Progressive-LoRA (Qwen 2.5 1.5B)
19
+
20
+ **4-phase progressive training with magnitude pruning (the predecessor to Dream Pruning).**
21
+
22
+ ## What is this?
23
+
24
+ This is the original Progressive-LoRA model using **magnitude pruning** (zeroing small weights) instead of SVD Dream Pruning. It was the first version of the progressive cognitive architecture.
25
+
26
+ ## 📊 Results
27
+
28
+ | Metric | Progressive-LoRA (this) | Dream-LoRA | Flat-LoRA |
29
+ |--------|------------------------|-----------|-----------|
30
+ | Exact Accuracy | 37.0% ± 0.5 | 58.6% ± 2.9 | 60.6% |
31
+ | Number Sense | 57.7% ± 0.5 | 60.0% ± 0.8 | 0.0% |
32
+ | Metacognition | 98.5% | 100.0% | 0.0% |
33
+
34
+ Dream Pruning (SVD) significantly improved upon magnitude pruning by preserving more of the learned information during compression.
35
+
36
+ ## 🚀 Quick Start
37
+
38
+ ```python
39
+ from transformers import AutoModelForCausalLM, AutoTokenizer
40
+ from peft import PeftModel
41
+
42
+ base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B", device_map="auto")
43
+ tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B")
44
+
45
+ # Note: adapters are in lora_adapters/ subfolder
46
+ model = PeftModel.from_pretrained(base_model, "dexmac/progressive-cognitive-lora", subfolder="lora_adapters")
47
+
48
+ inputs = tokenizer("Calcola: 347 + 891 =", return_tensors="pt").to(model.device)
49
+ outputs = model.generate(**inputs, max_new_tokens=20)
50
+ print(tokenizer.decode(outputs[0], skip_special_tokens=True))
51
+ ```
52
+
53
+ ## 📄 Related
54
+
55
+ - **Dream-LoRA (improved)**: [dexmac/progressive-cognitive-dream-lora](https://huggingface.co/dexmac/progressive-cognitive-dream-lora)
56
+ - **GitHub**: [dexmac221/progressive-cognitive](https://github.com/dexmac221/progressive-cognitive)
57
+
58
+ ## 📜 License
59
+
60
+ Apache 2.0