Minimum RAM required to run this model
#1
by
Arete7
- opened
What is the minimum RAM required to run this model?
What is the minimum RAM required to run this model?
I am curious on this as well. Wondering if it will run on 128gb Mac Studio
Here's the peak memory you can expect at different context lengths
| Context | Prompt TPS | Gen TPS | Peak Memory |
|---|---|---|---|
| 1k | 423.3 | 52.0 | 111.9 GB |
| 2k | 586.7 | 50.2 | 112.5 GB |
| 4k | 622.8 | 47.9 | 112.9 GB |
| 8k | 596.4 | 45.3 | 113.5 GB |
| 16k | 506.1 | 39.0 | 114.9 GB |
| 32k | 374.0 | 31.8 | 117.7 GB |
| 64k | 239.5 | 23.0 | 123.9 GB |
| 128k | 133.8 | 15.7 | 136.0 GB |