How much ram needed
#1
by satvikahuja - opened
will it work on 32gb m1 max?
@satvikahuja
Unified memory usage
- 8B 4bit : 7.5GB
- 8B 8bit : 10GB
- 70B 4bit : 39GB
- 70B 8bit : 70GB
So, it will cause swapping with 32gb😱
Here's an easy tool to calculate the ballpark memory needed: https://llm-calc.rayfernando.ai
Sadly, it looks like you'll need a hardware update (just like I do)