LOOM
Making a 30B local model practical on an 8 GB M1 Mac.
How usable can a 30B-class local model become on an 8 GB M1 Mac if the surrounding runtime, storage and expert-access path are engineered aggressively?
- Model
- 30B model
- Unified memory
- 8 GB unified memory
- Decode
- 7.557 tok/s
- 1,155-token cold prefill
- 11.306 tok/s