How many t/s would you expect? I think I feel perfectly fine when its over 50. A... | Hacker News

Hacker News new | past | comments | ask | show | jobs | submit

login

mrtksn 64 days ago | parent | context | favorite | on: Apple M3 Ultra

How many t/s would you expect? I think I feel perfectly fine when its over 50.

Also, people figured a way to run these things in parallel easily. The device is pretty small, I think for someone who wouldn't mind the price tag stacking 2-3 of those wouldn't be that bad.

yk 64 days ago | [–]

I think I've seen 800 GB/s memory bandwidth, so a q4 quant of a 400 B model should be 4 t/s if memory bound.

behnamoh 64 days ago | [–]

I know you’re referring to the exolabs app, but the t/s is really not that good. it uses thunderbolt instead of NVlink.

Consider applying for YC's Summer 2025 batch! Applications are open till May 13
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact