If Q4 takes less than 1.3x as many tokens as bf16 or q8, it could still end up being faster, given how decode tends to be bandwidth bound.
The kv cache was still bf16, so a few ops are the same between quants.
I've been using onshape for 3d printing and it never occurred to me that this kind of thing was a possibility. My mind is being blown, thank you for this comment!
Would love to see video of the application in action. Was recently researching eink displays for motorcycle navigation, but the refresh rates were not up to par for realtime automotive navigation..
The photograph does have its own expiration timeline, which is not printed on the card. So yes, it was a little surprising to receive that letter, calling me into the office for another photo shoot.
This remarkable event, along with the an accessible history of the evolution of precision manufacturing is covered in Simon Winchester's excellent book The Perfectionists[0]. The audiobook is narrated by the author, which is quite enjoyable as well.
[0]https://www.youtube.com/watch?v=8qrriKcwvlY
FTFY
reply