Page speed is still a ranking factor. Optimizing pages and reducing bloat does have a positive effect in all SERPs. By my understanding though, AMP is more specific to mobile. A light page may or may not render well on mobile, but AMP pages almost always will.

I'm still undecided on if I like AMP or not, but I do see the reason for the distinction.