Java applets had a lot working against them from a technical and UX perspective: they were insecure, and popped up quite scary warning notices about security that had to be clicked through in order to run the applets. Java was also poorly optimized for consumer-facing applications: JIT startup and warmup time was sufficiently long that they felt more sluggish than HTML/CSS/JS, despite the fact that in the long run the JVM outperforms JS by a significant margin.
In an aside, the JVM has only relatively recently been an effective general-purpose compile target, and it wasn't designed as such. The JVM's failure for frontends doesn't give much indication as to where the pitfalls of targets like PNACL or asm.js might be: it died (deservedly) because of reasons that apply to neither technology — slow startup time and lack of effective sandboxing. Both PNACL and asm.js start quickly, lean on hardware-acceleration in the browser to react smoothly, are designed from the start as compile targets, and are sandboxed to the point where browser vendors don't feel the need to pop up warning boxes that scare off consumers.
In an aside, the JVM has only relatively recently been an effective general-purpose compile target, and it wasn't designed as such. The JVM's failure for frontends doesn't give much indication as to where the pitfalls of targets like PNACL or asm.js might be: it died (deservedly) because of reasons that apply to neither technology — slow startup time and lack of effective sandboxing. Both PNACL and asm.js start quickly, lean on hardware-acceleration in the browser to react smoothly, are designed from the start as compile targets, and are sandboxed to the point where browser vendors don't feel the need to pop up warning boxes that scare off consumers.