Agreed, I keep hearing it's difficult yet user om2 on the webkit team says they were able to come up with multiple attacks internally in the webkit team once they'd heard about the trick [1].
Safari/webkit have since rolled out mitigations to prevent the attacks that they figured out but it puts the lie to the idea that Spectre is only a theoretical attack that we've yet to see an exploit for.
From what I've seen. There's been demonstrated attacks using Javascript in Chrome to dump the saved passwords from the browser using these bugs
If an attack is that easy to pull off, I don't think it's reasonable to make it an "opt in"