If your website is security or cryptography related, then you'll need to cater to your demographic. These are the overly cautious types who use NoScript and disable javascript everywhere.

Otherwise, I wouldn't worry about it.