The ones you wouldn't remember probably weren't sufficiently different from the mean to warrant a positive or negative score in the first place. Meanwhile, the ones that you do remember probably were.

Just make opting out of each individual question easy (add a N/A or "don't remember" option). Sure, you'll collect less data, but I imagine that you'll get at least partial cancellation after multiplication by the SNR, possibly much more than partial cancellation.