For evidence of the risk please see my reply below http://news.ycombinator.com/item?id=1839044 . I believe SIDS is not yet very well understood, but there's also risk of asphyxiation.

Also if you sleep fine, good for you. I wouldn't and couldn't sleep well knowing there's a risk (although small) of hurting a baby.