No, you'll sleep fine and there's no more risk of SIDS in co-sleeping babies than those that sleep in cribs and some serious research to suggest that co-sleepers are actually safer.
For evidence of the risk please see my reply below http://news.ycombinator.com/item?id=1839044 . I believe SIDS is not yet very well understood, but there's also risk of asphyxiation.
Also if you sleep fine, good for you. I wouldn't and couldn't sleep well knowing there's a risk (although small) of hurting a baby.
