There can be different kinds of downsides. Not getting union pension plan is one kind of downside, not being able to get hired because union contract mandates preference to union members - another. Which kind are we dealing with here? If the second one then membership is practically a requirement to non-marginal employment, unless you're a superstar that can choose their own projects and doesn't need any security or continued employment.