I also didn't have great experience of team competition as a kid. I've since learnt that kids can be introduced to sport far more inclusively and positively, and now enjoy coaching my kids sports teams. But my own activities remain largely solo, uncompetitive and somewhat introspective - there are plenty of very physical activities that fit this bill, and you don't have to go anywhere near a gymn (which I personally don't like at all). Think surfing, skating, climbing, cycling, hiking, kayaking... You can get a great workout without "working out".