I have a few Verbatim "Tuff and Tiny" USB drives. Like this but without the plastic clip part. I can fit them in my wallet because its about the thickness of 2 credit cards which are also in my wallet.
Reminds me of the credit card sized (literally [1]) USB stick I still have somewhere but it was too annoying to carry around and hope that next time that cheap stick still works...
Using the phone directly still seems the cleanest and most reliable way. Or maybe a combination of both, like those magnetic plugs [2] but with an integrated USB stick. Bonus points if you don't have to take it out at all (until needed) by either connecting the other magnetic part for data transfer and charging or data through USB OTG and wireless charging. One can dream... but the technology will shrink even further so who knows.
USB sticks attached to keychains are already widespread in some communities (DJs for example), I'm sure us software people could do it too if we wanted to :)