It annoys me when I have gone to the trouble to hear about a specific topic only to have the hosts chat for the first 8 minutes about their personal lives or about stuff that is completely off topic. Focus would be appreciated.
For me it's the same. I don't have much time to listen to podcasts so general chat or low density talk just isn't efficient.
I like some conversational podcasts like After On where they tend to stick to discussing the topic at hand but I couldn't get into Stuff You Should Know or Joe Rogan as there was a lot of general chat in there or not high enough information density.
I don't want to hear 5 minutes of why somebody thinks something is worth doing a podcast on: just tell me the interesting stuff and I'll judge for myself.