Have you never had an interesting conversation and then looked up, realising that 4 hours had passed?
In my experience, I find it easier to focus with someone next to me than by myself. To the point that I take half as much Ritalin on pairing days than when I work alone.
Maybe you're right? It has to be the right person/people. Also, combine pairing with an open office and now you have two people with double the chances of being interrupted. So it's circumstantial.