So, the TLDR is that the author wasn't getting documented data from APNS, and this was verified by digging through GDB and OpenSSL.
While digging through everything is fun and worthwhile, this could have been checked much quicker with wireshark. tcpdump -w packets_from_apns.pcap -n -s 0 -i whatever src port 2195; wait for the problem to happen, check it out in wireshark -- if the circumstances are as the author describes, APNS will only send a TLS Alert packet, and not any TLS Application Data packets, so it should be obvious they're not sending any data.
Also, re "I'm now pretty familiar with the OpenSSL source (it gets a bad rap, IMO C is more at fault)": I disagree, since I'm more familiar with the OpenSSL source than I'd like.
Many things are way more obtuse than they should be: if you want to support TLS 1.0 and up, you need to disable SSL 2 and 3, and call SSLv23_server..; if you call TLSv1_server..., you get TLS 1.0 only (this seems to have been changed recently, there's now a TLS_server... , but a neutral name should have been chosen, and earlier)
Also, if you want to support an extension OpenSSL doesn't, the only way to do it is to register for a debug callback.
When the author first stated that the socket was readable but they were getting no bytes, I actually thought this might have to do with BEAST. If I recall correctly, at least some implementations of SSL work around BEAST by sending an empty packet prior to the real packet (the empty packet randomizes the IV, which breaks BEAST). And depending on how read is implemented, it might see the empty packet and always return zero bytes (instead of continuing onto the next packet which hopefully was received already). Incidentally, other implementations don't do the empty packet thing, instead they send a packet with just the first byte of the message, followed by a packet with the rest.
Of course, in the end it turned out that the author was querying if the OS buffer had bytes in it, as opposed to querying if there was any application data available. But anyone else doing something similar may want to be aware of this.
While digging through everything is fun and worthwhile, this could have been checked much quicker with wireshark. tcpdump -w packets_from_apns.pcap -n -s 0 -i whatever src port 2195; wait for the problem to happen, check it out in wireshark -- if the circumstances are as the author describes, APNS will only send a TLS Alert packet, and not any TLS Application Data packets, so it should be obvious they're not sending any data.
Also, re "I'm now pretty familiar with the OpenSSL source (it gets a bad rap, IMO C is more at fault)": I disagree, since I'm more familiar with the OpenSSL source than I'd like.
Many things are way more obtuse than they should be: if you want to support TLS 1.0 and up, you need to disable SSL 2 and 3, and call SSLv23_server..; if you call TLSv1_server..., you get TLS 1.0 only (this seems to have been changed recently, there's now a TLS_server... , but a neutral name should have been chosen, and earlier)
Also, if you want to support an extension OpenSSL doesn't, the only way to do it is to register for a debug callback.