Centralized logging, a la Splunk is the gold (with what they charge, probably platinum) standard here. Watching someone who knows what they're doing at a Splunk console for a distributed system they understand is a thing of beauty. It's just a shame they charge so much.