Please do get started. This sound like an awesome story! (For HN, that is, maybe not so much for dinner parties).
Just thinking about the thought process you had is entertaining in my end, because my procrastinating self would have delayed the part about going into the ceiling as the last possible resort... So what evidence was strong enough that you figured there must have been something wrong with a particular cable, or all cables, and just a specific section?
That particular section of ethernet, which served about half of a floor of an entire building was intermittent. It would work the vast majority of the time, but would have mysterious periodic outages, depending on the phase of the moon, or other mysterious cosmic events.
Working at a university, no one would pay for me to have a time domain reflectometer (aka TDR), which would have helped determine if there was a bad spot in the Ethernet cable somewhere.
One of the assistant directors of the lab told me that I didn't need a TDR, because I could just diagnose the problem with a signal generator and an oscilloscope. (Which is what a TDR basically is, but just bundled up to be convenient to use.) I ended up doing just that, and was able to determine that there was a reflection happening on that section of Ethernet cable, but IIRC, narrowing down the location relied on me having more knowledge than I had about the speed of light in an RG-58 cable. And also, wheeling around a working with the oscilloscope was a PITA.
Eventually I found someone to loan me a real TDR, and was able to determine how far down the cable the problem was occurring. Of course, even with that knowledge, determining where the problem was occurring was a challenge, since the cable snaked in a out of everyone's offices.
I followed the cable, applying the TDR at various points, until I got close to where the reflection was occurring, and it seemed to be occurring where the cable ran through the ceiling for a while.
I should note that all while I'm doing this, people are griping heavily, since it required disconnecting that section of Ethernet, meaning that people couldn't get their work done.
In any case, I got a ladder, pried up some ceiling tiles, looked up into the ceiling, and found a section of the Ethernet cable that had been spliced. At first I figured that one of the splices was bad, but eventually I noticed that the cable that had been spliced in was RG-59.
In case you don't already know, RG-58 and RG-59 look almost identical to each other. IIRC, the only real way to tell the difference was by reading the print on the cable.
Whoever spliced in that piece of cable should be drawn and quartered, but once I replaced that bit of cable with RG-58, everything then worked fine from then on, with no more intermittent outages.
Just thinking about the thought process you had is entertaining in my end, because my procrastinating self would have delayed the part about going into the ceiling as the last possible resort... So what evidence was strong enough that you figured there must have been something wrong with a particular cable, or all cables, and just a specific section?