If you're going to spam the system, why not just use Win32 API's and control their client like a zombie instead of trying to play the cat and mouse game reverse engineering their protocol?
You can send instructions to manipulate the client and have it do what you want almost as easily as you could if you knew their protocol. And that's without all the cat and mouse headaches.
I used to write bots for Yahoo's (and a few others) chat rooms back in the mid 90s. Nothing malicious, I was just a bored teenager so we're talking pretty dumb stuff like spamming naughty words in public chat rooms, etc. Anyhow, my point was this: the bots I wrote did just what you described, I used Win32 APIs to control the chat clients. As you said, it took only a few seconds to bang out the code and the bot could work on against multiple different chat clients with very little additional work.
The only time I've every bothered to write my own chat protocol client was for IRC in the late 90s (at which point I'd left college, was working full time so had turned my programming skills to good) and the only reason I bothered to write my own IRC client was because I was utterly fed up with the quality of Windows clients.
Back in the old days, Win32 APIs were so insecure that you could have all sorts of fun with them. I'm not sure how things stand these days though; my development these days are almost exclusively Linux and UNIX based.
I think he's asking how it got so obfuscated in the first place. Seemingly, the spammers could just pilot a standard up to date client with a bot, rather than try and figure out the protocol.
I'm sure it's more complicated than that however, like client side rate limiting or something.
I bet they would put the rate limit in the client, since they did that with the banned word list, but rate limiting would make more sense at the server. Maybe they did both.
Now you could fire up a new VM for each client or use a botnet to do your bidding. Oh, how the Internet has advanced.
You can basically read anything from controls, and trigger callbacks at will, as if you had actually clicked a button or written some text. This means that you can write "expect-like" software -- just start up the program, and have another program read input from it's text fields and issue commands to it.
I have actually done a lot of this, putting old sourceless win32 and win16 programs run in the background on virtual machines on the server and building new web-based interfaces on top of them.
Actually it ranges from simple event spoofing (user clicked here, user dragged there) to injecting a DLL + spawning a thread under your control.
Event spoofing is pretty limited. While having a thread under your control gives you full power as you have full access to the process' memory and can call any function you want.
That's what I meant to use it for but the API is used at the root of applications to draw windows, handle mouse click events, accept keyboard input, create icons in the system tray and anything else that would involve Windows UI.
In the same way applications use the win API to create their UI, others could use it to manipulate and control the interface of other programs. It's powerful.
You can send instructions to manipulate the client and have it do what you want almost as easily as you could if you knew their protocol. And that's without all the cat and mouse headaches.