Does the irc bot you imported use websockets? I've been researching how to do a Twitch-plays game in Unity/C# and am thinking websockets to avoid rate limiting, but was wondering how much that'd do for latency, given it's push-based instead of polling.
You can take a look at the code on Github, it's a very straight forward Python IRC client.
The delay experienced is all in the stream broadcast back to the Twitch servers and on to the viewers.
When I'm watching the chat and also watching the local view of Doom the commands are executed almost instantly. It then takes ~8s to show on my Twitch view. When I started it was around 15s but some tweaking in OBS has brought that down a bit.