I wonder if the considered Heka (https://hekad.readthedocs.org/en/v0.10.0/), made by Mozilla? It's written in Go and, as far as I can tell, solves many of the same problems and more.
We use Heka in production and it has changed what would have been an ELK stack to a EHK stack with 0 regrets. The fact that the collector nodes can ship directly to ES without having to go through an intermediary node is extremely special...it reduces system complexity a lot.
Heka looks good and does a lot more. It doesn't appear to support Redis and S3 out of the box, so we would have probably had to evaluate, learn, and change the third-party plugins had we known about Heka beforehand.
I looked into heka a while back and was turned off by its seeming reliance on lua scripts for almost every feature. It just seemed overcomplicated to deploy and maintain for a Go app as a result, and I didn't understand why all that functionality didn't come built in.
If you want to write your own plugins, you can do them in lua or Go. Lua is the recommended way for most types - I think matching is supposed to be faster, and also then you don't need to recompile the heka binary.
If you don't want to write your own plugins, you'll never interact with lua.
Don't you need to ship all those builtin lua plugins around with your deployment of heka though? That was what turned me off - most of the other options can be used with a binary + config file.
Yeah, I like heka a lot in my limited experience with it. Easy to setup, TOML config is pretty nice and forwarding filesystem logs is fast. Interestingly, parsing the logs can be done with custom defined Lua in addition to static methods like regex.
