The one feature which may not be obvious yet is the ability for workers to mark a job successfully completed or failed with a result and then retrieve it later. The workflow looks like this:
* Client A: Backgrounds a Job A
* Client A: Backgrounds a Job B
* Client A: Backgrounds a Job C
----
* Worker A: Picks up Job A + Completes
* Worker B: Picks up Job B + Completes
* Worker C: Picks up Job C + Completes
----
* Client A: Picks up the result for Job A,B,C.
This allows a single client to concurrently process multiple jobs within a single process and retrieve its result. This is what I like to call "Gearman mode"[0], as it was modeled after that project also. Useful in languages that do not have well defined concurrency. This is a niche use case and may not be needed by everyone, but very useful as soon as you need it. This will become more obvious when I have clients for these languages.
Lastly there are some subtle enhancements such as retry support and synchronous processing (submit and wait for result).
Thanks for the great question. This is a very popular question and I will FAQ it.
It is possible, but explicitly through workers. You can have a worker block on the "result" command (it allows for a wait-timeout) and wait for the successfully completion of Job X and enqueue Job Y.
There is no way to define the dependency automatically, the workers can create any type of workflow though, including if Job X fails.
RabbitMQ can do much of what Workq can do from a purely messaging standpoint since input and output looks about the same.
Workq is built on the higher level concept of a job so the feature set is refined around what a job is. In Workq, a job must successfully complete or fail, and optionally a result passed back to the client. A job can be retried when it has timed out or even when it has explicitly failed outright (maybe there was an temporary error with your API provider..etc). You can say: retry the job if it has timed out up to 5x BUT let it explicitly fail only once. These small refinements help streamlined the concept of processing a job fully, not just a blob of message.
Also there are some other key things such as job scheduling based on time which don't exist in RabbitMQ, but are usually offered in libraries such as DelayedJob..etc.
There is no individual progress streaming at the moment. It was on the drawing board, but was slashed for simplicity and time. Just curious, what would you use it for in your case?
A big yes to this! Signal handling will be covered in a future update. This is critical to Workq, being that it is intended to run as a standalone. A zero downtime deployment is required. There will be a more in-depth roadmap in the repo soon. Someone earlier just asked for it also.
A job in Workq doesn't directly target a system command or anything. The short answer is a job could be thought of as a function call. The name of the job is the function, and the job's payload is its parameters. This allows a worker to connect to Workq and pick up available jobs enqueued previously by a client. It is up to the worker to decipher the payload and act on it, which may be run “ping 10.10.10.10” on the worker node.
The ping example was just given so I could enqueue a job named “ping” as a client and respond with a “pong” text result from a worker. Just a silly example :). Real use cases would be background jobs such as sending emails, http downloads, image resizing…etc.
The text interface primarily has simplicity on its side. The goal was to implement a set of commands with a small footprint. HTTP would have provided "too much" for me to worry about in terms of designing the commands. However, HTTP2 was considered at one point, but it proved to be "too much" at the time.
From my own experience, the text commands are easier to test against, especially its boundaries (inputs to the server) which makes client development significantly easier.
I can see how designing the command language can be simpler and cleaner in text than say XML. HTTP offers so much tooling, I might have gone with text over HTTP or JSON with a `command` entry.
As for HTTP2, isn't that handled by the HTTP server/client implementations and is from the application code the same as HTTP?
As for the HTTP2 portion, the server details would be abstracted out especially with HTTP2 support in Go 1.6. At the time I looked, HTTP2 clients for various languages were still popping up and stabilizing (I think they still are). I didn't want that to be a factor when I was developing clients outside of Go (for example PHP). In addition, an important goal was to develop extremely small clients, where I understood exactly what was going over the wire.
If there are enough direct tooling benefits that HTTP2 can offer, it would be fun to experiment with it as an alternative interface. Funny enough, the first prototype name of the project was "httpq".
The one feature which may not be obvious yet is the ability for workers to mark a job successfully completed or failed with a result and then retrieve it later. The workflow looks like this:
* Client A: Backgrounds a Job A
* Client A: Backgrounds a Job B
* Client A: Backgrounds a Job C
----
* Worker A: Picks up Job A + Completes
* Worker B: Picks up Job B + Completes
* Worker C: Picks up Job C + Completes
----
* Client A: Picks up the result for Job A,B,C.
This allows a single client to concurrently process multiple jobs within a single process and retrieve its result. This is what I like to call "Gearman mode"[0], as it was modeled after that project also. Useful in languages that do not have well defined concurrency. This is a niche use case and may not be needed by everyone, but very useful as soon as you need it. This will become more obvious when I have clients for these languages.
Lastly there are some subtle enhancements such as retry support and synchronous processing (submit and wait for result).
Thanks for the great question. This is a very popular question and I will FAQ it.
[0] http://gearman.org