We use NSQ at Dell for the commercial side of dell.com. We've been in Production with it for about 2 years.
> what are the typical use cases for NSQ ?
In the abstract, anything which can tolerate near real-time, at-least-once delivery, and does not need order guarantees. It also features retries and manual requeuing. It's typical to think order and exactly-once semantics are important because that's how we tend to think when we write code and work with (most) databases, and having order allows you to make more assumptions and simplify your approach. It typically comes at the cost of coordination or a bounded window of guarantees. Depending on your workload or how you frame the problem you may find order and exactly-once semantics are not that important, or it can be made unimportant (for example, making messages idempotent). In other cases order is important and it's worth the tradeoff; our Data Science team uses Kafka for these cases, but I'm not familiar with the details.
Here are some concrete examples of things we built using NSQ, roughly in the order they were deployed to PROD:
- Batch jobs which query services and databases to transform and store denormalized data. We process tens of millions of messages in a relatively short amount of time overnight. The queue is never the bottleneck; it's either our own code, services, or reading/writing to the database. Retries are surprisingly useful in this scenario.
- Eventing from other applications to notify a pinpoint refresh is needed for some data into the denormalized store (for example, a user updated a setting in their store, which causes a JSON model to update).
- Purchase order message queue, both for the purpose of retry and simulating what would happen if a customer on a legacy version of the backend was migrated to the new backend; also verifying a set of known 'good' orders continue to be good as business logic evolves (regression testing).
- Async invoice/email generation. This is a case where you have to be careful of at-least-once delivery and need to use a correlation ID and persistence layer to define a 'point of no return' (can't process the message again beyond this point even if it fails). We don't want to email (or bill) customers twice.
- Build system for distributing requests to our build farm.
- Pre-fetching data and hydrating a cache when a user logs in or browses certain pages, anticipating the likely next page to avoid having the user wait on these pages for an expensive service call. The client in this case is another decoupled web application; the application emitting the event is completely separate and likely on a different deployment schedule from the emitting application. The event emitted tells us what the user did, and it's the consumer's responsibility to determine what to do. This is an interesting case where we use #ephemeral channels, which disappear when the last client disconnects. We append the application's version to the channel name so multiple running versions in the same environment will each get their own copy of the message, and process it according to that binary's logic. This is useful for blue/green/canary testing and also when we're mid-deployment and have different versions running in PROD, one customer facing and one internal still being tested. I think I refer to this image more than any other when explaining NSQ's topics and channels: https://f.cloud.github.com/assets/187441/1700696/f1434dc8-60... (from http://nsq.io/overview/design.html).
Operationally, NSQ has been not just a pleasure to work with but inspirational to how we develop our own systems. Being operator friendly cannot be overrated.
This article is a complete strawman. His description of a supposed "REST"ful API is the least RESTful API I've seen in a while:
> The what-we-actually-indended-to-use request method embedded in the request payload, e.g. DELETE
Don't do this.
> The what-we-actually-indended-to-use response code embedded in the response payload, e.g. 206 Partial content.
Don't do this!
> If you’ve ever worked with a RESTful API, you know they are almost impossible to debug.
You're begging the question.
There exist plenty of APIs that abuse the HTTP methods and status codes, which I feel like is really the core argument being made. But completely ignoring it and what its purpose is is throwing the baby out with the bathwater. Read and understand the RFCs for HTTP, for a start; unfortunately, I'd wager that far too many devs of ostensibly RESTful APIs do not do this, and it shows when you get a response with a status code that makes zero sense. The vast majority of HTTP APIs I've interacted with violate both the semantic meaning of the methods and that of the status codes. (GitHub's is about the best I've ever seen.) A "RESTful HTTP API" is an API that uses the mechanisms in HTTP to accomplish the ideas of REST; (i.e., I'm not trying to equate HTTP and REST, I just think abuses of HTTP are a major impediment to understanding REST. Using HTTP well will naturally help you accomplish REST.)
> can anyone out there explained to me what 417 Expectation failed really means?
You've not read and understood the RFCs for HTTP. 417 Expectation Failed is obvious if you have; an expectation (on the request) failed (cannot be met by the server). An "expectation" is denoted on the request through use of the "Expect" header. The only existing expectation is 100-continue. Even if you do not know this by heart (and I don't expect that), it's readily findable:
1. Google "http rfc status codes"; unfortunately it's the second result; Google doesn't understand that the second result is an updated version of the first. Regardless, if you go for the first result, it points you to the second (-ish, b/c the RFC was split into multiple).
2. You select "417 Expectation Failed" in the Table of Contents.
3. You read the extremely straight-forward explanation. If you don't understand what the Expect header is for, the RFC links you to it.
So what are the ideas of REST? Start at its (de-)acronym: "Representational state transfer". That is, transfer of a resource. A "resource" is just an "object" or a concept, a thing — the actual concrete thing represented by a resource is going to be determined by your domain specific problem. E.g., "a user's profile data", "a message in a thread", "a news article" are all "resources" in that they embody some concept or idea that we want to communicate the underlying state of. You also need a standard, or uniform method of uniquely identifying, or locating these resources, which is what a URL is for (you then see why it's lit. uniform resource locator). So we build URLs to stand in as names for resources.
In order to transfer the state of a resource, embodied at a URL, you need to send it across a wire. You need to serialize it into some representation that's going to get transferred. That's what HTTP is supposed to help you do.
If you were writing a RESTful API, embedding things like the status of the operation in the response body should feel wrong, because the status isn't conceptually a part of the resource you were trying to operate on in the first place; go back to our example of "a post in a thread" — whats a status got to do with that? While technically, yes, HTTP's entity body is capable of transferring arbitrary binary data, the end result of using it that way results in simply the re-invention of wheels, such as needing to signal the success or failure of operations on resources. HTTP's purpose, alongside the ideas of REST, is to pull out the common bits that occur when writing code that transfers representation of stuff around, such as, e.g., caching, getting an ok to transfer large content prior to writing it all out to the wire, knowing the status of the operation, or pagination of collections of resources. (I cannot count the number of times I've witnessed API designers reinvent pagination, badly!)
The manner in which people like to use HTTP, with effectively only GET and POST (maybe!) is more akin to RPCs to me. It works. You can do that. But you then need to handle caching, pagination, status of operations, etc. on your own, and you'll end up, I believe, reinventing HTTP. Doing a lot of that (I think, and I think this was Roy's original point) is more effective if you structure your operations around transferring representations of resources around; this is especially visible in caching, because in caching you need the representation of a resource, because that's what a cache works with by its very nature. (As opposed to, say, making opaque method calls on a remote instance.)
Go read [1]; nothing really in there is bound to HTTP, just that HTTP makes a lot of it easier. Also, while I understand that Roy has a lot of arguments around how representations should be hypertext — and I agree with them, mostly — I think that comes second to the ideas that:
1. A URL represents a resource
2. The point of GET / PUT / DELETE is to transfer the state of that resource.
If you don't understand those two points, I don't think you'll understand the arguments behind hypertext.
> They are easy to debug since transaction information is found in easy-to-read JSON inside the payloads using a single, domain specific vocabulary.
Pushing everything into an opaque blob removes the ability for any tooling to pull out high level information. Chrome devtools, httpie, etc., all disprove this point.
> Problem #1: There is little agreement on what a RESTful API is
I agree. I also feel like too many people who think they know have not read anything from Roy Fielding, or even the HTTP RFCs.
> The REST vocabulary is not fully supported
depends mostly on
> most client and server applications don’t support all verbs or response codes for the HTTP protocol. For example, most web browsers have limited support for PUT or DELETE. And many server applications often don’t properly support these methods either.
This isn't true: JavaScript in all browsers in respectable use supports this; Android and iOS fully support HTTP; most server-side development languages have excellent tooling for this. This is a completely false statement, and requires proof, or at least a concrete example to back it up. (Were it true, it only invalidates the use of HTTP as an aid to accomplishing REST.)
> Problem #3: The REST vocabulary is not rich enough for APIs
POST is, essentially, a catch-all for odd operations not supported by other verbs.
> Imagine we create an application where we want to send a “render complete” response back to an HTTP client
If we have a resource that represents a rendering job, if you GET a representation of that render job, it can include some indication of completeness. The bigger problem here is actually HTTP's polling, IMO. Websockets might serve this specific example better, but this singular example doesn't invalidate that most web APIs boil down to CRUD $object of $type, which HTTP supports phenomenally well.
> Problem #5: RESTful APIs are usually tied to HTTP
Well, yes. At the end of the day, it has to be tied to something. HTTP is a pretty good something, with decent tooling.
> They use only one response code to confirm proper receipt of a message - typically 200 OK for HTTP.
Yes, you can send a single bit back. HTTP attempts to be a bit more rich than this. E.g., if something is in progress, and attempting to retrieve the data I just stored will fail until some serve-side job is complete, HTTP can easily signal this. Muxing them into 200 OK removes that information.
> They completely separate the response content from the transmission mechanism. All errors, warnings, and data are placed in the JSON response payload.
HTTP already has this: the response content is the body of the response. The transmission mechanism is HTTP. I don't want errors, warnings, etc., in the payload, where they are opaque and unusable by common tooling.
> They can easily be moved or shared between transmission channels such as HTTP/S, WebSockets, XMPP, telnet, SFTP, SCP, or SSH.
These channels do not support transferring the state of something. (Okay, HTTP does.) telnet is a mostly dumb pipe, same for SSH. You would need to build some custom, non-iteroperable layer on top of that. HTTP is the standardized version of that.
We use NSQ at Dell for the commercial side of dell.com. We've been in Production with it for about 2 years.
> what are the typical use cases for NSQ ?
In the abstract, anything which can tolerate near real-time, at-least-once delivery, and does not need order guarantees. It also features retries and manual requeuing. It's typical to think order and exactly-once semantics are important because that's how we tend to think when we write code and work with (most) databases, and having order allows you to make more assumptions and simplify your approach. It typically comes at the cost of coordination or a bounded window of guarantees. Depending on your workload or how you frame the problem you may find order and exactly-once semantics are not that important, or it can be made unimportant (for example, making messages idempotent). In other cases order is important and it's worth the tradeoff; our Data Science team uses Kafka for these cases, but I'm not familiar with the details.
Here are some concrete examples of things we built using NSQ, roughly in the order they were deployed to PROD:
- Batch jobs which query services and databases to transform and store denormalized data. We process tens of millions of messages in a relatively short amount of time overnight. The queue is never the bottleneck; it's either our own code, services, or reading/writing to the database. Retries are surprisingly useful in this scenario.
- Eventing from other applications to notify a pinpoint refresh is needed for some data into the denormalized store (for example, a user updated a setting in their store, which causes a JSON model to update).
- Purchase order message queue, both for the purpose of retry and simulating what would happen if a customer on a legacy version of the backend was migrated to the new backend; also verifying a set of known 'good' orders continue to be good as business logic evolves (regression testing).
- Async invoice/email generation. This is a case where you have to be careful of at-least-once delivery and need to use a correlation ID and persistence layer to define a 'point of no return' (can't process the message again beyond this point even if it fails). We don't want to email (or bill) customers twice.
- Build system for distributing requests to our build farm.
- Pre-fetching data and hydrating a cache when a user logs in or browses certain pages, anticipating the likely next page to avoid having the user wait on these pages for an expensive service call. The client in this case is another decoupled web application; the application emitting the event is completely separate and likely on a different deployment schedule from the emitting application. The event emitted tells us what the user did, and it's the consumer's responsibility to determine what to do. This is an interesting case where we use #ephemeral channels, which disappear when the last client disconnects. We append the application's version to the channel name so multiple running versions in the same environment will each get their own copy of the message, and process it according to that binary's logic. This is useful for blue/green/canary testing and also when we're mid-deployment and have different versions running in PROD, one customer facing and one internal still being tested. I think I refer to this image more than any other when explaining NSQ's topics and channels: https://f.cloud.github.com/assets/187441/1700696/f1434dc8-60... (from http://nsq.io/overview/design.html).
Operationally, NSQ has been not just a pleasure to work with but inspirational to how we develop our own systems. Being operator friendly cannot be overrated.
Last thing, if you do monitoring with Prometheus I recommend https://github.com/lovoo/nsq_exporter.