Hacker Newsnew | past | comments | ask | show | jobs | submit | kjnilsson's commentslogin

It depends on the current throughput of the system, how many queues a message is routed to, size of the message etc. But a mostly idle RabbitMQ cluster with fast disks should confirm a message published to a single quorum queue in a couple of ms.


Genuine question: what is it that you need from Kafka partitioning?


Not OP, but what I need from Kafka partitioning is guaranteed message ordering (per partition key).

I don't know about RabbitMQ, but with Apache AMQ there is message grouping, which is kind of similar, but not quite the same. With Kafka it's unavoidable, which is good.


Guaranteed ordering per partition while keeping the ability to scale across partitions.


Log rentention is probably the main reason. Event sourcing typically would need some kind of snapshot to be calculated to replace the head of the log before it is deleted.

If you had unbounded storage you could perhaps.


Kafka's key based log compaction can be applied to this.


For me one of the nicest things is that you can have both "traditional" messaging _and_ streaming in the same system. Feeding messages published to exchanges into both queues (for processing) and streams for archiving, auditing and analysis.

Best of both worlds :)


AFAIK there is no proper standard streaming protocol which is why we went with a dedicated protocol that works really well with our approach to streams.


No they do not replicate like classic mirrored queues do. They are much more similar to quorum queues in that they only (asynchronously) replicate the delta after a disconnection. After all both streams and quorum queues use log replication. They are also both quorum systems in terms of availability.

W.r.t quorum queue features set we are working on Message TTLs. Priorities we'll have to see. We want to provide something there but it may not be a priority queue as provided by classic queues as this isn't the best way to do priority based messaging.


Proxying isn't hard to implement but in the erlang Raft implementation, Ra we decided only to do it when the client explicitly declares that it does not care about ordering. When proxying it is always possible that the client will find out the current leader whilst the proxied request is in flight. This may be more of a problem inside erlang where it is easy to address any processes within the erlang cluster.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: