Your justification of why "it's essentially impossible to reasonably port it to a sensible transport protocol" appears to be an explanation of how one could reasonably port it to a sensible transport protocol (JSON over some flavour of socket) ;)
Totally agreed that HTTP is inefficient and increases the potential bug surface a bit. However, the ubiquity and convenience empirically seems to outweigh the inefficiency and risk. Just to reiterate: the websockets transport does (in part) what you propose: it doesn't use URIs, it doesn't use HTTP status codes; it shoves a stream of JSON over a websocket. We even wrote an implementation of it: https://github.com/matrix-org/matrix-websockets-proxy. And NOBODY in over a year has felt the urge to actually implement a client against it - instead there have probably been 40-50 new clients, all who were quite happy to use the ugly HTTP API instead. This doesn't feel like people facepalming and walking off, but just saying "huh, let's see if the HTTP thing is that bad in practice. oh, it's fine, i'll concentrate on writing my bot/bridge/client/whatever".
> Your justification of why "it's essentially impossible to reasonably port it to a sensible transport protocol" appears to be an explanation of how one could reasonably port it to a sensible transport protocol (JSON over some flavour of socket) ;)
Except for the fact that it's necessarily redefining quite a bit of the protocol, also necessarily further complicating any possible universal implementations because this kind of construct doesn't even provide any sensible layering abstractions.
> Just to reiterate: the websockets transport does (in part) what you propose: it doesn't use URIs, it doesn't use HTTP status codes
Except that the client and the server still need to understand HTTP in order to then implement a bad version of TCP on top of it, more or less, which makes this whole mess even more bloated and error-prone.
> This doesn't feel like people facepalming and walking off, but just saying "huh, let's see if the HTTP thing is that bad in practice. oh, it's fine, i'll concentrate on writing my bot/bridge/client/whatever".
What exactly would people facepalming and walking off feel like, that this doesn't feel like it?
In case you hadn't guessed it yet, that was exactly my reaction when I looked at the spec a while ago, and it just so happened that I stumbled upon this thread here, or you most likely wouldn't ever have heard about it, and I can only guess that I am not the only one.
(And no, that reaction was not just based on the use of HTTP, but more generally on the impression from the spec that it's all a huge dysfunctional mess that's essentially impossible to implement securely, and that probably will result in even greater interoperability problems than email with all its historical baggage in a much shorter time frame. All of which in turn makes me suspect that the people who are busily writing software for it just are lacking knowledge and experience to see the flaws, and examples like that "C" client do not exactly help to convince me of the opposite.)
From my perspective, the defining bit of Matrix is the data model (eventually-consistent decentralised Merkle DAGs), the schema, and the semantics of the API - e.g. what arguments do you pass when you paginate? what data do you get when you sync? how do you tell it where to sync from? etc. I genuinely don't care whether you do RPC via an HTTP PUT, a COAP PUT, an MQTT Publish, or some funky custom RPC of your own devising - or whether you encode the data as JSON, CBOR, BSON, protobufs or whatever. To say that switching transport from HTTP to Websockets is "necessarily redefining quite a bit of the protocol" entirely misses the point. It's just a transport! We want people to propose alternative transports!
In terms of the websocket transport proposal requiring HTTP to set up the websocket... yes, this is true. I guess you really don't like HTTP - I'm amazed you're willing to use it to communicate on HN! In terms of Matrix being a "huge dysfunctional mess that's essentially impossible to implement securely" - you're entitled to your opinion; it's just a shame it doesn't seem to be particularly well-founded.
> From my perspective, the defining bit of Matrix is the data model (eventually-consistent decentralised Merkle DAGs), the schema, and the semantics of the API
Great, and where is that actually specified?
> or whether you encode the data as JSON, CBOR, BSON, protobufs or whatever
Except that you still need at least a sort-of JSON generator for the signing stuff, thus only adding bloat by using a different format?
> It's just a transport!
Except that it isn't. HTTP is "just a transport" for JPEG files. You don't need to touch JPEG encoders and decoders to swap HTTP for SFTP, that is what "just a transport" means.
> We want people to propose alternative transports!
Which is just idiotic, both because you make it unnecessarily difficult if that really is what you want, and because a federated system that doesn't have a common transport protocol just doesn't work. And if you need to expose an HTTP engine for compatibility anyway, it would be just braindead to add further complexity on top of that.
> I guess you really don't like HTTP - I'm amazed you're willing to use it to communicate on HN!
I don't like unnecessary complexity because it makes it unnecessarily hard to build secure software. The HTTP engines in browsers are probably some of the best-audited ones out there, so it's somewhat acceptable to use them ... but even they have their history of vulnerabilities. The last thing we need is a new large distributed system that's creating even more botnet material than we already have.
> it's just a shame it doesn't seem to be particularly well-founded.
The encoding-agnostic schema for Matrix events is defined at https://github.com/matrix-org/matrix-doc/tree/master/event-s.... We haven't yet written up a transport-agnostic description of the client-server API, mainly because the spec is WIP and we haven't got to it yet, and because in practice it's trivial to procedurally convert the HTTP endpoints into whatever RPC mechanism you prefer. PUTing some JSON to a HTTP URI trivially maps to a JSON-over-TCP RPC mechanism or whatever it is you're pushing for.
Your complaint about the layering seems to be that the HTTP URIs sometimes include values (e.g. event types) associated with the JSON payloads, and that this is a layering violation. This is because HTTP isn't being used as a dumb data transfer transport (e.g. your HTTP v. SFTP example) but as an RPC transport - and you pass parameters when you make an RPC call. Sure, it means you have to map the RPC mechanism if you switch transports, but it's hardly the end of the world. In future I guess we'll specify the CS API semantics as some IDL style format to make it easier to map to other RPCs.
The idea that you can't expose alternative transports because "adding additional complexity when you are already exposing an HTTP engine is braindead" is just obtuse. If you really don't trust HTTP then by all means turn off the HTTP listener the client-server API and only support clients which talk your better transport protocol. You're obviously stuck talking it on the federation API for now, but we've always reserved the right to switch to an entirely different federation mechanism (e.g. a pure TCP line protocol, WebRTC datachannels, the IPFS DAG or whatever) in future - the migration would be similar to bridging Matrix with any other new network.
The idea that HTTP client libraries like python's, go's or curl's are somehow less reputable or audited than those used in web browsers seems questionable at best. But agreed that the last thing we need is a large distributed network which can be used as a botnet, and obviously it's in our interests to avoid that. Unconvinced that vulnerable HTTP stacks is the biggest risk there though...
> The encoding-agnostic schema for Matrix events is defined at [...]
Are the algorithms for the distributed data structure actually documented anywhere? Just schemas don't really get you particularly far is you want to build a reliable global scale network.
> Your complaint about the layering seems to be that the HTTP URIs sometimes include values (e.g. event types) associated with the JSON payloads, and that this is a layering violation.
No, it's not a layering violation, it's just that that makes HTTP not "just a ...
> This is because HTTP isn't being used as a dumb data transfer transport (e.g. your HTTP v. SFTP example) but as an RPC transport - and you pass parameters when you make an RPC call.
... transport", but an RPC mechanism. Which means that you cannot swap it out for another transport, but only for another RPC mechanism. Which is completely unnecessary.
Also, you keep writing "HTTP", as if you were using HTTP. You still are not.
> but it's hardly the end of the world.
Said about everyone responsible for the protocols and frameworks and stuff that are the causes for the mess of vulnerabilities we find ourselves in today. Yes, actually, exactly that is the end of the world, so to speak.
> In future I guess we'll specify the CS API semantics as some IDL style format to make it easier to map to other RPCs.
Yay, more bloat!
> If you really don't trust HTTP then by all means turn off the HTTP listener the client-server API and only support clients which talk your better transport protocol.
Now, that is obtuse. "If this really turns out to be a security problem, just reinvent it all on your own in better, and the problems will be fixed!" Thanks, yes, I was aware of that option. But it's not exactly a good argument for doing it the way you propose.
> You're obviously stuck talking it on the federation API for now,
Exactly!
> but we've always reserved the right to switch to an entirely different federation mechanism (e.g. a pure TCP line protocol, WebRTC datachannels, the IPFS DAG or whatever) in future - the migration would be similar to bridging Matrix with any other new network.
AKA: It won't happen. Most likely. That's network effects rearing their ugly head. Reserving the right means exactly nothing: If the matter of fact is that the world speaks SMTP, reserving the right to switch email transport to MUCHBETTERFORREALMTP will have exactly zero consequence. Even the pain with IPv4 is causing hardly any movement ... or maybe it is now, slowly, more than a decade late. Heck, even explicit compatibility mechanisms don't help, see TLS: Implementations from big vendors fuck(ed) up version negotiation so badly that more than one mechanism in the protocol had to be abandoned in newer versions because they couldn't negotiate backwards compatibility with the old version with those broken implementations, so introducing a new TLS version in clients would have had tons of website break.
> The idea that HTTP client libraries like python's, go's or curl's are somehow less reputable or audited than those used in web browsers seems questionable at best.
I didn't say that, but those are three of hundreds of implementations, and none of them is exactly light-weight.
And I am not saying that that makes curl a bad library.
> Unconvinced that vulnerable HTTP stacks is the biggest risk there though...
Well, for one, it's maybe the biggest risk that you cannot easily avoid in an implementation (because it's something that you cannot do without, unlike, say, parsing image formats in the client, which probably would be the biggest risk otherwise), but also, see above: It's almost always the small stuff that's "not the end of thr world" that actually is, in the end. If you asked the people that have built the protocol specs and the software that ended up being used to build the botnets of the world, what would they say? "Yeah, we intentionally put in this huge hole so people could build a botnet"? I guess not.
Totally agreed that HTTP is inefficient and increases the potential bug surface a bit. However, the ubiquity and convenience empirically seems to outweigh the inefficiency and risk. Just to reiterate: the websockets transport does (in part) what you propose: it doesn't use URIs, it doesn't use HTTP status codes; it shoves a stream of JSON over a websocket. We even wrote an implementation of it: https://github.com/matrix-org/matrix-websockets-proxy. And NOBODY in over a year has felt the urge to actually implement a client against it - instead there have probably been 40-50 new clients, all who were quite happy to use the ugly HTTP API instead. This doesn't feel like people facepalming and walking off, but just saying "huh, let's see if the HTTP thing is that bad in practice. oh, it's fine, i'll concentrate on writing my bot/bridge/client/whatever".