> From my perspective, the defining bit of Matrix is the data model (eventually-consistent decentralised Merkle DAGs), the schema, and the semantics of the API
Great, and where is that actually specified?
> or whether you encode the data as JSON, CBOR, BSON, protobufs or whatever
Except that you still need at least a sort-of JSON generator for the signing stuff, thus only adding bloat by using a different format?
> It's just a transport!
Except that it isn't. HTTP is "just a transport" for JPEG files. You don't need to touch JPEG encoders and decoders to swap HTTP for SFTP, that is what "just a transport" means.
> We want people to propose alternative transports!
Which is just idiotic, both because you make it unnecessarily difficult if that really is what you want, and because a federated system that doesn't have a common transport protocol just doesn't work. And if you need to expose an HTTP engine for compatibility anyway, it would be just braindead to add further complexity on top of that.
> I guess you really don't like HTTP - I'm amazed you're willing to use it to communicate on HN!
I don't like unnecessary complexity because it makes it unnecessarily hard to build secure software. The HTTP engines in browsers are probably some of the best-audited ones out there, so it's somewhat acceptable to use them ... but even they have their history of vulnerabilities. The last thing we need is a new large distributed system that's creating even more botnet material than we already have.
> it's just a shame it doesn't seem to be particularly well-founded.
The encoding-agnostic schema for Matrix events is defined at https://github.com/matrix-org/matrix-doc/tree/master/event-s.... We haven't yet written up a transport-agnostic description of the client-server API, mainly because the spec is WIP and we haven't got to it yet, and because in practice it's trivial to procedurally convert the HTTP endpoints into whatever RPC mechanism you prefer. PUTing some JSON to a HTTP URI trivially maps to a JSON-over-TCP RPC mechanism or whatever it is you're pushing for.
Your complaint about the layering seems to be that the HTTP URIs sometimes include values (e.g. event types) associated with the JSON payloads, and that this is a layering violation. This is because HTTP isn't being used as a dumb data transfer transport (e.g. your HTTP v. SFTP example) but as an RPC transport - and you pass parameters when you make an RPC call. Sure, it means you have to map the RPC mechanism if you switch transports, but it's hardly the end of the world. In future I guess we'll specify the CS API semantics as some IDL style format to make it easier to map to other RPCs.
The idea that you can't expose alternative transports because "adding additional complexity when you are already exposing an HTTP engine is braindead" is just obtuse. If you really don't trust HTTP then by all means turn off the HTTP listener the client-server API and only support clients which talk your better transport protocol. You're obviously stuck talking it on the federation API for now, but we've always reserved the right to switch to an entirely different federation mechanism (e.g. a pure TCP line protocol, WebRTC datachannels, the IPFS DAG or whatever) in future - the migration would be similar to bridging Matrix with any other new network.
The idea that HTTP client libraries like python's, go's or curl's are somehow less reputable or audited than those used in web browsers seems questionable at best. But agreed that the last thing we need is a large distributed network which can be used as a botnet, and obviously it's in our interests to avoid that. Unconvinced that vulnerable HTTP stacks is the biggest risk there though...
> The encoding-agnostic schema for Matrix events is defined at [...]
Are the algorithms for the distributed data structure actually documented anywhere? Just schemas don't really get you particularly far is you want to build a reliable global scale network.
> Your complaint about the layering seems to be that the HTTP URIs sometimes include values (e.g. event types) associated with the JSON payloads, and that this is a layering violation.
No, it's not a layering violation, it's just that that makes HTTP not "just a ...
> This is because HTTP isn't being used as a dumb data transfer transport (e.g. your HTTP v. SFTP example) but as an RPC transport - and you pass parameters when you make an RPC call.
... transport", but an RPC mechanism. Which means that you cannot swap it out for another transport, but only for another RPC mechanism. Which is completely unnecessary.
Also, you keep writing "HTTP", as if you were using HTTP. You still are not.
> but it's hardly the end of the world.
Said about everyone responsible for the protocols and frameworks and stuff that are the causes for the mess of vulnerabilities we find ourselves in today. Yes, actually, exactly that is the end of the world, so to speak.
> In future I guess we'll specify the CS API semantics as some IDL style format to make it easier to map to other RPCs.
Yay, more bloat!
> If you really don't trust HTTP then by all means turn off the HTTP listener the client-server API and only support clients which talk your better transport protocol.
Now, that is obtuse. "If this really turns out to be a security problem, just reinvent it all on your own in better, and the problems will be fixed!" Thanks, yes, I was aware of that option. But it's not exactly a good argument for doing it the way you propose.
> You're obviously stuck talking it on the federation API for now,
Exactly!
> but we've always reserved the right to switch to an entirely different federation mechanism (e.g. a pure TCP line protocol, WebRTC datachannels, the IPFS DAG or whatever) in future - the migration would be similar to bridging Matrix with any other new network.
AKA: It won't happen. Most likely. That's network effects rearing their ugly head. Reserving the right means exactly nothing: If the matter of fact is that the world speaks SMTP, reserving the right to switch email transport to MUCHBETTERFORREALMTP will have exactly zero consequence. Even the pain with IPv4 is causing hardly any movement ... or maybe it is now, slowly, more than a decade late. Heck, even explicit compatibility mechanisms don't help, see TLS: Implementations from big vendors fuck(ed) up version negotiation so badly that more than one mechanism in the protocol had to be abandoned in newer versions because they couldn't negotiate backwards compatibility with the old version with those broken implementations, so introducing a new TLS version in clients would have had tons of website break.
> The idea that HTTP client libraries like python's, go's or curl's are somehow less reputable or audited than those used in web browsers seems questionable at best.
I didn't say that, but those are three of hundreds of implementations, and none of them is exactly light-weight.
And I am not saying that that makes curl a bad library.
> Unconvinced that vulnerable HTTP stacks is the biggest risk there though...
Well, for one, it's maybe the biggest risk that you cannot easily avoid in an implementation (because it's something that you cannot do without, unlike, say, parsing image formats in the client, which probably would be the biggest risk otherwise), but also, see above: It's almost always the small stuff that's "not the end of thr world" that actually is, in the end. If you asked the people that have built the protocol specs and the software that ended up being used to build the botnets of the world, what would they say? "Yeah, we intentionally put in this huge hole so people could build a botnet"? I guess not.
Great, and where is that actually specified?
> or whether you encode the data as JSON, CBOR, BSON, protobufs or whatever
Except that you still need at least a sort-of JSON generator for the signing stuff, thus only adding bloat by using a different format?
> It's just a transport!
Except that it isn't. HTTP is "just a transport" for JPEG files. You don't need to touch JPEG encoders and decoders to swap HTTP for SFTP, that is what "just a transport" means.
> We want people to propose alternative transports!
Which is just idiotic, both because you make it unnecessarily difficult if that really is what you want, and because a federated system that doesn't have a common transport protocol just doesn't work. And if you need to expose an HTTP engine for compatibility anyway, it would be just braindead to add further complexity on top of that.
> I guess you really don't like HTTP - I'm amazed you're willing to use it to communicate on HN!
I don't like unnecessary complexity because it makes it unnecessarily hard to build secure software. The HTTP engines in browsers are probably some of the best-audited ones out there, so it's somewhat acceptable to use them ... but even they have their history of vulnerabilities. The last thing we need is a new large distributed system that's creating even more botnet material than we already have.
> it's just a shame it doesn't seem to be particularly well-founded.
lol