Matrix server backdoors aren't an issue though? It's about the client where decryption happens. If those aren't required to upload decrypted contents, you can always overlay some encryption protocol like OTR over any chat mechanism. I remember using it on MSN via Pidgin

Don't worry about the servers. Worry about mandated software on the client