If I can look at the code, decide I trust the implementations of the primitives being used, how they're being used, how identity is established, and how initial key exchange works, I don't need to know what's running on the server. That's sort of the point of end to end encryption.
You mean using the algorithm to verify that the observable input leads to the observable output? That would make sense and would allow you to form an opinion about the "primitives" like you said.
