Yeah, you'd need a proxy server (websockets) to establish WebRTC connections. Even after a WebRTC connection is established, you'd need a STUN or a TURN (coturn supports both) server to get around NATs. Pls somebody correct me if I'm wrong!
You should only need a STUN or TURN server if you are attempting to connect to a machine that is behind a NAT. OpenTTD servers are usually on public IPs (similar to HTTP servers), so a STUN or TURN server should not be required.
The biggest issue is that WebRTC connections do not allow you to send raw data over TCP or UDP[1].
That being said, it shouldn't be too difficult to just create a WebSocket server that translates between the native network protocol and HTTP WebSockets[2].