Mixture of Experts is absolutely not what they're describing. MoE has to be one of the most misleadingly named things ever. It's completely confusing as to what it actually is.
MoE is literally exactly what they're describing. The classifier being described is baked into the model and is the thing that makes it MoE.
Imagine I had [a model that was good at math], [model that was good at code], [model that was good at writing], [model that was good at general knowledge]. If I then had [a model that was good at determining whether the user query would be best served by one of those models and sent it to it, leaving the rest of the models inactive], that is the platonic version of what MoE is. In practice, it works a bit differently. It instead basically restricts the number of pathways that can be utilized in solving problems during training, which allows for "expert neuron groupings" to form and "classifier layers" to form earlier on in the structure, but the effect is the same (better, even, since it allows some overlap between structures of experts). It also allows "routing to an expert" to happen token-by-token rather than at the prompt level.
Note the “but free from unnecessary inductive biases” part of my comment. By that I meant a decision to make each expert good at a human-defined thing.
It uses a third server to facilitate initial p2p connections but I keep loosing/fail to connect to this server. I don't know if it's because of many restarts during development or something else.
Windows Defender nukes this from orbit, making it nearly impossible to ship to clients in a trusting fashion. But I guess any program which punches through the firewall is suspect.
Windows Defender is an interesting challenge. It would be interesting to know if signing the executable has a positive effect here. At $previouscompany we had a software that looked very keylogger-like, and all our Windows Defender issues vanished once we started using EV codesigning certificates. They are not cheap ($300/year), but Defender seems to take the fact that the code is bound to a verified legal entity as a strong trust signal
That's interesting, because the connection to the relay server is established using HTTP1.1 over TLS. Followed by a WebSocket upgrade. It should look like any other webserver connection on the internet. Could be worth investigating your network conditions and filing an issue for this.
Nobody is building commercial plants any time soon; it's still in the experimental phase, with new discoveries happening almost every month.
I see it similarly to the difference between a car with a combustion engine and an electric one. Combustion engines are fully developed. We're reaching the maximum possible performance and utilisation. It's a dead end. However, with electric cars, for example, new battery development is far from over. E.g sodium batteries.
And just off the top of my head, in fusion, the discovery of better electromagnets, as happened a while back, can quadruple energy output.It's not a dead end, and writing it off would be short-sighted.
If I remember correctly it actually is a typo. Someone typed "verloren" wrong and thought it was a good name for the game. The story behind names is often not so glamorous.
I kinda hate many current game names, I prefer this. Otherwise, making a current_year name is easy: noun adjective: noun. How about "Dark Kingdom: Multitude"?
