agree there, but then again using ort in Rust is faster again.
You cannot compare python with a onxx executor.
I don't know what you used in Python, but if it's pytorch or similar, those are built with flexibility in mind, for optimal performance you want to export those to onxx and use whatever executor that is optimized for your env. onxxruntime is one of them, but definitely not the only one, and given it's from Microsoft, some prefer to avoid it and choose among the many free alternatives.
Why would the two not be entirely comparable? PyTorch may be slower at building the models; but once the model is compiled and loaded on the NPU, there's just not a whole lot of Python involved anymore. A few hundred CPU cycles to push the input data using python; a few hundred CPU cycles to receive the results using python. And everything in-between gets executed on the NPU.
I really wish Python wasn't the language controlling all the C code. You need a controller, in a scripting language that's easy to modify, but it's a rather hideous choice. It would be like choosing to build the world's largest social network in PHP or something. lol.
I've just spent a week writing Neural net code in C++, so i have direct insight into what a C++ implementation might look like.
Much as I dislike python, and having to deal with endless runtime errors when your various inputs and outputs are mismatched, the inconvenience pales in comparison with having to wade through three pages of error messages that a C++ compiler generates when you have a single mismatched templated Eigen matrix with the incorrect dimension. The "Required from here" message you are actually interested in is typically the 3rd or fourth "Required from here", amongst a massive stack of cascading errors, each of which wraps to about 4 lines when displayed. You know what I mean. Sometimes you don't get a "Required from here" at all, which is horrifying. And it's infuriating to find and parse the template arguments of classes involved.
Debugging Python runtime errors is kind of horrible, and rubs me the wrong way on principle. But it is sweetness and light compared to debugging C++ compile-time error messages, which are unimaginably horrible.
The project: converting a C++ Neural Amp Modeller LV2 plugin to use fixed-size matrices (Eigen::Matrix<float,N,M>) instead of dynamic matrixes (Eigen::MatrixXf) to see if doing so would improve performance. (It does. Signficantly). So a substantial and realistic experiment in doing ML work in C++. Not directly comparable to working in Pytorch, but directly analogous in that it involves hooking up high-level ML constructs, like Conv1D, LayerT, WaveNet ML chunks.
Yes, that was the case. I was being sarcastic. Zuck wrote facebook in PHP and spent millions of dollars then writing a custom interpreter to let his janky code run slightly faster than normal.
Zuck's obvious mistake: he should have written the PHP compiler to precompile chunks of GPU code that would be the only code that actually runs when serving web pages. </sarcasm>
Facebook isn't really a comparable problems, because ALL of the performance-critical code in PyTorch does runs on a GPU.
You cannot compare python with a onxx executor.
I don't know what you used in Python, but if it's pytorch or similar, those are built with flexibility in mind, for optimal performance you want to export those to onxx and use whatever executor that is optimized for your env. onxxruntime is one of them, but definitely not the only one, and given it's from Microsoft, some prefer to avoid it and choose among the many free alternatives.