But it looks like the sources of non-determinism in PyTorch are known, and can be avoided with a lot of work and loss of performance?
And for the general case, I don't think it's impossible to write deterministic code for multi-core processors?
> The errors are small rounding errors
But rounding errors don't imply non-deterministic answers, right? Just that the answer is different from the true answer?
Calculating the square root of 2 will have a rounding error with 32-bit floating point, but are you saying that you'll get different bit patterns in your FP32 due to rounding errors?
Ok, I see what you mean.
I can see how that could be the case. It depends on how the software is designed.
Now that I looked up it, I was surprised to see that PyTorch may generate non-reproducible results: https://pytorch.org/docs/stable/notes/randomness.html
But it looks like the sources of non-determinism in PyTorch are known, and can be avoided with a lot of work and loss of performance?
And for the general case, I don't think it's impossible to write deterministic code for multi-core processors?
> The errors are small rounding errors
But rounding errors don't imply non-deterministic answers, right? Just that the answer is different from the true answer?
Calculating the square root of 2 will have a rounding error with 32-bit floating point, but are you saying that you'll get different bit patterns in your FP32 due to rounding errors?