Agreed. I used TensorFlow 1.x before PyTorch matured. Today, all of my deep learning is PyTorch based because TensorFlow 2.x didn't fix the core TensorFlow 1.x issues (e.g., poor reproducibility, unintuitive API design) and PyTorch performs better than Keras w/TensorFlow backend.

For what it's worth, I've found Pytorch to be much more rigid than TF. Maybe I just haven't found the easy way to do things. For example here's a function that applies an N×N box filter to all but the first 2 dimensions of a tensor (apologies to mobile users):

    def boxfilter(image, N=3):
        shape = image.shape
        image = tf.reshape(image, [1, shape[0], shape[1], -1])
        C = image.shape[-1]
        conv = tf.nn.conv2d(image, tf.eye(C, C, [N, N]), 1, "SAME")[0]
        return tf.reshape(conv, shape) / (N * N)

Is there a simple way to do this in Pytorch? Preferably without having to inherit from the base class for convolution. It seems to me that Pytorch is like Keras and Tensorflow is like Numpy.

You can almost 1:1 translate this by swapping "tf" and "torch". No need to use nn.Conv2d -- there's a functional API for all these layers:

https://pytorch.org/docs/master/nn.functional.html#conv2d

Torch doesn't have "same" padding, so you have to manually calculate the correct padding value for your input/output shapes.

That looks great, thanks!

I read somewhere saying TF2 is good for production while Pytorch is good for research(and papers), is this true? I'm more interested in putting one of them into real products, esp standalone embedded devices.

It used to be true, but nowadays there are several options to deploy Pytorch models:

1. PyTorch C++ API which can trivially load and execute your models exported via JIT

2. ONNX export and inference in TensorRT (highest performance inference option)

3. Or just deploy straight up PyTorch Python code - it'll run fine in "production".

One place where PyTorch is weaker than TF is mobile. TFLite is a lot more mature and has all sorts of acceleration support (GPU, DSP). So if that's what you need, at this point there's really no other good choice IMO.

Does it train just as fast on gpus?

In my experience, not quite as fast for fully-tuned code, but the difference is small - and given the same project deadline, the PyTorch version will probably be faster.

Properly tuned (DistributedDataParallel + mixed precision) it will train faster, and consume a lot less RAM, allowing you to use larger batches, and higher LR.