This is the way of thinking about it that works best for me. As you fall into the gravity well of the planet from behind it, the planet is moving in the same direction as you. That means you spend more time in its gravity well catching up to it than if it was stationary. As you pass the planet you have picked up as speed boost from its pull. But as you travel away from it, because you are going faster you spend less time in its influence, so the deceleration is less than the acceleration you picked up on the approach.
As I try to wrap my brain around this could I assume then in order for this to work, on approach to the planet you have to be going in the same direction as its orbit?
Theoretically no, but if you're going the opposite direction the approach velocity is vastly higher and thus you need a (1) very massive planet and (2) much lower periapsis to have meaningful gravitational slingshot effect. In practice, between atmospheres (the big planets being Jupiter, Saturn, and Neptune), and (for non-gaseous planets) planets not being perfectly smooth spheres, that probably decreases the tolerances below any acceptable level in this solar system.
Yes but in the time-reversed version, the planet would be moving the opposite direction so you'd be doing a "reverse gravity assist" which slows you down.
I don't think that is the right way to think about it, it has more to do with trajectory bending. For example, let's say an object is going radially outwards away from the sun at 10 km/s when it encounters a planet that is orbiting at 30 km/s.
Let's say the trajectories are at right angles, so their relative speed difference before the encounter is 31.6 km/s.
I now claim that this relative speed difference is the same before and after the encounter, so there's no "less time spent decelerating on the way out". (This is an approximation, I'm pretending that the planet provides an inertial reference frame with conservation of energy. The centripetal/sunward acceleration is small enough for that.) Let me show how you can still get a gravity assist under this assumption.
If the object passes "in front" of the planet in such a way that its trajectory gets bent towards the retrogade direction of the planet, then its speed becomes 30 - 31.6 = -1.6 km/s. So 1.6 km/s relative to the sun, and direction retrograde of the planet.
If the object passes "behind" the planet in such a way that its trajectory gets bent towards the prograde direction of the planet, then its speed becomes 30 + 31.6 = 30 + 31.6 = 61.6 km/s relative to the sun, this time in the prograde direction of the planet.
I hope I've convinced you that gravity assists work by taking the relative speed difference, and reorienting it.