Indeed. If you're hitting AppEngine or GCP Functions, they auto scale workers up for you to manage long running tasks. Ideally though, you finish as quickly as possible by breaking the work down into more tasks. That way, you can parallelize as much as possible.
It is all configurable, but I've scaled up to hundreds of workers at a time to blast through tasks and it wasn't expensive at all.
Workers being an HTTP endpoint makes them super easy to implement and even better... write tests for.
It is all configurable, but I've scaled up to hundreds of workers at a time to blast through tasks and it wasn't expensive at all.
Workers being an HTTP endpoint makes them super easy to implement and even better... write tests for.