I manage ~3000 customized websites based on the same template code. Sometimes we make changes to the template code that could affect the customizations - it is practically impossible to predict what might cause a problem due to the nature of the customizations. We'll take before and after screenshots of every page on every site, so it can get into the 100s of thousands of screenshots. We'll then run a diff on the screenshots to see what changed, reviewing the screenshots with the most significant changes. Then we'll address the problems we find and deploy the fixed release.

When we do these large screenshot operations, the EC2 instances are running for maybe 15 or 20 minutes total. It's not exactly cheap, but losing clients because we broke their site is something we want to avoid. The sites are hosted on a 3rd party service, and we're rate-limited by IP address, so to get this done in a reasonable amount of time we need to spin up 10,000 EC2 instances to distribute the work. We have our own software to manage the EC2 instances. It's honestly pretty simple, but effective.