Hi Cory, this was a fantastic writeup! Your explanation of the value of generating a codebase for people instead of having them use a project template is compelling; it articulates that aversion I've always felt towards large templated projects.
The section about testing is particularly interesting to me. I'm working on django-simple-deploy [0], a project that automates people's initial deployments of a Django project. When people first hear about this project, they assume it's just for beginners. But when a lot of people started migrating away from Heroku, we saw how much work it is even for experienced developers to read through a new platform's docs and configure their project correctly for deployment. simple_deploy does all that configuration in one pass, and then all of the platform-specific configuration is contained in one commit.
A fun question I've shared when talking about this project is "How do you test a standalone management command, whose goal is to act on a variety of Django projects?" In this project there's no settings file, no models, no urls, no views, etc. The project also needs to support multiple dependency management systems, multiple target platforms, and since it works on the user's local system it needs to be cross platform as well.
To deal with this efficiently, the unit test suite copies a sample project to a temp directory. It then builds a venv, and runs a commit. It runs simple_deploy against the project, and then does as many tests as it can against the configuration that was done for that project. Then, instead of starting over, it uses git to reset the sample project to the original state. This lets it run ~90 tests in about 10 seconds.
Even more interestingly, that temp project is still available when the test suite finishes. pytest takes care of deleting it at some point, but not immediately. So, debugging can be really efficient. If you run `pytest -x`, you can open that test project and poke around to see what didn't work. The setup uses an editable install, so you can modify simple_deploy, reset the test project manually, and run the command again (outside of the test suite).
pytest is an amazing tool. Just yesterday I wrote a plugin that lets you run `pytest -x --open-test-project`. When the test suite exits, it pops open a new terminal window at the test project, with an active virtual environment and the output of `git status` and `git log`. This turns the test suite into a development tool.
I'm curious if you use your test suite for development in any way, or if it's just for catching bugs and regressions? Also, with that tree of configuration options, have you integrated randomization into your tests at all? That seems like it might be a good way to hit branches you haven't tried before, but then again the majority of the 33M branches are probably not meaningful to test.
Wow, that sounds like a really nice test setup! And makes sense that simple-deploy would be a very complicated thing to properly test.
For now the tree-like test suite is mostly for regressions. The inner test suite for the built projects is useful for dev, but not the one that does all the permutations.
Randomization is a fun idea! It would be interesting to randomize some number of runs with each build. I wonder if that would catch anything. Biggest problem is keeping my Github Actions bill manageable...
The section about testing is particularly interesting to me. I'm working on django-simple-deploy [0], a project that automates people's initial deployments of a Django project. When people first hear about this project, they assume it's just for beginners. But when a lot of people started migrating away from Heroku, we saw how much work it is even for experienced developers to read through a new platform's docs and configure their project correctly for deployment. simple_deploy does all that configuration in one pass, and then all of the platform-specific configuration is contained in one commit.
A fun question I've shared when talking about this project is "How do you test a standalone management command, whose goal is to act on a variety of Django projects?" In this project there's no settings file, no models, no urls, no views, etc. The project also needs to support multiple dependency management systems, multiple target platforms, and since it works on the user's local system it needs to be cross platform as well.
To deal with this efficiently, the unit test suite copies a sample project to a temp directory. It then builds a venv, and runs a commit. It runs simple_deploy against the project, and then does as many tests as it can against the configuration that was done for that project. Then, instead of starting over, it uses git to reset the sample project to the original state. This lets it run ~90 tests in about 10 seconds.
Even more interestingly, that temp project is still available when the test suite finishes. pytest takes care of deleting it at some point, but not immediately. So, debugging can be really efficient. If you run `pytest -x`, you can open that test project and poke around to see what didn't work. The setup uses an editable install, so you can modify simple_deploy, reset the test project manually, and run the command again (outside of the test suite).
pytest is an amazing tool. Just yesterday I wrote a plugin that lets you run `pytest -x --open-test-project`. When the test suite exits, it pops open a new terminal window at the test project, with an active virtual environment and the output of `git status` and `git log`. This turns the test suite into a development tool.
I'm curious if you use your test suite for development in any way, or if it's just for catching bugs and regressions? Also, with that tree of configuration options, have you integrated randomization into your tests at all? That seems like it might be a good way to hit branches you haven't tried before, but then again the majority of the 33M branches are probably not meaningful to test.
- [0] https://django-simple-deploy.readthedocs.io/en/latest/