I was going through the list, nodding as I read all the reasons, agreeing with them all. But also thinking each time I’ll get one anyway. I guess I agree with the last sentence of the post haha. If anything, I’ll justify this by saying I want to choose with my money and encourage this further. But that’s really just to give me good conscience.
Thanks for sharing! Your article illustrates well the benefits of this approach.
One drawback I see is that property-based tests inevitably need to be much more complex than example-based ones. This means that bugs are much more likely, they're more difficult to maintain, etc. You do mention that it's a lot of code, but I wonder if the complexity is worth it in the long run. I suppose that since testing these scenarios any other way would be even more tedious and error-prone, the answer is "yes". But it's something to keep in mind.
> One drawback I see is that property-based tests inevitably need to be much more complex than example-based ones.
I don’t think that’s true, I just think the complexity is more explicit (in code) rather than implicit (in the process of coming up with examples). Example-based testing usually involves defining conditions and properties to be tested, then involves constructing sets of examples to test them and which attempt to anticipated edge cases from the description of the requirements (black box) or from knowledge of how the code is implemented (white box).
Property based testing involves defining the conditiosn and properties, writing code that generates the conditions and for each property writing a bit of code that can refute it by passing if and only if it is true of the subject under test for a particular set of inputs.
With a library like Hypothesis which both has good generators for basic types and good abstractions for combining and adapting generators, the latter seems to be less complex overall, as well as moving the complexity into a form where it is explicit and easy to maintain/adapt, whereas adapting example-based tests to requirements changes involves either throwing out examples and starting over or revalidating and updating examples individually.
> Property based testing involves defining the conditiosn and properties, writing code that generates the conditions and for each property writing a bit of code that can refute it by passing if and only if it is true of the subject under test for a particular set of inputs.
You're downplaying the amount of code required to properly setup a property-based test. In the linked article, the author implemented a state machine to accurately model the SUT. While this is not the most complex of systems, it is far from trivial, and certainly not a "bit of code". In my many years of example-based unit/integration/E2E testing, I've never had to implement something like that. The author admits that the team was reluctant to adopt PBT partly because of the amount of code.
This isn't to say that example-based tests are simple. There can be a lot of setup, mocking, stubbing, and helper code to support the test, but this is usually a smell that something is not right. Whereas with PBT it seems inevitable in some situations.
But then again, I can see how such tests can be invaluable, very difficult and likely more complex to implement otherwise. So, as with many things, it's a tradeoff. I think PBT doesn't replace EBT, nor vice-versa, but they complement eachother.
You’re right it’s always a trade off. One unexpected but very welcomed side effect of having those stateful property tests is we could use them to design high fidelity stubs. I wrote a follow-up blog post about it https://blog.tiserbox.com/posts/2024-07-08-make-good-stubs-w...
My experience is that PBT tests are mostly hard in devising the generators, not in the testing itself.
Since it came up in another thread (yes, it's trivial), a function `add` is no easier or harder to test with examples than with PBT, here are some of the tests as both PBT-style and example-based style:
@given(st.integers())
def test_left_identity_pbt(a):
assert add(a, 0) == a
def test_left_identity():
assert add(10, 0) == 10
@given(st.integers(), st.integers())
def test_commutative(a, b):
assert add(a, b) == add(b, a)
@parametrize("a,b", examples)
def test_commutative():
assert add(a, b) == add(b, a)
They're the same test, but one is more comprehensive than the other. And you can use them together. Supposing you do find an error, you add it to your example-based tests to build out your regression test suite. This is how I try to get people into PBT in the first place, just take your existing example-based tests and build a generator. If they start failing, that means your examples weren't sufficiently comprehensive (not surprising). Because PBT systems like Hypothesis run so many tests, though, you may need to either restrict the number of generated examples for performance reason or breakup complex tests into a set of smaller, but faster running, tests to get the benefit.
Other things become much simpler, or at least simpler to test comprehensively, like stateful and end-to-end tests (assuming you have a way to programmatically control your system). Real-world, I used Hypothesis to drive an application by sending a series of commands/queries and seeing how it behaved. There are so many possible sequences that manually developing a useful set of end-to-end tests is non-trivial. However, with Hypothesis it just generated sequences of interactions for me and found errors in the system. After each command (which may or may not change the application state) it issued queries in the invariant checks and verified the results against the model. Like with example-based testing, these can be turned into hard-coded examples in your regression test suite.
For sure, the hardest part is to create meaningful generators for the problem at hand which can test interesting cases in a finite amount of time. That’s where the combinatory explosion takes place in my experience.
I wanted to highlight one unexpected but very welcomed side effect of having those stateful property tests is we could use them to design high fidelity stubs. I wrote a follow-up blog post about it https://blog.tiserbox.com/posts/2024-07-08-make-good-stubs-w...
> Since it came up in another thread (yes, it's trivial), a function `add` is no easier or harder to test with examples than with PBT
Come on, that example is practically useless for comparing both approaches.
Take a look at the article linked above. The amount of non-trivial code required to setup a PBT should raise an eyebrow, at the very least.
It's quite possible that the value of such a test outweighs the complexity overhead, and that implementing all the test variations with EBT would be infeasible, but choosing one strategy over the other should be a conscious decision made by the team.
So as much as you're painting PBT in a positive light, I don't see it that clearly. I think that PBT covers certain scenarios better than EBT, while EBT can be sufficient for a wide variety of tests, and be simpler overall.
But again, I haven't actually written PBTs myself. I'm just going by the docs and articles mentioned here.
> Come on, that example is practically useless for comparing both approaches.
Come on, I admitted it was trivial. It was a quick example that fit into a comment block. Did you expect a dissertation?
> that implementing all the test variations with EBT would be infeasible
That's kind of the point to my previous comment. PBTs will generate many more examples than you would create by hand. If you have EBTs already, you're one step away from PBTs (the generators, I never said this was trivial just to preempt another annoying "Come on"). And then you'll have more comprehensive testing than you would have had sticking to just your carefully handcrafted examples. This isn't the end of property-based testing, but it's a really good start and the easiest way to bring it into an existing project because you can mostly reuse the existing tests.
Extending this, once you get used to it, to stateful testing (which many PBT libraries support, including Hypothesis) you can generate a lot of very useful end-to-end tests that would be even harder to come up with by hand. And again, if you have any example-based end-to-end tests or integration tests, you need to come up with generators and you can start converting them into property-based tests.
> but choosing one strategy over the other should be a conscious decision made by the team.
Ok. What prompted this? I never said otherwise. It's also not an either/or situation, which you seem to want to make it. As I wrote in that previous comment, you can use both and use the property-based tests to bolster the example-based tests, and convert counterexamples into more example-based tests for your regression suite.
I help privacy and data sovereignty enthusiasts take back control of their data without needing to change their habits.
I’ve been working for the past 3 years on SelfHostBlocks https://github.com/ibizaman/selfhostblocks, making self-hosting a viable and convenient alternative to the cloud for non technical people.
It is based on NixOS and provides a hand-picked groupware stack: user-facing there is Vaultwarden and Nextcloud (and a bunch more but those 2 are the most important IMO for non technical people as it covers most of one’s important data) and on the backend Authelia, LLDAP, Nginx, PostgreSQL, Prometheus, Grafana and some more. My know-how is in how to configure all this so they play nice together and to have backups, SSO, LDAP, reverse proxy, etc. integration. I’m using it daily as the house server, I’m my first customer after all. And beginning of 2025 it passed my own internal checkpoint to be shared with others and there’s a handful of technical users using it.
My goal is to work on this full time. I started a company to provide a white glove installation, configuration and maintenance of a server with SelfHostBlocks. Everything I’ll be doing will always be open source, same as the whole stack and the server is DIY and repair friendly. The continuous maintenance is provided with a subscription which includes customer support and training on the software stack as needed.
I used a similar setup a long while ago and wanted a way to connect to databases easily. So I wrote some code to parse the MySQL and Postres config files to extract database info in Emacs: https://stackoverflow.com/a/46114944
This is the business model I want to have: I work on a stack of fully open source software and package them in a turn-key server that you own. You can use it on your own for free if you’re knowledgeable and I offer a subscription where I’m the sysadmin of the box you own and that I built for you. I do the maintenance, the updates, etc. There’s no lock-in because you can stop the subscription anytime or even just pick another sysadmin that would know the stack. The only reason you’d keep me around would be that the service I offer is pretty damn good. Would something like that appeal to you?
It’s based on NixOS to provide as much as possible out of the box and declaratively: https, SSO, LDAP, backups, ZFS w/ snapshots, etc.
It’s a competitor to cloud hosting because it packages Vaultwarden and Nextcloud to store most of your data. It does provide more services than that though, home assistant for example.
It’s a competitor to YUNoHost but IMO better (or aims to be) because you can use the building blocks provided by SelfHostBlocks to self-host any packages you want. It’s more of a library than a framework.
It’s a competitor to NAS but better because everything is open source.
It still requires the user to be technical but I’m working on removing that caveat. One of my goals is to allow to install it on your hardware without needing nix or touching the command line.
Love it! I've been thinking about this a lot lately. It's crazy how many great FOSS alternatives are out there to everything – and while they might be relatively easy to install for tech-people ("docker compose up"), they are still out of reach for non-tech people.
Also, so many of these selfhostable apps are web applications with a db, server and frontend, but for a lot of use cases (at least for me personally) you just use it on one machine and don't even need a "hosted" version or any kind of sync to another device. A completely local desktop program would suffice. For example I do personal accounting once a month on my computer – no need to have a web app running 24/7 somewhere else. I want to turn on the program, do my work, and then turn it off. While I can achieve that easily as a developer, most of the people can't.
There seems to be a huge misalignment (for lack of a better word) between the amount of high-quality selfhostable FOSS alternatives and the amount of people that can actually use them. I think we need more projects like yours, where the goal is to close that gap.
I will definitely try to use selfhostblocks for a few things and try to contribute, keep it up!
My guess as to why most apps are now a web UI on top of a DB is because it’s easy to “install”. SelfHostBlocks is admittedly geared towards a central server serving web apps. Or at least apps with a desktop or mobile component but geared towards synching to a central server.
Feel free to give it a try though, I’d love that! Also feel free to join the matrix channel UF you have any questions or just to get some updates.
> My guess as to why most apps are now a web UI on top of a DB is because it’s easy to “install”.
That plus web dev is trendy and everybody is learning it. I wouldn't know how to code a proper desktop app right now, I've not done it in years.
I don't want to criticize that or the centralization aspect – there will still be ways to put these centralized things on a PC for example.
I love that application. I plan to make some improvements to the web UI. I’d love to have multiple tabs with saved reports. That would allow my spouse to use it quite easily. I’ll be adding that at some point.
