I've found it effective for anything that handles arbitrary input, especially from end-users. But if that data is coming from within your systems where you have full control over everything, less valuable.
It's useful for things that would be too painful to enforce via types. For example, "this list should be non-empty" is pretty easy to enforce via types (use a pair of head element and possibly-empty tail list). On the other hand, something like "map of non-overlapping date ranges to counts, with automatic splitting on insertion" would be pretty tricky. It's easier to state the desired properties as universally-quantified functions rather than types, at which point we might as well throw them into a few property checkers to see if they find anything we didn't think of.
The Choice Gradient Sampling algorithm shown in the paper could be used to integrate/exploit feedback (coverage, etc.).
Hypothesis can use a fuzzer as its input source, but it's marked as experimental.
It doesn't seem like "proper" fuzzers are used very often with Haskell (e.g. there's an example at https://www.tweag.io/blog/2023-06-15-ghc-libfuzzer/ ), though many people seem to have hooked up the HPC code coverage tool to property checkers, e.g.
If the arbitrary input is, say, a text box which takes a name and puts it into a database, it probably won't uncover any bugs.
It has some use if you build something like a complex pricing engine, numerical code or a parser for a mini DSL. I find that problems of this type don't crop up a lot though.
I've used it for things like "regardless of where you are on the page, tab n times and shift tab n times leaves you on the original item".
I found a bug in our tv ui library which was actually a bug in the spec. Regardless of how you built the ui, if you press a direction and focus moves, pressing the opposite direction takes you back - but we had another rule that broke this. We had tests for both, and it was only when I made the general test (for all ui, items in it and directions) it found the inconsistency.
It was also pretty easy to write.
I've also found issues around text processing due to lowercasing not always resulting in the same length string and more. I found a bug demoing pbt for a contact gig I was going for that was around some versioning.
To be honest I've never implemented it, even for a demo, and not found a bug.
> To be honest I've never implemented it, even for a demo, and not found a bug.
Me too. I tend to roll my own property testers / fuzzers per project instead of using a library. But my experience is similar to yours. Out of maybe 25 testers, I think the only times I didn’t find any bugs was when I messed up the tests themselves.
I used to build these too, my example was one in actionscript (always fun on restricted devices) and it was a bug in a library I wrote. I was actually testing the property tester and found the bug then, after realizing it wasn't a bug in the new test/tester.
> It’s incredibly humbling work
Absolutely. Glad others feel it too.
I almost feel like you need a decent reason why not to use them in places.
Ideally to can describe the behaviour of something in general terms around what stays true. If you can't, is it too hard for people to understand?
I will however say I don't usually write pbts but have a strong feeling I should add more.
