If the arbitrary input is, say, a text box which takes a name and puts it into a database, it probably won't uncover any bugs.
It has some use if you build something like a complex pricing engine, numerical code or a parser for a mini DSL. I find that problems of this type don't crop up a lot though.
I've used it for things like "regardless of where you are on the page, tab n times and shift tab n times leaves you on the original item".
I found a bug in our tv ui library which was actually a bug in the spec. Regardless of how you built the ui, if you press a direction and focus moves, pressing the opposite direction takes you back - but we had another rule that broke this. We had tests for both, and it was only when I made the general test (for all ui, items in it and directions) it found the inconsistency.
It was also pretty easy to write.
I've also found issues around text processing due to lowercasing not always resulting in the same length string and more. I found a bug demoing pbt for a contact gig I was going for that was around some versioning.
To be honest I've never implemented it, even for a demo, and not found a bug.
> To be honest I've never implemented it, even for a demo, and not found a bug.
Me too. I tend to roll my own property testers / fuzzers per project instead of using a library. But my experience is similar to yours. Out of maybe 25 testers, I think the only times I didn’t find any bugs was when I messed up the tests themselves.
I used to build these too, my example was one in actionscript (always fun on restricted devices) and it was a bug in a library I wrote. I was actually testing the property tester and found the bug then, after realizing it wasn't a bug in the new test/tester.
> It’s incredibly humbling work
Absolutely. Glad others feel it too.
I almost feel like you need a decent reason why not to use them in places.
Ideally to can describe the behaviour of something in general terms around what stays true. If you can't, is it too hard for people to understand?
I will however say I don't usually write pbts but have a strong feeling I should add more.
It has some use if you build something like a complex pricing engine, numerical code or a parser for a mini DSL. I find that problems of this type don't crop up a lot though.