Hacker News new | past | comments | ask | show | jobs | submit login

Wouldn't a vendor tuning the benchmark be ... cheating?



Vendors are tuning their implementation (e.g. the schema, indexing, and queries used to implement the benchmark) and also their database system, but they cannot tune the benchmark workload itself. Apologies if this was ambiguous in my comment.

It is quite typical that vendors find performance issues in their systems when they first adopt the benchmark. This then leads to improvements in their operators, optimizer, storage, etc. These improvements will be included in a new release of the system – which is great, as a main point of the benchmark is to drive innovation and raise the general level of performance in the technology space.

It's important that the audits can only be conducted on systems that are generally available. We don't allow audits on some one-off version of a system that has special features just to perform well on the benchmark (these are known as "benchmark specials" and are also illegal in TPC benchmarks).


I've worked on similar testing process (but for filesystems): https://www.spec.org/sfs2014/ (and it has increments every so many years).

"Tuning the benchmark" is probably "tuning your system to score high on the benchmark". There are always caveats of course. You, as a developer, may be able to find some implicit assumptions in the test and exploit it to "game" the test. Suppose, for example, the test assumes that acknowledging deletion means deletion completed, but your code simply schedules a file for deletion instead of actually deleting it before acknowledge is sent.

You'd have to argue somehow that your system is behaving in such a way that scheduling and deleting are essentially the same thing which would allow you to pass the test, but that's about the level of "cheating" that you can do on these tests.

Also, by the nature of things, the way these systems are sold, you don't really want to pull these numbers out of nowhere. This kind of unsophisticated fraud will get you disqualified very quickly, but the investment into building such systems is usually on the scale of upwards from five or more years until it even gets into the stage where you can show something to the customer. So, it's a large investment that you don't want to risk.




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: