This is exactly correct. A work-sample test needs to capture the normal parameters of the work. If team members aren't routinely called on to solve programming problems in a high-stakes stick-the-landing-or-you're-fired exercise on a whiteboard in front of an audience, then the prediction the test is trying to make is confounded by all those factors.
* Eliminate live audiences
* Let people work in the environment they'll be able to choose on the job
* Ideally, let people work in their own comfortable environs, even if they won't be able to do that on the job
* If you're worried about cheating, build that assessment into your in-person followup interview
Originally he'd simply asked if I could be there whilst he did the test so that he could talk aloud and bounce ideas off of me as it was timed and he gets nervous in tests and was afraid he'd not think clearly.
He froze. Totally.
I took over and did the test for him, and he aced it and was offered the job at the top salary band.
He is a perfectly good engineer and the company were very satisfied with their hire, but he never did that test. The test in it's entirety was completed by myself.
I cannot imagine this is such a rare thing with remote technical tests.
I believe both of you, but our process did/does nothing to overtly catch cheaters, we hired directly off its conclusions, we hired at a rate faster than most VC-funded cash-flow-positive YC companies, including in SFBA, and we never let anyone go (nor did anyone ever quit while I was there) once we made an offer.
(Matasano is also not a company where it's easy to duck attention and coast; the tempo is 2-3 week engagements that wrap up with metrics that everyone cares deeply about).
The conclusion I draw is that cheating just isn't as big an issue as people think it is.
I've seen it several dozen times. A solution to one of our sample tests made it out to Github and that was all she wrote. For a while we kept the same sample test as a honeypot, but disqualifying half of the candidates for cheating was tiring.
* Eliminate live audiences
* Let people work in the environment they'll be able to choose on the job
* Ideally, let people work in their own comfortable environs, even if they won't be able to do that on the job
* If you're worried about cheating, build that assessment into your in-person followup interview