I see wonderful symmetry here. There are TrendyCos (hot startup unicorns), BigCos (established companies like google or amazon or microsoft) and UnknownCos (not in the limelight so there is not much information about them). Likewise there are TrendyDevs (hotshots who produce one heavily github-starred framework after another), BigCoDevs (multiple years of experience at one of the BigCos, probably were responsible for some important part of one of their numerous services) and UnknownDevs (been there done that, hard to say).
And here is the rub. There are certainly many undervalued gems among UnknownCos and UnknownDevs but also many abysmally bad workplaces and programmers. So you either have some inside knowledge about them (a referral, an acquaintance working at UnknownCo), have some magic method for separating the gems (like tptacek claims to possess) or it is just too risky to consider them.
There's no magic to it at all. Have every candidate work on programming problems related to the work you do at your company. Have every candidate work on the same problems, and let them do it from home. Build and iterate on a rubric for grading those challenges.
It is amazing to me that almost nobody does this, but: almost nobody does this. They have programmers write code on a whiteboard, or on some whiteboard-coding site; they have them do programming puzzles ("solve Towers of Hanoi non-recursively"), they have them talk about code, or, more likely, computer science trivia. They'll have them "pair off" with one of their own programmers and "fix a bug". They'll have them work as a 1099 for a month to see if they're any good.
In reality, most companies are trying to hire on trendiness; they want people from the right schools and/or the right cohort of companies. They're aggressively courting friends and colleagues of their existing team, and the special people get very different interviews than everyone else. The actual technical evaluation is mostly a facade.
My last job search wasn't typical (I thought of starting a consultancy, so didn't search for a job per se), but one before that showed to me that it's exactly what people do. A company that was looking for a guy to write mobile SDKs asked me to build an Android app with embedded webview and animate html stuff with JS based on gyro movement. A slot game company asked me to build a minimalistic slot game from scratch (thanks to them, I have a clean minimalistic and complete game on github now). Another game company asked me to build a small endless shooter — and I ended up being the only candidate who moved texture offsets instead of actually moving game objects to infinity.
Each of those tasks took from 4 to 8 hours, and 2 of those got me an offer. And of course, I used the same method when I consulted others on hiring decisions or hired people myself. So, based on personal experience, companies do that, and it works.
I'm glad that I'm starting to get feedback like this. I believe you that more companies are doing work sample tests.
For the the gigs that asked you to do these problems: did you also get interviewed with a standard programming interview? If you had to guess, between 1% and 100%, how much of the weight of your technical evaluation do you feel was on the sample challenges you did at home, versus Q&A and whiteboard coding on-site (or on the phone)?
Don't remember details about interviews, but I'd tell that homework was about 75%, because in those two cases they told me that I exceeded their expectations and they learned something new from my projects.
I agree that this is "better" than the alternative, but it can be absolutely exhausting for candidates actively searching for a job. I feel like it's recently become much, much more common (from my small-ish sample of me and some friends).
My issue with this approach is fourfold:
1. Most companies have no idea how to structure a problem that is both informative to them and also not abusive to the candidates time.
2. Companies generally do this right after the recruiter phone screen, which most likely doesn't give the candidate enough information to decide if the next steps are worth their time.
3. Most companies still do a whole suite of normal tech screens after you work on a take home problem.
4. If you're actively looking, getting a bunch of these over a short period of time is likely. I know during my full time search, more than 50% of companies had a take home test right after the recruiter screen. Most of these were 4-8 hours of work each, due within the week.
A lot of startups structure it more like hazing or a barrier to entry than an evaluation criteria. I have some fun (read: horrifying) anecdotes from my recent search that illustrate the problems above, but I don't think any of my points are surprising.
A nice alternative would have been to simply have one or two projects completed that are straightforward to evaluate and walk companies through them, letting them ask me questions.
Here's a really simple test for whether a work-sample scheme is effective, or just a bullshit trend-chasing afterthought:
Does the work sample test offset all or most of the unstructured tech evaluation the company would otherwise do?
If it does, that means they have a work-sample test rubric they believe in, and they're hiring confidently. If it doesn't, they don't believe in what they're doing, and the programming challenges they're assigning can thus reasonably be seen as more hoop-jumping.
In the latter case, it's up to you to decide whether a prospective job is worth extra hoop-jumping. Some teams are worth it!
I think that's fair. I've had both the former and the latter, but unfortunately most of my experiences fall into latter case, where it's simply been hoop jumping. Most of my friends (all about to graduate, so a good number of examples) are experiencing the same.
For example one company gave a problem with five parts, with the final part being solve longest path on a bipartite weighted graph (which is quite a hard and time consuming problem). After that, the next step was a phone technical screen, then an on-site with 4-5 more interviews, most being white-boarding. It was basically hazing instead of an evaluation criteria.
An alternative is my last job, which had a take home test that took about 6 hours, but that was the whole technical part of the process. Being on the other side reviewing them, the problem absolutely gave enough information.
I totally get there's a right way to do it, but like most interviewing trends, companies seem to just be adding this as a step instead of revamping their process.
Does the job they're interviewing involve finding the longest paths on weighted bipartite graphs? Or is this just non-recursive Towers of Hanoi pretending to be a realistic work sample?
No, the position most definitely had absolutely nothing to do with longest path or combinatorial optimization.
Anyway, my larger point is that what I've been seeing interviewing is that these tests are becoming much more common at US startups without companies removing/reducing the rest of their technical evaluation process, nor really structuring the problems to be a good signal.
In an ideal world where companies do take home tests right, I think its a great solution. But what I've been seeing more often than not doesn't support that, making it hard to support.
I'm really curious what you've been seeing at Starfighter. Are partnering companies still going on to do a full technical interview? Or does Starfighter largely replace their normal technical evaluation?
Ignoring the fun of the challenges themselves (which probably isn't entirely fair), the latter makes it very compelling for a candidate. The former does not.
Most of our partners have a somewhat abbreviated interview for our candidates, but everyone (as far as I know) still techs our candidates out.
I'm actually fine with that! We make no pretense of having designed a screening process that is appropriate for every job. What I'm less fine with is the fact that the norm, even for abbreviated tech-outs, is 7-8 hours of on-site whiteboard interview.
Maybe this is an US thing. I changed jobs last year and nearly every single recruitment process involved exactly that, after a remote interview them giving me a small project worth 2-5 hours of work and then going over what I produced one week later. This was for companies in central and northern Europe.
No, there's no principle, there's a ~20 minutes old idea that we engineers are so obviously awesome that companies should actually be paying us for the privilege of interviewing us.
There may exists engineers obviously awesome enough for that to be feasible, and great for them (but they are probably also obviously awesome enough to not have to go through coding tests, so it's a moot point) and even if I could perhaps (judging by certain recruiter-emails) scrape by as one of them, I certainly couldn't five years ago.
Back then, if I'd had to be good enough on paper alone to warrant not just being put through the recruitment process, but to be paid for it was well, I am not convinced I'd have been considered (and yes, I did a take-home test, and I aced it and it made up for my near-total lack of on-paper qualities).
Of course, companies shouldn't waste their applicants time with needlessly extensive tests, but there certainly exists no 'principle' by which you have a claim to be reimbursed for spending a few hours on an application.
Actually, for actors it does. If you go to a casting session, you should get payed for it. That's because they had a first chance to filter you out and they are not entitled to waste people's time. For the first filter, they use their CV and their reel (video of the actor showing itself).
I think that the same rules should apply for programming jobs. Just look at my resume and my publicly available code (that is linked from my resume). If you decide I am good enough to interview me, pay a reasonable hourly rate. This would stop the abusive practice of giving big problems to solve in our free time.
If this is true of programmers as well (and I have no opinion on whether it should be), it should be even more true of standard on-site job interviews, which are more disruptive to your work schedule and more demanding of your time and attention. But, of course, people do not generally get paid to go on job interviews.
It's at least reasonable to cover travel costs for interviewers. I was given a $40/day stipend for food and reimbursed gas and hotel when I interviewed at my current job.
This is a sound-bite, not something that holds up to scrutiny.
If you decide not to interview with possibly great places based on this, you're essentially saying that all future expected gain isn't worth some set hourly rate for 2-8 hours of your time.
Also, what's your free time worth? For me, sometimes it's worthless, and sometimes I feel like I'd sell my soul for 10 minutes to myself.
I have had a few of these the last year or two. Never been paid. More than half of them are a waste of time. I have refused quite a few of them.
The last "shouldn't take you more than two hours" exercise that I was given involved jumping through hoops to get set up on Instagrams API, only to discover that I was in a sandboxed mode, and needed my account approved before i could get anything more than metadata out). I didn't even get started coding, so I am not sure what exactly the test was meant to achieve.
I was not, at any of them. Yes, I was effectively giving them my time but it doesn't sound unreasonable when you compare it to the alternative: I'd spend that same time answering whatever obscure coding questions or far-fetched exercise they had but in a far less comfortable setting (and arguably less representative of my skills). Not to mention how I'd probably have to do the interview at a time they're working, instead of doing it on a snowy Sunday evening, at my discretion.
I'd so much rather do a project with my own tools and from the comfort of own home than commute several hours across the city for a several hour interview.
If all it required was an hour phone call and a small project, I'd totally do it for the right company.
Have every candidate work on the same problems, and
let them do it from home
To be fair, I've seen a lot of good arguments against programming assignments. I think at the end of the day, the employer needs to conduct some method of determining if the employee has the technical capabilities needed for the specific job they're being hired for. However, there are MANY other factors too like "how well they get along with the team" that need to be considered. That's what they're trying to determine with the pairings and other stuff you mentioned I think. Whether that works is another story.
Every team I've talked to starts out with some X-factor they think they need to assess for. It's my belief that if you can have the discipline to stop filtering for X-factors, you'll build better, more effective teams, because those factors are really just vectors for personal biases.
Regardless of whether you agree with me about that, I think we can all stipulate that if work-sample technical evaluations work (and: they do), most of what companies try to evaluate in on-site interviews is stupid. No part of working effectively with a team requires timed recall of how to implement a stable quicksort, or reversing a doubly linked list at a whiteboard.
At the very least, using work sample tests allows you to build an on-site interview process that honestly engages with "team fit" (or whatever your X-factor is).
My guess, though, is that when more teams adopt work samples and then go through the motions of trying to design a pure team-fit interview, they're going to realize --- once they don't have "implement Bellman-Ford on this whiteboard" to fall back on --- how unequipped they always were to evaluate team-fit in the first place.
The problem could also be that many teams simply don't have the time or possess the knowledge to do an assessment like you're suggesting (which I agree is a good way to assess candidates). It requires someone to actually design an assignment (which is a task that many coders might not be good at), and requires one or more people to evaluate it.
The ridiculous whiteboard coding of puzzles probably stems from laziness or inability to implement what you're suggesting. I know that personally, if someone asked me to developed a work sample evaluation for my job, it would take me many hours to come up with something, and frankly I even once complete we would have no way of knowing if it is actually a good predictor of whether or not someone is the right employee.
Hours? It could take a week, and if you're going to hire more than one person this quarter, it will still be worth it just in the time savings from not having developers deliver bad interviews.
Have you done this? My worry would be that sooner or later you'll have a candidate who posts the assignment online ruining your weeks of effort. It also seems like it makes it much easier to cheat if you make it a take home.
Yes, I ran a process like this for several years. I ran recruiting for the largest software security firm in the US; before that firm bought us, I used this process to more than double the size of my company. When I left, to work on a recruiting startup, not one of the people I'd overseen hiring had quit or been fired.
We paid the market median to new hires; we definitely didn't buy our way to that turnover (NCC pays better than a lot of early stage startups, but not better than late-stage ones).
No, I attribute the turnover to the recruiting mechanism. We found great people who were sorely mispriced by the market, and we took advantage of that to create a win-win scenario: people without the resume to get a similar job at a competing firm got an extremely impactful resume bullet and a good-paying job, and we got people who genuinely wanted to be on our team and weren't applying as a once-a-year lateral job-hopping gamble.
(I have no problem with people job-hopping, by the way, but every employer is trying to minimize their exposure to that.)
You were also recruiting for a "cool" sector where people would be learning marketable skills. Do you have any evidence that the same technique would work for more "boring" companies? (I ask because I'd like to convince my manager that creating a work-sample test wouldn't be a waste of time)
I agree completely. But in many organizations recruiting and hiring would be considered overhead instead of strategic, and thus a target for outsourcing to someone in HR or some outside firm. The problem there of course is that those people have neither the ability nor the inclination to create or evaluate a work sample challenge.
Really my comment was just to help demystify why this sort of thing doesn't happen in many companies based on my own observations.
If you substitute the candidate's time in an in person technical interview with the estimated time for the take home assignment, I can't see the issue. I would gladly give 3 hours of bullshit algorithm white boarding exercises for 3 hours of homework.
I love this approach. I'm also wondering what you think about giving well-designed work-sample tests over Google Hangout instead.
The candidate shares their screen with you, so you can watch as they solve the problem in their own dev environment. You can understand how they approach problems (quick and dirty, slow and methodical, lots of rewriting, etc), and you can ask questions at the end. You get a good sense for how they work as an engineer even before having to bring them onsite.
This seems to resolve the time asymmetry of take-home tests as well -- the interviewer spends as much time watching as the candidate spends working.
The only downside I can see is that you have to design your problems to take about an hour instead of the 2-5 hours you could imagine for a take-home test. But, you can break them up into multiple rounds, and give additional exercises to the candidates who do well on the first one, for no more total time cost than a collection of onsite interviews.
For what it's worth, I've done this at my startup and hired a great developer, and got very positive feedback about the process from the candidates I didn't end up hiring.
As someone who's had to do a few take home tests as well as coding with someone looking over my shoulder, I am definitely not a fan of the latter. I prefer to sit down and think about a problem, maybe even read it and let it sit in my head for a day or two before I actually start the coding. I also feel like shortening the time and making it synchronous would remove the opportunity to revise and polish the code I've written, which is one of the advantages of the take-home test format.
My ideal interview format would involve a work sample test, followed by a code review of said work sample, and also a reverse code review (candidate is given some code to review). As an added bonus, all of this can easily be done over email without wasting too much of anyone's time, especially compared to an all day in-person interview with lots of whiteboard time.
FWIW I presented this option to my senior management and HR at a "household name" software company and was essentially told we can't do this because it looks like a test and we'd have to do the same test for every candidate or else open our selves up to discrimination issues, no matter which team the position is for, and that was a no go.
1) HR (presumably taking direction from in house counsel) takes the position that it will be difficult to distinguish between a work sample test and an IQ test
2) We are a large organization with very broad roles. The Performance Team hiring for a Senior Software Engineer will probably want a different work sample than the Analytics Team...
It is simply not true that the Perf team must deliver the same work-sample test as the Analytics team.
Let's not dignify that argument. I absolutely believe you that your company has allowed HR to sabotage your hiring process, and that sucks. But let's not pretend HR is right to do it.
Ironically, a work-sample test which WAS the same across every developer would be closer to an IQ test than if that work sample test were tailored to the role.
There's probably a solid argument for giving every candidate for the same role the same work sample test; otherwise you might naturally want to randomize some parameters or choose from a bank (if you think "cheating" is likely).
The HR involved must be applying cargo cult rules of thumb without understanding the actual rules and the work the company does (which is, unfortunately, distressingly common for HR organizations.) If the actual work you did was such that a work sample would be indistinguishable from an IQ test (which it isn't, for almost any real work anywhere, and you'd have to be ignorant of either what the work is or what IQ tests are to make that mistake for most jobs) then IQ tests wouldn't be problematic in any case: IQ tests aren't specially prohibited in employment, they were just the immediate subject of one notable case which held that anything with a greater impact on a protected class is illegal if that impact is not warranted by the validity of the filter at issue as a measure of performance in the job for which it is used as a filter.
Not only that, but there are several very large companies that do in fact use IQ tests during screening (that's a stupid policy, for what it's worth), so I'm pretty dubious about the claim that IQ tests are unlawful.
They are lawful when you can demonstrate a link between on the job performance and the IQ test. Given how g-loaded software development this should not be hard to show.
I'm looking for a leadership role. Whether that happens at my current company (I'm an IC with a lot of technical leadership and mentorship responsibilities) or elsewhere only time will tell.
If you or anyone you know is interested I'm a software developer and spend a lot of my time on cloud architectures/containers/multi tenancy optimizations but just generally enjoy solving business problems with technology. In a past life I cofounded a startup that didn't turn out to be viable after going through an incubator (not YC). I have a resume, LinkedIn, Github etc.
> "we can't do this because it looks like a test and we'd have to do the same test for every candidate"
Oh, what a terrible tragedy, to give candidates consistent evaluations that are actually comparable between each other... /s
I get the point you're presenting, but at the end of the day it still boils down to "we can't use tests because then we'd have to be held accountable for objectivity, we prefer instead a system that lacks any kind of rigor such that any result may not be effectively challenged, officially or otherwise".
How much time do you give candidates to complete their work samples? I'm curious whether work-sample tests filter out, say, parents with young children, in which a company with a 4-6 hour technical interview might be preferred over an unbound take-home assignment (in addition to the possibility of more interviews).
>It is amazing to me that almost nobody does this, but: almost nobody does this.
There isn't enough feedback when someone has a bad idea on hiring. People will very quickly tell you if your code sucks though (this is a good thing), which might be the kind of feedback needed to get hiring changed.
I am fine with asking for candidates to work as a 1099 for a month if you pay them the standard 1099 contractor rate, which is almost invariably more than 2x their FTE rate. I am not fine with the industry norm of paying people their eventual FTE salary on a 1099 basis; that's a scam.
I think it's pretty dumb to do that, though, because most software developers can't take a month off of work from their current job to see if they're a fit for your company, and none of them will put their health insurance in jeopardy to do that.
I also think it's pretty silly to design a process that deliberately demands a month of time to make a decision that could be made effectively in a week.
That strikes me as a very costly way to build a team. Independent contractors are expensive; they know, better than most other developers, what their time is worth.
It shouldn't be on top of that! It wasn't for our process. Of course: we did in-person interviews. And in-person interviews are disruptive no matter what you do. But:
* Our in-person interviews were shorter than typical in-person interviews because they weren't tasked with fully teching candidates out.
* Because they didn't try to tech candidates out, they weren't as stressful, and so were less draining and unpleasant.
* Because our in-person interviews were entirely scripted (the interviewers had very little latitude with what they could ask and how they could ask it --- which they hated, by the way), they were easy for everyone to deliver.
* Before candidates arrived to their interview, they already knew (because we told them!) that they were likely to receive an offer from us based on their performance on the work-sample tests.
I understand why people resist the idea of coding challenges, given:
* They're not going to get feedback for weeks, months, or maybe ever
* The challenge is going to be graded pass-fail, or whimsically, without rigor or consistency
* They're going to have to do the exact same grueling bullshit draining dehumanizing programmer interview anyways
* They're not going to see the challenges coming, or, for that matter, know exactly what happens next after the challenges are done
That process is super common, even at companies that ostensibly do challenges. It sucks. I agree, no company can really get away with this in the long term.
But those are also chickenshit problems. Switching from unstructured interviews to work-samples is hard: you have to change your mindset on how to qualify candidates, and there's a leap of faith involved. But getting candidates feedback, telling them what to expect with your process, keeping a schedule, and having processes in place to be consistent aren't hard problems. They are table-stakes common sense business execution, and if your team can't handle that right now, your team is mismanaged.
Can you share an example of challenge given to candidates? In the past, I've used "script a file sharing tool that handles encryption", which worked well and took candidates ~4/8 hours to complete, but isn't something we used directly in our projects.
Sure: we wrote a very simple retail trading system as a one-file Ruby app. It used a custom binary protocol. We had candidates reverse the binary protocol, implement enough of it to test the system, and find vulnerabilities in it.
We had a grading rubric for the challenge that was based on the kinds of things people found in the application. We evaluated performance on the challenge on a 1-5 score; depending on how you did on other challenges, a 3 would keep you in the game, and a 4-5 probably assured an in-person interview.
It saved us enormous amounts of time at Matasano. When we designed our work-sample rubric, our #1 concern was filtering out people who interviewed well and/or had great resumes but weren't worth a billable hour on a real project, but our #2 concern was interviewing faster and getting results back to candidates quickly. (We didn't discover the best reason to do work-samples --- discovering hidden talent --- until after we'd started doing them).
With standardized interviews and challenges, our whole hiring pipeline became entirely mechanical. We could send our a challenge and its instructions in 2 minutes, and evaluate the response the next day (or week, or whatever) later in less than 5 minutes. 7 minutes, to produce a technical evaluation that crushed a competing battery of tech interviews by several senior staff members that took 4-5 hours to do.
Aha! I sort of like that, then. Assuming of course that it actually does work.
Back when things were simpler, we had a multiple guess questionnaire that did the similar. For onsites, we then looked for evidence of rigor and thoroughness, the ability to estimate how much communication was required, had they deployed real systems?
You understand that it sounds like you "fixed" hiring, and so we're all sorta skeptical, right? If you could bottle that, there's your billion.
I am talking my book in the sense that my business works a lot better if companies get saner about how they hire people, but Starfighter is not the commercialization of the idea I'm talking about here.
Yes: like I said, the goal was to get results back to candidates faster.
The biggest complaint we'd been getting from candidates before work samples was that our process took too long. Our goal became to get the entire process done under two weeks (from its worst point, a month or two earlier, of 1.5 months, when someone finally posted an anonymous complaint to Glassdoor about us).
After iterating a couple times, we could reliably go from first contact to hire/no hire, with a motivated candidates (ie: one who would work with us to schedule the interview process aggressively) inside of a single week.
As a candidate, I would be a little wary of this process. You're asking me to put in multiple hours of work, but you only put in 7 minutes. So the incentives are misaligned: you are incented to give these challenges to many people, even if they have a low chance of passing through. But as a candidate, I don't want to sped multiple hours if I have a low chance of passing through. This is one reason why full-day on-site interviews aren't so bad - if I've gotten to that stage, I'm probably pretty likely to get an offer.
First, I don't concede the idea that this is a concern I need to ameliorate, because the work-sample process consumes fewer hours of candidate time than the conventional interview does, and, better still, consumes those times as, when, and where the candidate chooses to make them available: an hour a night during the week, say. Do the work from your favorite quiet bar. Do it during your coffee break.
I am spending a lot of time these days talking to people interviewing in the valley, and what I'm seeing is that the norm candidates are subjected to is 7+ hours of 6-7 on-site interviews. Candidates have to go through all the interviews, even if the first interview has effectively ejected them from the process.
Compared to that horrid process, I don't believe I have to justify anything about my process.
But, if you read downthread, you'll see that we in fact did a lot to ameliorate the (bogus, I think) concern that we were incentivized to soak up hopeless effort from lots of people.
I was thinking something similar. When one party can waste the other party's time at little cost to their own, the situation can be abused. If a job ad wants you to submit to a test before you even talk to a hiring manager, this is a signal that the employer doesn't care about wasting your time. I can see the advantages to automating the hiring process but as a candidate I am less inclined to engage with a party that has no "skin in the game".
Think of the design world:
Potential client "A" asks multiple desperate artists to work on spec in hopes that they will get the commission.
Potential client "B" call you up and talks to you, sends you some napkin sketches and generally engages with you for an hour before asking you to do a design.
Both clients want you to do a design (test) but one clearly doesn't have any skin in the game.
Assuming you're a decent designer, which client do you respond to?
Its a crap waste of time. Take home exercises have little more in common with the job than a white board exercises. And not all devs are going to do half a days work for free.
People say this in every thread about hiring, and it never makes any sense to me at all.
Companies that don't hire using objective at-home tests hire instead with grueling on-site interviews that knock out an entire business day (most of them last a whole business day, some recent interviews peers have gone on have taken multiple business days, and all of them at the very least kill the day for anything else). The latter is obviously worse than the former; I don't even see how any other argument could be colorable.
What I'm beginning to think is that this kind of pushback comes mostly from developers who are well-connected, and so they never experience the grind that less well-connected but equally-capable developers do with interviews.
Yes: if you're at a point in your career where you can get a job in any of 6-7 different companies just by raising your hand, saying "I'm available", and having a 30 minute conversation with the VP/Engineering who you worked with 2 jobs ago, everything is a waste of time. I am not here to tell you that you should make things harder for yourself.
If used wisely, at-home coding tests should be far better than the average mess of a technical interview.
The problem is in the hands of TrendCo, they can be used crudely to find the trendy hire who has the exact same ideas about engineering as the hiring person. Because giving the test costs TrendCo nothing, they are happy to throw people at it until they randomly find someone who is exactly what they are looking for.
Anecdotally, I and 2 people I later befriended applied for the same role. We each spent 4 hours on the task, all 3 of us strong coders who made a solution that would be suitable for any startup, all 3 taking different approaches based on our styles. We were all rejected, because none of the 3 of us hit upon the exact approach the company was hoping for, but didn't ask for. There was no opportunity to ask for this feedback either. And it took them nearly a month to bother looking at my code (even after I called).
Since then, I won't take these kind of tests unless I have reason to believe they are being given in good faith as a way to determine if a programmer is capable, not as a way to find their perfect ideal of a programmer.
If you had asked at Matasano how we evaluated our work-sample tests, we'd have told you. In fact: the first contact any new candidate had at Matasano was a 45-minute-long phone call with a director+ (for a year and a half: me), during which we explained our process in excruciating detail and answered any questions that came up. If we got even a hint from a candidate that they might not know exactly what we were looking for, we'd get their mailing address and fedex them a stack of free books; we gave them cheat-sheets on what parts of those books to read, as well.
I have the strong impression that a typical SFBA tech hiring manager thinks it's crazy to give candidates cheat sheets for interviews, and that helping candidates with technical evaluations decreases rigor. THE EXACT OPPOSITE THING IS TRUE. Evaluating technical ability under unrealistic pressure and with unrealistic restrictions confounds the results of the evaluation.
Of course, you have to design tests that are valid predictors for candidates who have received tips and assistance and practice and resources. That sounds hard! But it isn't. The best predictor for a candidate's ability to do the kind of work you really do is, simply, the kind of work you really do. Not problems from a book, not trivia questions, not one algorithm poorly implemented on a shared coding site and a Skype call, not whatever bug is at the top of the bug tracker today, but a sample assignment given to a real member of your team, the same one, for every candidate, graded the same way.
I completely agree with all of that! If only more hiring managers were so thoughtful. Now, I'd only do this kind of at-home work if I can talk to someone who will explain how their hiring processes works, and I get the sort of good vibes that your post gives.
The problem is that a lot of companies put out the take home test and then expect a long on-site as well. One or the other would be fine but take home tests are easy for the company to send out a lot don't take them that seriously.
And here is the rub. There are certainly many undervalued gems among UnknownCos and UnknownDevs but also many abysmally bad workplaces and programmers. So you either have some inside knowledge about them (a referral, an acquaintance working at UnknownCo), have some magic method for separating the gems (like tptacek claims to possess) or it is just too risky to consider them.