We have the same belief in the limited usefulness of resumes at Triplebyte. We found talking about projects with candidates both more enjoyable and interesting than looking at words on a resume. Especially when people are enthusiastic about what they built.
The difficulty we had was not seeing a strong correlation between talking about projects and doing well at programming during an interview.
What is your programming interview like? If it's whiteboarding algorithms, it's likely to be eliminating a lot of good people who get nervous, and selecting for people who are good at studying interview questions and coding under pressure in an artificial environment. There is good evidence it doesn't predict job performance well, see: http://www.wired.com/2015/04/hire-like-google/ and https://twitter.com/mxcl/status/608682016205344768
We completely agree. We do all of our interviews via screen share, allowing candidates to use whichever language and environment they're most comfortable with. We also give a selection of problems, one of which is algorithm based, allowing candidates to choose which they prefer.
Ultimately though people are aware they're being watched and assessed under timed conditions so it's going to be somewhat stressful. If we think someone is so nervous they're clearly not able to code at all, we'll offer a take home test as well before making a final decision.
As a matter of interest, does your company also ask managers to go through a management test as part of the hiring process? Let them run a team for twenty minutes or an hour and evaluate them on that basis? If not, why not? :-)
I think "take-home" style tests are the only reasonable way to evaluate this kind of technical competency. They are the closest to realistic. Throw people a small problem of the kind they'll actually be asked to work on. Let 'em deal with it--including documenting their solution!
I think the other thing to look into is what "talking about projects" means. I'm willing to bet that there's some version of talking about projects that correlates well with programming ability. I've had to screen ~ 500 engineers and we basically use the same method aline describes as a first filter and then talk about the project. Imo it's other things like reliability, speed in real world conditions etc that are a bit harder to verify but those are hard to verify in a programming interview as well.
Not the GP, but I focus on tradeoffs and roads not taken. Get them to talk about a project they've worked on, and then ask about alternative design decisions around some interesting feature and see what they say.
Example: I developed a little state-machine framework for managing complexity in a large, legacy code-base. It allowed me to refactor a lot of ad hoc distributed logic into the transition table and clean up a lot of weird corner cases that made the code fragile and difficult to change.
Questions might include: "Why did you write your own rather than use an existing state machine framework like the one in boost?" (for C++ frameworks there's pretty much always one in boost, so even if you don't know anything about the area you can throw this in for fun and see what they say). Also: "Why a state machine rather than some other approach to refactoring?" And so on. This process gets at taste and good judgement, it gives you a sense of how tolerant of alternatives they are, and so on.
Additional edit: one of the things I look for in answers is people who say, "Yeah, that particular decision might have been a mistake... I always wondered what would have happened if instead I had..." Good developers are able to admit that not everything they do is perfect, and are willing to give alternative views a bit of credence.
Less an explicit set of questions but more recursively/iteratively asking why/what/when/how at increasingly deeper levels of the problem they're talking about. It requires the interviewer to be at least familiar with the general domain but you don't have to be an expert. It's borrowed from how Elon Musk says he does assessments. People who know their stuff remember everything about every part of something they worked on. They know why they did /didn't do something all the way down to the most trivial level of detail and they have intelligent thoughts about how they would improve and what tradeoffs they made. Most people fail after one or two levels of this "interrogation".
I like this style of interviewing as well. What throws me still is when I'm talking to a candidate with only large corp, individual contributor engineering experience. Their answer for why is almost always some variant on "that was the spec".
What do you do with candidates like this besides weed them out at the resume review stage?
Your interviewing process is broken when it comes to those with line-of-business backgrounds. The type of work those people do can sound like "project management", when in fact it is evidence of the "most valuable" programming skill, organizing complexity [1]. Reducing reliance on resumes is less useful when the same "someone like me" and algorithm-knowledge biases still seep through.
I do wish more people would do this. I've written about a number of times but I'm afraid its not all that popular. People feel a strange aversion to just talking.
I've been enjoying reading the TripleByte posts about what you all have been learning while trying to build a better hiring funnel. (Thank you for sharing!)
Have you been able to place anyone yet? I'd love to know out of the 300 or so interviews you've done, how many have led to accepted offers, and what those successes had in common.
The difficulty we had was not seeing a strong correlation between talking about projects and doing well at programming during an interview.