One of the things I learned in the classical music world is that the things we ask people to do in auditions–play the hardest parts of all the most difficult orchestral music without any context–is really creating a new problem that doesn't need to exist.
And in my experience listening to auditions over the last several decades, what you have now is a bunch of musicians who can only do what's on the test. When you try to plug them into an orchestra, they are completely fucked.
My experience interviewing and hiring tech people is that this is one of the worst possible approaches. Anyone who needed to could figure this out. It's not relevant. What you need to figure out in an interview is if the person is curious enough to find the answer.
I don't ask for code or algos in my interviews. I ask about the person. What's your favorite programming language? Why is it your favorite? What do you not like about it?
What's the project you are most proud of? What was hard about it? What made you happy? What did you find challenging about that project?
These are 100% totally humane questions that tell you everything you could possibly need to know about a candidate in about a half hour. Maybe less.
If you can't figure out if this person is technically competent from this set of questions then you aren't competent to interview or hire.
Interviews don't have to be only about how good you are at interviews. They could be about trying to understand if the person would be good at the job and fit in well with the team.
I think your approach is fine, but I should mention that it will, like any interview approach, misfire with some people.
Episodic recall and emotional recognition are both talents that vary across the population. If I want to do well at an interview like this, I will have to spend a bunch of time in advance refreshing my memory of various projects I have worked on so that I can simulate a neurotypical response to those questions. If I don't, I may just stare at you nervously and break out in a sweat when I realize I don't have anything filed under, "project I am most proud of". Or that pulling up the narratively engaging details of a project I haven't thought about in years is a process that will take me hours of specific effort. And it's not just me; I've hired great developers who were also terrible at "humane" interview questions but who did great when we just sat down and coded together.
This very thing happened to me recently in an interview. This was my first interview after not having actively interviewed for a few years now, so I was quite rusty. I was asked what some issues were that I worked on at an initial phase of the project and I blanked the fuck out and then instantly went into self-scrutiny and sweating as I spiralled into "oh crap I should really know this" and "this is making me look stupid" and wondering how I was coming across to the interviewer. I have some anxiety problems, that's there too - but point is that if you're not really prepared with answers to questions like these, it can seriously throw you off mid-interview.
And lots of good developers skew anxious! What in normal life counts as being overly careful or or overly sensitive is just good practice in coding.
That's why I think it's especially important to make interviews as low-anxiety as possible. They'll always be worse than the job, of course, but the closer you get them, the more you're likely to see how people will really perform on the job.
I too have to prepare for such questions. I thought everyone else also did.
Not only do I not remember past successes very well, preparing for these questions is literally the first time I asked myself what I'm proud of. I didn't even consider doing this and reject it as irrelevant before. It just never came up.
I have arbitrarily terrible episodic memory, especially so for professional stuff. If I don't have the commit history to check, I have trouble describing what I did in the last month in anything like a lucid level of detail.
I've always had trouble with the kind of behavioral interviews that are "Tell me about a time when . . ." since I can often recognize that I've been in relevant situations but can't for the life of me remember the details (even with time to prepare).
This honestly makes me more nervous about interviews than leetcode shenanigans. At least with those, I know how to improve.
(At work I rely on a combination of having the code in front of me, written notes, and whatever project management tools we have so it's not much of an issue, but most of that isn't available to me in an interview setting.)
I agree this is true, but I think that's in contradiction to the notion that one can just ask some breezy "totally humane questions" and then make optimal hiring decisions. The more separate the domain, and the more preparation helps, the more you're testing for something pretty different than the work.
You don't make "optimal" hiring decisions by just talking to someone. I would say whether your hiring decision was optimal can only be judged in hindsight. Hiring is getting a limited pool of people and filtering down to find the ones that fit the role most.
In the end all you could actually do is guesstimating who would do best at the job. And sometimes the answer is: "none of the ones who showed up".
If you are a good engineer or programmer yourself you can tell a lot about a person by talking to them for an hour. You can learn what is important to them in the craft, how they deal with criticism, what they aspire to, what kind of problems they tend to work on, if they are more autodidactic or more influenced by other's opinions and so on. This is all knowledge directly inpacting the question whether they are the right person for the job.
And who the right person is depends on the job, so there might not be "right" answers to the questions. For a very niche database job you might actually want someone who is very accurate, very in the detail and very focused. For other jobs maybe some entirely different traits are better.
Of course the problem here is that the people you hire could just be very good actors or liars who cannot do the things they say, so a little technical testing might also be needed.
I think the issue is that, and I'm paraphrasing something which someone else said but which I think reflects what I've seen, nobody (at least in larger companies) wants to be responsible for a bad hiring decision.
While yes, you can get someone who has a good track record of being a good judge of character to make hiring decisions. If that decision goes wrong and the blame game starts then you inevitably ask this person to explain their hiring decision. This person will then reply: well it was my opinion that X, Y and Z.
In the alternative case where you just get someone to follow a rigid process you can answer: Well he scored well on this exercise and the opinion of the team was in his favour and the work history ticked the right boxes. This effectively dilutes the responsibility. You can no longer blame one person for the bad hire.
Lastly there's also the diversity aspect. With companies increasingly being scrutinized for their hiring practices, "it was my opinion" just gets translated to "this person's internal biases guided the hiring process and as such it was not fair."
I agree this is all making hiring much harder than it needs to be. And certainly you can still find companies where the hiring decisions are made on highly experienced and well honed gut instinct (its the only kind of interview I have been a part of) but it also makes sense how in our increasingly overcomplicated world we are developing increasingly overcomplicated hiring processes.
My recommendation is if you want a sensible hiring process (with maybe a telephone screen followed up by one normal interview) you should stick to smaller and less corporate companies.
Unfortunately, "gut instinct" is just another phrase for "unconscious bias". Sometimes those biases are helpful, some less so. And some can result in hiring results that are unfair enough to invite lawsuits.
I know plenty of smaller, less corporate companies that still use careful hiring processes with thoughtful rubrics. They do that not because of risk avoidance; they are generally still ok firing people who don't work out. They do it because they want to run a fair process while maximizing ROI.
Out of curiousity, did your high school or college teach you much about interview prep?
Both of mine did. Much of their advice was irrelevant (nobody in tech gives a shit about your suit or how firm your handshake is) but they both stressed the importance of preparing in advance to show how good your skills are; and giving thought to cliche lines of discussion like 'what's your biggest weakness?'
I love the comparison to music auditions, and I mostly agree with the advice.
I would still require the candidate however to write some code. Something rather simple, not leetcode, ok, not fizzbuzz either, and something that proves their familiarity with at least one language and platform.
I should be a GM by now simply watching chess analysis videos on Youtube, learning the names of some openings etc. I can talk about it all day long. But since I don't play at all (I'm not interested) when I actually do play on occasions I revert to 500 ELO...
You can easily tell a seasoned veteran from a rookie just by looking at them code for a few minutes, working on a simple task. If you can't, you have no business in interviewing others :)
Aesthetic judgements, and the ability to justify and defend them, are very important, but not the only important thing IMO.
Someone who can explain why they love Rust and hate Python (or vice versa) is a convincing candidate but I think you need to see a couple of other things from them. In addition, sometimes you just don't hit on the right question to get evidence of their aesthetic judgement. (Like the candidate who doesn't care about Rust vs Python, but who hates Postgres and loves MariaDB..)
From what I've seen, getting a regular job in classical music is difficulty and pressure on a level few of us can imagine. Might be easier to get tenure at Harvard than a rostered spot in an orchestra.
I think the odds of getting a high-profile orchestra slot are far lower than that.
Source: Multiple orchestra musicians and management speaking about the audition process and the now-defunct myauditions.com website
Process:
Show early talent
Get the right instruction
Get into a preparatory program
Make it into a conservatory
Do well once there
Get a spot in an ensemble, probably lesser-known (not easy) and establish a reputation
See an audition notice with repertoire to prepare, apply
Spend a few weeks or months intensively practicing (on top of other commitments)
Maybe get an audition, realizing the Music Director may be promoting his buddy past these rounds
Travel at your own expense to the audition
Play demanding repertoire behind a screen and respond to any directions
Most likely get rejected
Play in any following rounds
If not rejected, maybe you get a qualifying week rehearsing with your potential colleagues and playing a couple of concerts when the Music Director is in town
If you beat out the couple of stellar artists who've also gotten that far, you may get an offer
Or, the MD's buddy may get the offer
Or, there may be a no-hire
If you get and accept the offer, you are "on approval" for a couple of years at which point there's the Tenure Decision
And in my experience listening to auditions over the last several decades, what you have now is a bunch of musicians who can only do what's on the test. When you try to plug them into an orchestra, they are completely fucked.
My experience interviewing and hiring tech people is that this is one of the worst possible approaches. Anyone who needed to could figure this out. It's not relevant. What you need to figure out in an interview is if the person is curious enough to find the answer.
I don't ask for code or algos in my interviews. I ask about the person. What's your favorite programming language? Why is it your favorite? What do you not like about it?
What's the project you are most proud of? What was hard about it? What made you happy? What did you find challenging about that project?
These are 100% totally humane questions that tell you everything you could possibly need to know about a candidate in about a half hour. Maybe less.
If you can't figure out if this person is technically competent from this set of questions then you aren't competent to interview or hire.
Interviews don't have to be only about how good you are at interviews. They could be about trying to understand if the person would be good at the job and fit in well with the team.