It's a programming job. Matasano wasn't a netpen firm; every project involved us digesting and processing some giant blob of other people's code. Everyone on the team had to write tooling and, more importantly, quickly understand other people's code well enough to outguess the authors.

I don't accept the argument that we were hiring for an easier job than "programmer"; I think we were looking for a subset of the population that ordinary software development teams look for.

The bored and under-exercised VB programmer, by the way, helped found the NCC Cryptography practice, and moved on to one of the greatest crypto software development jobs in the industry. It's not my place to say exactly which job, but if you make a mental list of the 5 coolest crypto software development jobs that might exist, it's probably one of those five.

The problem is everyone else runs hiring processes that assume the bored VB programmers are just VB programmers. They are wrong. Joel Spolsky was also wrong about this (despite the other nice things he said about VB).

> I think we were looking for a subset of the population that ordinary software development teams look for.

I think that's actually why your experience might be less representative, but I'm open to being corrected.

Security is generally considered harder than regular development, at least to the extent that it would typically be hard to get a security job without security experience.

From what I understand, your model was to take programmers with no security experience and train them on security. The effectiveness of this doesn't surprise me at all, because it's fundamentally a different problem than taking someone who's bad at programming and teaching them to program better.

I have absolutely no problem with hiring someone who's a good VB programmer and teaching them to do web programming. What people generally dispute is whether you can hire someone who's a bad programmer in general and "train" them to be a good programmer.

> I have absolutely no problem with hiring someone who's a good VB programmer and teaching them to do web programming. What people generally dispute is whether you can hire someone who's a bad programmer in general and "train" them to be a good programmer.

How do you know if a programmer who is in a field you are not familiar with is good or not?

You can ask someone to show you some code they've written and teach you about it. Concepts transcend language and field: if you understand what you're doing and why you're doing it when writing a VB program, I have high confidence that with a bit of time you'll also learn to understand how and why to write other kinds of programs effectively as well.

Why does it have to be this complicated? Explain some of their existing code? Why not just give them a representative programming problem that your team has already solved, and ask them to code up their ideal solution? That's called "work sample hiring", and it's the gold standard for candidate evaluation.

It boggles my mind that everyone doesn't do this.

Frankly, your statements are extremely conflicting. Pure work sample hiring is in direct contradiction with hiring trainable people.

I do not expect a VB programmer to be able to code up a single-page React app during an interview, even though building a React app would be a representative problem. Do you?

So which one is it? Hire exclusively through work samples, or hire trainable people? I mean this seriously: how do you reconcile these two? You're a strong proponent of work sample hiring but also of training—do you expect people to come in and do a security audit for their interview, even when one of your hiring goals is to hire people with no security experience?

I would personally much rather hire someone who is a strong overall programmer (but might very well not be able to code a JavaScript app in an interview) over someone who knows a little JavaScript (so they can do a simple problem) but has relatively shallow abilities.

> That's called "work sample hiring", and it's the gold standard for candidate evaluation.

Having a discussion about someone's existing work is, in fact, another gold standard of hiring and is quite similar to having someone do the work on the job. It's not quite as good as having someone execute work, but it's the next best thing—especially when you're trying to hire people who don't have direct domain experience.

Hire through work samples. I think you're going to be surprised at who does well with them.

I think you're neglecting to remind folks that Matasano would give people a crypto textbook to study from first.

So if a VB programmer were applying for a React job, you'd say, "here's a React book; we're going to have you code a React component during your interview. Let us know when you're ready to schedule."

That's true, but how much credit do I want to give myself for extremely obvious hiring process flourishes? Yes: give candidates study material!

So how is your process really any better than Google's? They will also tell you exactly what to study and give you reference links + time to study it.

Google does not hire with a work-sample test. tptacek's claimed process is therefore easy to distinguish from Google's. Maybe better, maybe worse.

Google does allocate non-trivial time on training, though, both on the candidate's time before the interview (as you note), and on paid time after hiring.

What a low effort reply. Honestly, I expected better from you. This is pure argument from authority without any attempt to confront the fact that there is in fact a very real contradiction between hiring through work samples and investing in training.

For the record, I have done work sample hiring. It's definitely my preferred strategy, but it's also far from a panacea.

But it's also equally true that if you give someone a work sample to build a React app and their entire experience is in backend programming, they will do not very well. Would you really be surprised if someone fails to produce good work in an environment, language, and domain completely foreign to them?

Taken to the extreme, some companies traditionally hired engineers who had absolutely no programming experience and trained them to program on the job. Would your advice to them also be to "hire through work samples (of programming)?" How would candidates who, by definition, need training to even do the work be able to produce a work sample?

Your whole "I know best, so I don't have to provide logic or evidence" schtick is really getting old.

There's no contradiction. Set the bar where you need it to be, but make the bar about ability and aptitude, not about people talking their way through an interview.

If your bar needs to be super hard, set the bar there. If you need people to understand in detail when and how to use shoudComponentUpdate and how to write a pure functional component, do that. I'm not arguing that React shops should hire people who suck at React.

I think you think I'm saying something I'm not saying.

> I'm not arguing that React shops should hire people who suck at React.

I guess I misinterpreted that you were a proponent of training people.

Yes, if you want to exclusively hire people who are already good at React then of course work samples are an effective technique. I personally think the majority of good hires are not already familiar with my particular stack.

> Set the bar where you need it to be, but make the bar about ability and aptitude, not about people talking their way through an interview.

I think you might have misinterpreted my position as well. When I ask people to talk their way through their existing projects, it means they literally bring some code with them and we spend the interview working through it and them teaching me about the decisions they made in building it. You should try it some time.

Also, it's easy enough for a VB programmer to get started with React, he/she can quickly teach himself/herself some js and then follow a tutorial online, and after some point build a toy application with React. And that can be the work sample. It won't be a production ready app, but it will be fine for a junior React guy.

This perspective depends on mentorship quality being constant across all of these programmers.

Then mentorship and teaching need to be taught also.

In the startup world? I'm not holding my breath.

Yep, you'll have to do more work than just breathing. :-P

Oh, I agree about VB; once upon a time, a long time ago, I too was a VB programmer (before I discovered Turbo Pascal and then became aware that I found the distinction between Let and Set disturbing; I ended up loving Pascal so much I spent 5 years at Borland on the Delphi compiler).

While at Borland, another engineer (Allen Bauer) shared his personal theory, that some engineers were great at reading code, more were great at writing code, and not so many were great at both. Sounds like your job required people that were very good at reading code. It intuitively feels very testable.