We (software hiring managers in general) have an interview anti-pattern that revolves around canned questions, gimmicky coding exercises, white boarding, etc. It’s not typically effective, it’s not realistic, and it’s not useful for evaluating a persons potential.

If you’re a manager of a team you’ve likely played the 9-box exercise, where-in you examine each of your team members abilities and potential, and fit them into a spot in the grid. It’s not ideal but it gives you a starting point for self-evaluation of your team. The same checklist box-fitting exercise that we do with potential candidates ought to be the same, a starting point.

Where do you start? Well, obviously if we’re hiring an engineering role, then some sort of engineering experience is required. This is probably my most flexible criteria, you can find great technical people from all kinds of organizations, straight from university, or with 20 years of self-taught, self-employment. How you evaluate them is a well worn path, but for the record we use Hackerrank for our initial screening and some phone screen/on-site real-world exercises (nothing crazy) and it’s been working well. This area of evaluation is a math equation. There is really one way to solve for x, and the person either has it or they don’t. But people are far more complex and aren’t so easily boiled down to a binary yes/no.

So they tick the technical box, great! Now, what do you actually need the person to do day-to-day? For my quality engineers, they need to be independent problem solvers that work well with diverse teams of dev’s, PM’s, designers, so a personality helps. Communication skills are key. And the ability to bend and flex as you empathize with the challenges the teams your are supporting are facing is huge. Supporting being a key word in that sentence. They then need to work out appropriate test plans and approaches, take care of planning and coordination, and then finally do the technical work to validate the project (scripts, tools, tests, CI integration, etc.).

So we can start a checklist for our starting point (cause we love those):

  • Communication
  • Empathy
  • Emotional Intelligence
  • Attention to detail
  • Organizational skills

Finding an engineer really isn’t that hard, especially in the valley (okay, it’s hard, but the more specialties you look for the harder it gets), finding an engineer who relates and works well with diverse groups and understands human nature, that’s a bit harder, and that’s why I care more about interviewing for soft skills than checking some technical boxes and writing code on a white board.

That’s not to say those things aren’t important, but I think you can generally get a good enough understanding of technical abilities from a) job history walk through, spending a few minutes probing through a resume asking the candidate what they did, in detail, b) technical discussions about some of the tools/languages/whatever you use on the job. If you’re the interviewer, you have to push further than one question deep to start to see the limits of a persons knowledge. You can get an idea for how well they know their stuff based on how comfortably, quickly and passionately (and correctly, obviously) they answer. These kinds of questions are also way easier on the candidate than having them stand at a white board… something they will likely never do on the job. They will however have frequent deep technical discussions and work with many people to form solutions.

How do you qualify candidates against these kinds of skills? In the Quality Engineer case, maybe give them a project on-boarding meeting, where they need to ask questions of the team working on the project on what and how its being built, devise a high level test plan and identify appropriate dates, challenges, etc. Essentially a normal kickoff meeting with the candidate about a real project (past or present).

Or, have a conversation about a particular subject (a leading question might be “what’s a new technology you’ve used recently that excited you?”), but be analyzing the candidates responses, demeanor, body language, and try to understand how they think and how they approach their work.

To bring it all back (using the 9-box conceit), you’re assessing a candidates potential, not a candidates performance. You can’t assess performance in a single interview, or even within a single day. Performance assessment takes months to be accurate, and that’s what the 90-day probation period most companies have is for. Assessing potential requires understanding who a person is and how they approach situations, their personal standards, and their leadership ability or capacity for independent thoughts. A base line technical ability screen ensures you have a solid floor, but what you really want to know is the person’s ceiling. Approach your interviews more in this manner and you might find you get better people - finding good people is what matters.