What is this post about interviewing doing taking up valuable screen real estate on an engineering blog? Shouldn’t it be filled with discussions of cool new technology, code snippets, or coding how-to guides? Interviewing, like many things, is a problem that is best solved with an engineering approach. Since that’s the approach we have taken, our engineering blog seems like a good place to talk not only about what our process looks like, but about what it’s trying to accomplish and why. Who knows? Maybe you’ll even find it interesting!

Background

Interviewing is, in general, pretty low on the engineering priority list - it’s necessary, but seen as something that takes away from ‘real’ work like actually shipping code. It’s also an activity that lies at the intersection of social interaction, impartial judgment, and excellent technical ability. That combination means that it is squarely in an area that many software engineers are weak in, but which at the same time requires their skills to be done effectively. As a result, many of them choose to focus purely on the areas they feel comfortable while ignoring the rest. Compounding the issue, it’s not something that engineers typically want to spend a lot of time and thought on, either, which means that it normally ends up being approached with one of two methods:

  • “I’ll know it when I see it” and
  • “Smart and gets things done” (From Joel Spolsky).

The first involves putting a series of engineers in a room with a candidate and seeing if they like them or not, with very little guidance or structure. The advantage there is that it’s low-effort both to set up and conduct, but the tradeoff is that it’s highly variable and can be highly inaccurate.

The second has a bit more structure, and is a fairly decent guide for how an interviewer should behave during an interview. It’s also a pithy summation of what you should look for in a hire (Why would you ever want to hire someone who wasn’t smart or couldn’t get things done?). I’ve found it works pretty well if there’s only one person doing the interview. But for a full day with highly technical people, it doesn’t tap the full potential of the interview process.

Our Approach

While at WePay we used to be somewhere between the two, we’ve since built a third approach to interviewing, which is to tackle it the same way we would any other engineering problem. That means with a clear set of requirements, a clearly-documented solution that meets those requirements, and a way to monitor what we have to make sure that it keeps meeting those requirements. The requirements that we started with were that we wanted the process to be stable, scalable, and selective.

By stable, I mean that we want the process to, as much as possible, produce the same result for the same candidate regardless of the interviewers.

By scalable, I mean that it should be able to handle an increasing number of candidates as WePay grows without a reduction in quality — the process has to make sure that as we need more interviewers, they are being trained. A process that relies on a few ‘good interviewers’ to handle the load isn’t going to be viable long-term.

By selective I mean that we want to only be hiring ‘good’ candidates, but what exactly a ‘good’ candidate looks like needs a bit more elaboration. Furthermore, what ‘good’ looks like will vary more than the other components (for example, a good entry-level engineer and a good senior engineer will have very different interview performances), but as before having a clear and documented definition of what that means was needed. Fundamentally, we settled on trying to select people who can:

  • Code, defined as being able to comfortably take a problem description, translate that into a description of what needs to be done, and then write code that does that thing.

  • Work well in a team, defined as being able to communicate politely and effectively.

  • Continue to develop in the future, which means they show interest in improving their skills and learning about things outside of their immediate responsibilities.

There were also some less-core requirements — we wanted the interview process to be pleasant for the candidates as well, we didn’t want the interview to be a test, and we didn’t want to spend a lot of time on candidates that didn’t meet our requirements. However, these additional criteria served more to inform the process than drive it.

With these requirements in mind, we set out to build a process that would meet them. To address stability we thoroughly documented the questions that we ask. We listed what common approaches are, where people often get stuck and how to unstick them. We also put together clear definitions of what a ‘good enough’ solution looks like (which often goes beyond, and sometimes doesn’t include, simply completing code).

To address scalability, we have a training system where we have two people conducting each interview — one to lead the interview and one to observe, learn from and provide feedback to the leader.

To address the ‘good’ requirement, we built several different interview session scripts, each of which targeted a particular aspect of what good meant to us.

Advantages

Building and assigning several different scripts, rather than having four people conducting the same interview, has a number of advantages for us. A one hour conversation about, say, a candidate’s past work experience can go much more in-depth than a 15-minute segment before a coding question. Because the interviewers doing that past experience interview know the limited scope of what they are supposed to be evaluating, they don’t have to worry about not having an opinion on the candidate’s coding skills. Because the areas of responsibility are distributed ahead of time, the group knows that together they will cover all of the areas they need to. This avoids a situation where we conduct four seemingly in-depth interviews only to realize that nobody thought to test the candidate’s design skills, to name just one common oversight.

We’ve chosen the four areas that we cover for software engineers (practical coding, algorithmic coding, high-level architecture, and previous experience) to give a strong signal in each of the areas that we care about in terms of determining if a candidate is likely to be successful at WePay. We also chose them to maximize overlap between roles. When we were developing our mobile developer interview, we were glad that we could just reuse most of the interview material that we had already developed. A mobile developer will need some mobile-specific skills, but they will also still need to be able to talk about their previous work, perhaps particularly with someone who isn’t as familiar with mobile concepts as they are.

Optimizing for Time and Value

Everything I’ve mentioned so far describes our on-site interview process, which is great and high-signal but not really something we can afford to do for everyone who applies. An onsite interview takes about five engineer-hours (ten when you count the observer) to conduct, and even more than that when you also include the prep and debrief time. There’s also the expense of potentially flying candidates in, and the inconvenience on their side because we’re taking their entire day. Given the costs, we built a phone interview focused on our most common causes of failure (which we track — right now the most common stumbling block is coding, followed by communication). This gives us fail-fast behavior with regard to our candidates. Ultimately, we are looking for a balance. We want our phone interviews to cast a wide enough net, and be permissive enough, that we get a wide pool of candidates to evaluate in detail in person. But we also want that pool to be at least very close to our hiring bar, so we aren’t wasting an entire day of everyone’s time.

The final piece of the process is collecting metrics — how questions are doing, how interviewers are doing, how well the questions are selecting the right people. That lets us have confidence that what we are doing is working and also continue to improve the process as time goes on. As an example, we track pass rates in our onsite interviews by phone screen interviewer, which we can use to give feedback to the people who are conducting phone interviews — if too many of their phone interview candidates are passing their onsites, we can let the interviewers know that they are being too selective at the phone interview stage, or if their candidates are disproportionately failing the previous experience section we can focus more on that.

Conclusion

So overall (or, in case you just skipped to the bottom), we’ve built a modular, monitored system that we believe accurately measures the qualities we want in software engineers, and lets us course correct when it isn’t doing that. In a future post, I’ll discuss some of the decisions that we had to make (for example, how to define a good question or what distribution of coding questions we wanted) and why we chose the directions we did.