Interviewing With AI
The vibe-coding trend is alarming to anyone who values correctness in code. Putting AI generated code into production without full scrutiny adds operational risk to the extent that your system relies on correctness. With increasing use of AI tooling in development, how do you ensure that your new hires can both take advantage of the tools and also ensure correct execution?
At a recent meeting of the Austin CTO Club a member shared with me a useful approach to this problem when interviewing developers in an AI-enabled context. The idea is to split the interview into two parts that focus on each element in the question, first how to use the AI tools, and second understanding the outputs from those tools.
It works like this:
- Give the candidate a simple project to implement with the help of AI.
- Timebox the implementation.
- When time runs out, have them demo the results to you.
- Ask them to explain how the system works.
Candidates who are good tool users will do well on the implementation. Candidates who are highly technical will demonstrate good understanding of the system in their explanation. You want those who do well on both implementation and explanation.
As with most hiring decisions, you must gauge the upside potential of the candidate in your context. If your business is extremely sensitive to errors (say, fintech or medical) you may place higher weight on the understanding. If your business is less sensitive to errors (say, social media) then you may weight the tool use more heavily. In either case, this technique gives you good inputs to weigh.
The AI tools for development are changing rapidly. How frequently do they and will they produce correct outputs? Can a candidate keep up with this fast tool evolution? Does your environment support continuous improvements in tool use? Factoring your answers into your weights amounts to longer-term bets on tool capabilities, candidate, and your organization. Good judgement in hiring is always difficult. This technique provides important inputs for those judgements.