Making Interviews Objective with AI (Without Making Them Worse)
Everyone has opinions about candidates. That's the problem.
We're supposed to ask standard questions, evaluate people against the job description, and test whether they can do the work. Instead, we dig into areas where we think they're weak, ask different questions for each person, and end up testing our biases instead of their abilities.
This pattern mirrors what happens across business situations: we mistake personal impressions for objective reality, treating assumptions as facts rather than recognizing them as information gaps requiring systematic evaluation.
The scale of this problem is growing. 31% of employees now report experiencing unconscious bias during hiring, a 10% increase from the previous year according to TestGorilla's 2024 research. Yet only 1 in 4 companies have adopted core debiasing practices in their recruitment processes, McKinsey reports.
I recently made this exact mistake while trying to use AI to improve our interview process and learned something valuable about the right way to leverage AI in recruiting.
The Customization Trap
My first attempt seemed logical: take the job description, analyze the candidate's resume, and use AI to identify potential gaps. Then create a customized interview that would probe those areas more deeply.
This was a disaster.
I spent entire interviews testing candidates on perceived gaps instead of what they actually need to do in the role. If someone had five years of backend experience but their resume showed limited frontend work, I'd spend 30 minutes grilling them on CSS instead of validating their backend skills.
The result? I wasn't testing anyone on the full set of job requirements.
The Right Way to Use AI in Interviews
Here's what actually works: AI as a third-party judge, not as an interview customization engine.
My current workflow separates job requirements from individual candidate analysis:
Step 1: Create Standard Questions
Use the job description to produce a base set of behavioral and technical questions. This happens completely separately from any individual resume. The job is the job. Everyone needs to be evaluated against the same requirements.
One important caveat: job descriptions themselves can carry bias through narrow requirements, gendered language, or overemphasis on specific credentials. Before using a JD as your foundation, vet it for inclusive language and ensure it focuses on competencies that predict job success rather than surface-level qualifications. A candidate with transferable skills or nontraditional experience might outperform someone whose resume looks like a perfect match.
The key is focusing on what makes someone uniquely qualified for this specific role rather than general competence. Your questions should test whether they can do the actual work required, not whether they meet abstract "good candidate" criteria.
This structured approach matters. Meta-analysis research in industrial psychology shows structured interviews improve hiring accuracy by 25% compared to unstructured interviews, making them one of the most effective selection methods available. I'm a huge fan of Manager Tools, which teaches this as well.
Step 2: Analyze for Context (Not Customization)
Run the candidate's resume and job description through AI to identify areas where they might need to demonstrate growth or where their experience differs from traditional paths. I keep a running chat about each candidate in my tool of choice. This context is for my education going into the interview, not for changing what I test them on.
Step 3: Record and Transcribe
Record the interview and transcribe it. Skip video; you only need the transcript. This removes bias from body language, speaking patterns, accent, and appearance. The transcript objectifies the evaluation.
Be transparent with candidates: disclose that you're recording and explain how you'll use the transcript. This is both ethical and increasingly standard practice.
This follows the same principle behind successful data-driven approaches: replacing subjective impressions with quantifiable metrics that can be systematically analyzed rather than interpreted through personal bias.
Step 4: AI Evaluation
Feed the transcript, the candidate's resume, the questions asked, and the job description into AI for evaluation. The AI provides:
- A detailed assessment of how well they answered each question
- A scorecard based on job requirements, not personal impressions
- An evaluation of whether they can actually do the job
Step 5: Evaluate the Interviewer Too
Here's the part most people miss: have the AI evaluate your performance as the interviewer. What did you miss? What follow-up questions should you have asked? Did you talk too much? Did you forget to ask key questions?
This feedback loop makes you a better interviewer for future candidates.
Why This Works
The transcript-based approach removes the most common sources of interview bias. A University of Washington study from October 2024 found that AI resume screening tools favored white-associated names 85% of the time. The bias in those systems came from visual formatting, name recognition, and surface-level resume patterns. Transcript-only evaluation sidesteps these problems entirely by focusing AI analysis on what candidates actually said, not how their resume looked or what their name suggested.
By evaluating what they said against what the job requires, you get a more objective assessment of their ability to do the work.
This approach isn't perfect. Transcript-based evaluation can still favor candidates whose communication styles match dominant cultural norms. Non-native English speakers or candidates with different vernacular styles might be penalized despite having the skills you need. Prompt your AI explicitly: "Evaluate technical accuracy and problem-solving approach. Ignore grammar, filler words, and speaking style." Always have a diverse group of humans review the AI's output before making final decisions.
The HR Multiplier Effect
This approach solves a major problem for HR teams: getting consistent, objective feedback from interviewers. Scorecards are notoriously difficult to get people to fill out properly. Even when interviewers complete them, they often forget to ask key questions or focus on the wrong things. This is a classic example of doing good work that's not the right work: spending energy on activities that feel productive but don't deliver the strategic outcome you need.
With transcript-based AI evaluation, HR gets consistent scoring, objective data on what was actually tested, feedback for interviewers to improve, and protection against bias-driven decisions.
The Bottom Line
AI shouldn't replace human judgment in hiring. It should make human judgment more objective and consistent.
The key insight: don't use AI to customize interviews based on perceived candidate gaps. Use it to evaluate actual responses against actual job requirements, and to make interviewers better at their craft.
When everyone gets tested on the same job-relevant criteria and evaluated objectively on their responses, you end up hiring people who can actually do the work instead of people you happen to like.
Related Content
