The single biggest predictor of whether your interview will tell you something useful is not who you hire to run it. It is whether you wrote the questions before the candidate walked in.
Unstructured interviews (the friendly chat, the gut-check, the rapport round) consistently underperform structured ones on every measure that has been studied. The research is decades deep, the effect sizes are large, and the practical implication for SMB hiring is the same answer almost every time: the hiring manager running the interview themselves cannot also be the calibration layer. The structure has to be the calibration layer.
What follows is the framework we use at Join when we design interview loops for ourselves, condensed for SMB recruiters who don’t have a dedicated talent partner.
What 25 years of research actually says
The reference text for selection-method validity is Schmidt and Hunter’s 1998 meta-analysis in Psychological Bulletin, covering 85 years of research findings across 19 selection procedures (the link is to a 2016 update of that paper by Schmidt, Oh, and Shaffer, which reproduces the original figures). The headline number that gets quoted most: structured interviews predict job performance at a validity coefficient of around .51, while unstructured interviews land near .38. Combined with a general mental ability measure, structured interviews push to .63.
An earlier meta-analysis by McDaniel and colleagues (1994) in the Journal of Applied Psychology, drawing on data from 86,311 individuals, found similar patterns: situational interviews outperform job-related interviews, which outperform psychologically-framed interviews, and structured interviews outperform unstructured ones for predicting on-the-job performance (the two come out roughly level for predicting performance in training). More recent work from the SIOP community has nuanced the 1998 numbers without overturning the direction: structured interviews remain among the highest-validity selection methods available to a hiring team.
For a quick mental table:
| Selection method | Validity (≈) |
|---|---|
| Work-sample tests | .54 |
| Structured interviews | .51 |
| Cognitive ability (GMA) | .51 |
| Unstructured interviews | .38 |
| Reference checks (unstructured) | .26 |
| Years of education | .10 |
The interesting line is the .38-to-.51 gap. That gap is not produced by hiring smarter interviewers. It is produced by writing better questions in advance.
One caveat worth stating plainly: those .51/.38 figures rely on a range-restriction correction that Schmidt’s own later work flagged as the less accurate of the two methods available. The 2016 update cited above reruns the same data with an improved correction and finds structured and unstructured interviews land at nearly the same zero-order validity — .58 each. What still favours structure in that update is incremental validity on top of a general-mental-ability test (an 18% gain for structured interviews versus 13% for unstructured). The same correction revises several other figures in this piece: general mental ability alone moves from .51 to .65, the “GMA plus structured interview” combination rises from the 1998 figure of .63 to .76, and work-sample tests — reassessed with a much larger 2005 meta-analysis — average .33 for job performance generally rather than .54, though the original .54 may still hold specifically for traditional manual trades such as machinists and welders. So the case for structure holds, but several of the numbers above describe where the field stood in 1998 rather than where it stands now.
Question types that predict
Three categories of question carry the load.
- Behavioural questions. “Tell me about a time when you had to ship a project on a deadline you knew was unrealistic. What did you change, who did you talk to, and what happened?” These ask the candidate to walk through a specific past instance. The signal is in the level of detail they can recall and how clean their account of cause and effect is. Vague answers correlate strongly with vague past performance.
- Situational (hypothetical) questions. “You join the team next month. The product manager has shipped a feature that’s leaking customers; engineering says the fix takes six weeks; sales wants it in two. What do you do this week?” These ask the candidate to reason through a job-relevant scenario in real time. McDaniel found situational interviews edged out behavioural for predictive validity in many job families. They also feel like the actual job, which improves candidate experience.
- Work-sample tests. A 30-to-60-minute task that resembles the real work. A take-home for an engineer; a 30-minute live edit for a writer; a sales-call roleplay. Schmidt and Hunter put work samples at .54 validity, higher than any interview format. The trade-off is candidate time and the legal landscape around unpaid work, especially in DACH where unpaid work samples beyond a token amount are restricted.
Google’s published research on their own hiring practice (re:Work) found internally that structured interviewing predicted on-the-job performance across functions and seniority levels, and reduced adverse impact on protected groups. That second part matters: structure is not only more predictive, it is more defensible.
Question types that don’t predict anything
Three categories show up in almost every interview loop and produce noise.
- Brain teasers. “How many golf balls fit in a 747?” Google’s own research found these had no correlation with job performance and were quietly retired. They survive in interview loops as a way for the interviewer to feel clever. Cut them.
- Rapport / “tell me about yourself” questions. Useful for the first thirty seconds of a conversation, useless as a hiring signal. They reward fluency, polish, and confidence (correlated with class background more than competence) and they prime interviewers to like or dislike a candidate before any signal has been collected.
- Self-assessment questions. “What’s your biggest weakness?” / “How would your last manager describe you?” The answers are uniformly coached, the calibration is impossible, and the variance you see is variance in self-presentation skill, not in fit for the role.
Cutting these does not save interview time. It saves interview attention, which is the bottleneck.
Building a scorecard the hiring manager will actually use
A structured interview without a scorecard is half the structure. The scoring is what forces the interviewer to compare to a standard rather than to the previous candidate.
The minimum-viable scorecard:
- One row per question.
- Each row has a 1-to-5 anchor scale with at least two anchors written out (e.g., “3 = walked through one project, recalled outcomes but not specific decisions; 5 = walked through one project, recalled specific decisions, cause and effect, and what they would change about their approach in hindsight”).
- Interviewer scores each row before any debrief with other interviewers, to avoid post-hoc rationalisation and groupthink.
- Final hire/no-hire decision is a function of the scores, not a separate gut call.
This is also the section that gets reused the most. The same 60-day outcomes that go into the job description (see our guide to writing job descriptions) are the same outcomes you should be scoring against in interviews. If the scorecard does not map to the job ad, one of the two documents is wrong.
For SMB hiring managers running the interview themselves, this discipline is the entire game. There is no panel of senior interviewers calibrating each other in the room with you. The scorecard, written before the candidate arrived, is what stands in for that panel.
Where AI helps, and where it stops
Drafting interview questions from the job description is a 30-second task for the model and a productivity assist for the human. Summarising the candidate’s answers into a structured record for later debrief is also fine. Both fall on the limited-risk side of the EU regulatory line.
Scoring candidates with AI is on the other side of that line. The EU AI Act classifies AI systems used for the recruitment or selection of natural persons as high-risk under Annex III. Full obligations (risk assessment, bias testing, human oversight, transparency) were originally due to bite on 2 August 2026, but the July 2026 “Digital Omnibus” amendment (Regulation (EU) 2026/1744) pushed the compliance date for stand-alone high-risk systems — recruitment tools included — to 2 December 2027. An AI that ranks candidates or recommends hire/no-hire on the basis of interview answers is in scope. An AI that helps the human interviewer draft and remember is not.
The dividing line is the same one we apply across the product at Join: AI is the assistant, not the decision-maker. In an interview loop the scorecard is the assistant the hiring manager actually needs. The AI is a tool below that.
What this looks like in practice
For most SMB hires Join’s customers run, three to five structured questions per interview, two interviewers (one with veto), one work-sample step, and a scorecard filled in before debrief is the configuration that produces the cleanest hiring decisions. Adding interviewers past four rarely improves predictive validity in our customer pipelines; adding interview rounds past four rarely changes the decision. The thing that does change the decision is which questions got asked, and that is upstream of how many people are in the room.
The lesson from twenty-five years of meta-analyses, distilled: write the questions first. Calibrate before debrief. Treat AI as a drafting assistant, not a judge. The rest is editing.
