How to interview for working with AI agents

Teams are adding a line about AI to the hiring bar faster than they’re adding a question about it to the interview. Shopify is the best known example. It told its people that reflexive AI use is now a baseline expectation, and one company saying so still isn’t a trend anyone has counted. You can count what happens in the interview afterwards, and mostly the line gets waved through on a confident answer and a tool name.

AI fluency or AI tool use came up in 14.98% of the 331,872 interview sessions where the topics discussed could be detected. The subject is climbing fast from a low base: AI usage as a discussed topic went from 0.33% of conversations in the third quarter of 2025 to 4.54% in the second quarter of 2026. Both figures come from topic detection across Metaview’s aggregate corpus of 5.2 million candidate interviews, and topic detection is a proxy. It tells you the subject came up. It can’t tell you how well anyone assessed it.

A rubric won’t close that distance on its own, so it’s worth being blunt about what one can do. It can’t tell you who will be good with agents once they start, because nobody holds that evidence and Metaview holds no data on what happens after someone is hired. A rubric turns a vague line on a scorecard into a defined one, so four interviewers assess the same thing and their disagreement lands on the candidate instead of on what the words meant. That’s a smaller claim than most rubric articles make, and it’s the part that holds up.

Why the line keeps getting skipped

The interview is already full. That, more than indifference, is why the question keeps getting skipped. A single interview in that corpus touches a mean of 17.52 detected competencies, so a new one tends to arrive as the last two minutes of a loop, asked by whoever remembers. Adding it properly means deciding what it is and what it displaces, and that conversation runs slower than writing a phrase into a scorecard. The phrase spreads faster than the practice.

There’s a reasonable case that the trouble is warranted, and it lands better from someone who hires for a living than from a vendor.

That’s a practitioner’s read, and nobody has measured it. It argues for interviewing carefully, and it stops well short of showing that any particular level means anything. Judgment is the thing being assessed if an agent tracks the judgment of the person holding it, and judgment has never been readable from a tool name.

What a rubric can and cannot tell you

Start with the limit, because everything else depends on it. A delegation rubric scores an account of past work. The work itself never enters the room. A candidate who describes how they scoped a task, what context they gave, and where the output drifted is handing you a description, and a description can be detailed and rehearsed at the same time. Specificity gives you better material to probe, and it doesn’t make the account true.

The second limit: nothing connects a level to what comes after the offer. Metaview’s corpus covers what took place inside the hiring process and holds nothing about what happened once people started, and the four levels below haven’t been validated against anything at all. Read a level as a description of what the candidate told you, and leave the predictions out of it.

You still get something worth having. Four interviewers who share a set of anchors produce answers you can lay next to each other. A level with the evidence attached still reads a week later, at the debrief, when nobody can quote the example any more. The disagreement gets a subject when two people land on different levels for the same candidate: which part of the account they read differently, and what would settle it. A vague bar produces vague arguments, and the loudest voice in the room tends to win those.

The rubric is built against one specific trap, the one that shows up whenever a competency is new and nobody has agreed what good looks like.

The interview rewards whoever described the most AI use when nobody has set anchors, because volume of activity is the only thing an unanchored answer can be compared on. Anchors give the room something else to compare.

The four-level delegation rubric

Here’s one way to cut it. Four levels is a choice, and there’s nothing canonical about the number or the names. They earn their place only if your team agrees on them before the loop starts and uses the same words in the debrief. Read the fourth column hardest. It holds what the level doesn’t establish, which is most of what a level is.

The gap teams argue about most sits between delegator and supervisor, and it comes down to whether the person treats the agent as confident and often wrong. Define that gap carefully, because it’s where two interviewers most often score the same answer differently. Calling it a validated threshold would be inventing a finding.

The levels only sort anything if the questions surface behavior instead of opinions. Three moves do most of that work:

1. Ask for one handoff. “Walk me through the last real piece of work you handed to an agent, end to end. What did you hand over, and what did you keep?” A specific example gives you something to probe, and it says nothing about whether the work went the way it’s being described, which is why the follow-ups matter more than the story.
2. Inspect the limits they set. How did they scope it, what context did they give, and what did they check before using what came back? A delegator tends to stop at “it gave me a draft.” A supervisor can narrate how they kept hold of it.
3. Probe the catch. “When did an agent do something confidently wrong, and how did you notice?” This is where a thin answer usually shows, because the detail that makes a failure story credible is hard to improvise.

Write down what a strong, a middling and a weak answer sounds like at each level before the loop starts, and give the question to one interviewer instead of all of them. Spread it across a panel and you get four people asking whether the candidate uses AI, with nobody going deep enough to place them anywhere.

Where the disagreement should happen

All of this assumes the answer is still there when the score gets written. A delegation answer runs long and specific, and the specifics fade first. The anchors you wrote turn into decoration once the level gets assigned from an impression, and the debrief is back to comparing confidence.

That’s what capture is for. The Notetaker joins the interview as a participant with consent and records the conversation, then turns it into structured notes, each section linked back to the moment in the transcript it came from. Metaview drafts the scorecard from that conversation, and the interviewer reviews, rates and submits it. The level stays the interviewer’s call, made against the anchors the team wrote, from an answer that’s still on the page.

One level up, Reports covers whether a pipeline assessed the competency at all, and how the calls break down by team. That’s how you find out that half a loop skipped the question instead of scoring it low, and where two interviewers sit consistently a level apart. Closing that gap is a conversation your team has to have. A report shows you the divergence. It can’t calibrate anyone.

One piece of context belongs here, and the caveat goes first. Metaview’s 2026 AI and Hiring Alignment Report asked recruiting teams about their own use of AI. It asked nothing about the candidates they hired, so none of it is evidence about a delegation rubric, and it’s a single cross-sectional read, so what it found are associations.

The survey covered 505 respondents, 252 recruiting leaders and 253 hiring managers at companies with 200 or more employees across North America and EMEA. 85% of the companies that had already exceeded their hiring goals use AI in hiring at all. Teams where AI is core to hiring rate the recruiter and hiring manager relationship as excellent 55% of the time, and the report puts that at 3.8x the rate among teams that don’t use AI. 79% of all 505 respondents say they’re optimistic about AI’s future in hiring. Those are facts about recruiting teams, and the question here is about candidates. They’re part of why anyone is asking it at all, and no part of the answer.

What this means for your team

Pick one role you’re hiring for now and settle, out loud with the hiring manager, which level it actually needs. Most roles don’t need the top of the table. Writing that level down, with two lines on what a strong answer sounds like, is most of the work, and it’s the part that keeps getting skipped.

Then put it where the interview happens. Give the question to one interviewer, add it to your question bank and scorecard templates, and keep the answer available for the debrief. Revisit the anchors after a quarter of loops, because the first version of any rubric is wrong in ways only four people using it can show you. Our writeup on what separates good interviewers from bad ones covers the groundwork underneath all of this.

Working with agents landed on the hiring bar faster than most processes adapted, and honestly, nobody yet knows what any given interview answer means once the person starts. You can define the thing you’re asking about, ask it the same way every time, and keep the evidence so the argument at the debrief is about the candidate. That’s less than a rubric usually promises, and it’s the part that will still be true in a year.