AI Hiring in a Live Company: The Agent Filled the Funnel. The AI Interview's Labels Did Not Line Up With Who Is Still Working

AI hiring means software agents take over parts of a recruiter's job. They read applications, decide who gets an interview and book it. Sometimes they run the first interview too. The AI interview is the newest piece. The candidate opens a link and talks to a voice AI instead of a person. Then a model writes a verdict into the company's hiring system.

This is the second part of a series on AI agents doing management work in one live business, a US-facing telemedicine company. Part 1 followed an AI supervisor running the company's phone team. This part follows the same company as it hired people for that team. It covers the path from the first application to the first working day, and beyond that as far as today's data goes.

The short version. The agent took the slow, repetitive part of hiring and handled it at a scale no one on the team could match. The AI interview was meant to replace the human first round. Instead it ran alongside it. Its verdict showed no sign of separating the people still working from those who stopped, though the numbers are small. And a large part of the losses came not from choices but from no-shows and decisions left waiting.

The company's name is under a non-disclosure agreement. Days are counted from the day the first candidate entered the CRM (day 1). The snapshot is day 36. All numbers come from the company's CRM, the AI interview records and the hiring agent's own files. The method is in the companion technical report.

The job and the problem

The role is a phone job. The person calls partner physicians, helps them get started and keeps them active. It calls for people who have done outbound phone sales before and can work a full day from home.

One job posting brought in about 2,400 applications by day 32. A second, similar posting added more. Reading the 1,973 worth a close look would take one recruiter 65 to 100 hours, time this small team did not have. The person who would normally run the first interviews is the team's human supervisor, who also runs the floor every day.

So the company gave the front of the funnel to an agent.

What the hiring agent does

The hiring agent is a Claude agent. It reads incoming applications on the job site, including the answers to the posting's screening questions and the work history. It picks candidates by rules that leadership writes. It enters them into the CRM, offers a time and books an interview slot. The invitation goes out in the same message as the proposed time, with "reply if this does not suit you". So the data has no separate "candidate replied" step. A candidate moves from picked to booked at once.

Over 36 days the agent read 1,973 applications in detail. The CRM received 142 candidates, and 138 human interviews were booked for 130 of them.

The rules changed often. Leadership changed the selection rules eight times in the first 15 days. The changes included phone sales only, no second chance after a missed interview, and a stricter checklist built from the supervisor's past decisions. The agent applied each change from the next batch on.

Where the funnel actually leaked

Here is the human-interview part of the funnel, with all bookings counted.

Almost one booking in three ended in a no-show. In the second calendar week it was close to four in ten.

The second leak was waiting. On day 15, 37 candidates had passed the supervisor's interview, and the CRM showed no hiring decision for any of them, although a few had already been given CRM accounts. The oldest had been waiting 13 days. Nobody had decided "no". Nobody had decided "yes" either. The AI supervisor from part 1 found this pile and kept raising it in its daily summary, but it could not hire anyone. A hire needed a CRM account, a start date and someone to meet the newcomer, and all three were people's jobs.

The AI interview

On day 15, leadership decided to remove the human supervisor from the first round. From then on, candidates would get a link to an AI interview instead of a booking.

Here is how it works, in five lines:

1. The candidate gets a personal link. The page needs no login, only a microphone. It takes about 15 minutes. 2. Short pre-recorded videos of a company executive ask the five questions and open, connect and close the interview. The videos are an AI avatar. A voice AI asks one follow-up if an answer is vague and plays the physician in the role play. 3. Part one is five fixed questions: past outbound sales with numbers, a real working day, other commitments, a full-time schedule and a start date. 4. Part two is a five-minute role play. The AI plays a busy physician with objections, and the candidate tries to win a first commitment. After that, the candidate can ask questions. 5. A separate model scores the transcript, using only the content of the answers, not how they were delivered. It returns HIRE, MAYBE or NO, with a reason that can be pasted into a polite reply. The result lands on the candidate's CRM card.

The CRM stores a number alongside the verdict: 90 for HIRE, 50 for MAYBE, 10 for NO. In part 1, the AI supervisor's weekly report said three candidates had "scored 90/100" in the AI interview. That was a fixed label for HIRE, not a measured score. The part 1 report flagged it as misleading.

What the AI interview did

From day 16 to day 36, 41 candidates were invited. 22 finished the interview, or 54%. All 22 finished between day 16 and day 26. In the last ten days of the period, nobody finished one, even though invitations kept going out; the records do not show why. Human interviews, meanwhile, came back. Four were booked in the week of the rule that was meant to end them, and 23 more in days 21 to 36.

The final verdicts were HIRE 7, MAYBE 12, NO 3. One early NO was re-scored as MAYBE after the scoring rule was corrected.

For 16 candidates, there is both an AI verdict and a human interview result. The AI agreed with the supervisor on all five HIRE verdicts and disagreed on both NO verdicts: in both cases the supervisor had passed the candidate weeks earlier. MAYBE results split six passes to four fails across nine candidates (one was interviewed twice). The order varies: for some candidates the human interview came first.

Who is still working

Trial days began on day 21. A trial is one working day in the CRM with actual calls. The team lead judges it and records the result the same day. The table shows where everyone who received a CRM account stood on day 36.

(One more account went to a candidate who was invited to the AI interview but never took it. That person made calls and left.)

Thirty candidates received a CRM account. Eleven of them never made a single call. Nineteen started, and nine of those stopped after one to four days of calls: for five the company ended it after a trial day, one left on their own, and for three the records do not say who ended it. Ten are working.

The three candidates behind the AI supervisor's "90/100" line were all scheduled for trial days on day 21. One is still working. One never made a call. One made calls for one day.

The numbers are small. They do not prove that MAYBE is a better sign than HIRE. They do show no sign that the label sorted people the way its name suggests. Of the ten people now on the team, six came through as MAYBE and two as HIRE (Fisher exact test for 2 of 7 against 6 of 12: p ≈ 0.63).

Speed changed more clearly, though not only because of the AI interview. Compare everyone who entered the CRM with the same seven-day window. Of 112 candidates who entered before day 16, one made a first call within seven days of entry. Of 28 who entered on days 16 to 29, six did. Trial days, which started on day 21, are part of that change too. A CRM account turned into a first call in one to four days for all 19 who started. The wait always came before the account, never after it.

Who decides about people

The hiring agent decides who gets an interview. It does not decide who gets hired. (This is a different agent from the part 1 supervisor, with its own rules: it may record decisions in the CRM.)

The CRM holds 48 hire or reject stage changes; some candidates have more than one. People entered 36 and the agent entered 12. All four hires the agent entered cite the team lead's decision in the team chat; one also adds the agent's check of the trial day against CRM activity. The trial verdict belongs to the team lead.

There is one exception, and it matters. All three candidates with an AI NO were rejected on that verdict. The agent entered two of those rejections, and a person entered the third, citing "score 10". Two of the three had already passed the human supervisor's interview. There is no record that a person read the transcript before any of the three rejections. That is the point where an AI verdict alone ended an application. It is also where rules on automated hiring decisions, such as Illinois's reporting duty for AI-only screening, would focus if they applied.

The ethics block: what candidates were told

Two US laws make a useful yardstick, although neither applies to this case. New York City's Local Law 144 covers automated employment decision tools used for jobs in the city. It requires an independent bias audit within the past year and a public summary of that audit. It also requires notice to candidates at least ten business days before the tool is used, saying which qualifications the tool assesses and how to request an alternative process or an accommodation. Whether a tool "substantially assists or replaces" the decision, the city's test for coverage, is itself often an open question. Illinois's Artificial Intelligence Video Interview Act covers employers who use AI to analyze video interviews that applicants record for positions based in Illinois. It requires the employer to tell the applicant that AI may be used and to explain how it works and what it looks at. The employer must get consent first, limit who sees the video, and delete it within 30 days of a request. A newer amendment to the Illinois Human Rights Act, in force since the start of 2026, requires notice when AI is used in recruitment and hiring, video or not.

Here is how this AI interview measures up against them.

What it did. The page tells the candidate that answers are saved as text for the team. It says plainly that an AI plays the physician in the role play. The scorer was told to judge the content of the answers only. Candidates could record an answer again. Most HIRE and MAYBE candidates (14 of 19) also had a human interview.

What it did not do. No one was offered a human interview instead of the AI one, and the three NO candidates were rejected without a person making the decision on the transcript. The page says the questions come from a company executive on video. The candidate is only heard, never filmed. But the page does not say the video is an AI avatar. It does not say a voice AI asks the follow-up questions, or that a model writes the verdict. There is no separate consent step. There is no record of a bias audit. In one session, a candidate asked whether this AI interview was part of the hiring process. That question sums up the gap.

There is also the matter of what the scorer was asked to weigh. One NO was tied to working-hours availability. A full working day is a real requirement of the job. But a rule like that can fall harder on some groups, and impact ratios by sex and race/ethnicity, of the kind a Local Law 144 audit requires, is where that pattern could show up.

A page that met the yardstick would say this before the first question: you will talk to an AI. The video is an AI avatar of a company executive. A model will score your answers, and a person makes the decision. You can ask for a human interview instead. And no NO would end an application unless a person had read the transcript.

What it cost and whom it replaced

The agent replaced the reading. At two to three minutes per application, 1,973 applications come to 65 to 100 hours of a recruiter's time (the author's estimate). It also replaced the booking and the reminder messages.

It did not replace the human first round. The AI interview was meant to, and in the week of the rule few human interviews were booked. Then human interviews came back, and most people who passed the AI interview were also interviewed by a person. It did not replace the team lead's trial-day decision, the operations manager who creates accounts, or the onboarding call.

What comes next

Nobody hired through this funnel has reached 30 days yet. On day 36 the longest-working person had 22 calendar days since their first call. Version 1.1 of the technical report will add 30-day retention for everyone who started, split by AI verdict, once the first group passes that mark.

What to do if you are putting AI into hiring

1. Give the agent the reading and the booking first. That is where it saves the most and risks the least. 2. Measure show-ups and decision lag alongside the quality of screening. In this funnel, no-shows (37 of 118 held or missed bookings) removed as many bookings as human interview fails (37). 3. Treat an AI interview verdict as one input, and check it against who actually stays. Here, with small numbers, the label showed no sign of predicting it. 4. Store real scores or store labels, and call them what they are. "90/100" that means HIRE will be read as a measurement. 5. Tell candidates before they start that they will talk to an AI, that the video is synthetic, and that a model scores them. Offer a human path. 6. Never let a NO from a model close an application without a person reading why.

Caveats

This is one company, two similar job postings and 36 days. There is no control group. The AI interview was not randomized, and the people who took it entered the funnel at a different time from those who did not. The groups in the outcome table are small, so the differences between them are not statistically meaningful. Candidates were matched to CRM accounts by name. 30-day retention cannot be observed yet. Business results are under the NDA. The full method, formulas and limitations are in the companion technical report.

This is part 2 of the series "AI managers in a live company". Part 1: An AI Supervisor Ran a Call Center Team for Three Weeks.