Why an AI BDR demo cannot prove pipeline
Estimated reading time: 5 minutes
An AI BDR demo can show whether a workflow deserves a pilot. It cannot establish whether your offer will produce accepted sales opportunities in your market. That needs actual accounts, recipient decisions, sales follow-up, and enough time to observe what happens.
Keep the demo. Ask it to answer a narrower question: can this system perform the work you would authorize it to do? Then make the pilot test the parts that require a real buyer.
Separate the evidence by the decision it supports
A generated email gives you an output to inspect. It does not tell you whether the company fits, the recipient has the relevant responsibility, or the proposed conversation deserves their time. Even a correct, relevant email leaves the recipient's response unknown.
Anthropic distinguishes an agent's recorded actions from the outcome left in its environment. Its evaluation guidance combines offline tests with methods that observe live use. Those distinctions are useful when assessing outbound, although the guidance does not establish a sales conversion benchmark. Anthropic's evaluation guidance.
Use this demo-versus-pilot evaluation sheet. The suggested tests are our editorial framework, not a validated predictor of pipeline.
| Question | Evidence a demo can supply | Evidence the pilot must supply |
|---|---|---|
| Can it research this kind of company? | A trace from source to company fact, including unresolved questions | Reviewed outputs across the actual segment, with corrections and research time |
| Does it exclude unsuitable accounts? | A known bad-fit account rejected for the right reason | The disposition of the full assigned account set, including exclusions |
| Can it write within the approved offer? | A draft linked to source evidence and approved seller claims | Messages approved and sent, with edits and review effort recorded |
| Does it respect account history? | A blocked duplicate and a reply that cancels a pending step | Sending and stop events reconciled with actual account history |
| Do buyers want a conversation? | No direct evidence from generated copy alone | Recipient replies, reasons, and agreed next steps |
| Does sales accept the result? | A demonstration of the handoff fields | Sales acceptance or rejection with the reason and subsequent opportunity status |
Make the demonstration inconvenient in useful ways
Give the provider a small set of companies you select. Include an unsuitable company, an account with incomplete public evidence, and an account with prior contact history. Do not send anything during this exercise.
Ask the operator to show the inputs, source passages, instructions, and changes they make. Record help from a person as part of the workflow. Human help can be a sound operating choice; an evaluation should expose how much help the proposed service includes.
An unknown answer can be correct. If public evidence does not establish budget authority, a research note should leave it unresolved. A system that fills every field may produce a more complete-looking report while introducing claims you cannot use.
Include a stop test. Put a simulated reply into the test environment and inspect whether the pending follow-up remains scheduled. A screen saying “paused” is less useful than the actual queue state and event history. Confirm which channels and linked campaigns the stop covers.
These tests establish behavior under chosen conditions. A vendor can pass them and still struggle with an unfamiliar segment, inaccessible sources, or production integrations. Preserve those limits in the scorecard.
Read benchmark scores at their measured level
Microsoft's Sales Qualification Agent evaluation scores research, outreach, and qualification tasks with defined rubrics. It describes model judging, sampled manual calibration, and simulated qualification conversations. Those are task-quality observations from Microsoft's evaluation setup. A higher task score does not, by itself, establish more pipeline in your campaign. Microsoft's benchmark methodology.
Ask what the denominator contains. Were the examples all completed tasks, selected successes, or all assigned leads? Were the messages graded by a model, a seller, or actual recipients? Was the comparison a general chat interface or another configured outbound workflow? Keep those answers beside any claimed improvement.
Give the pilot a commercial evidence chain
Define a qualified conversation and an accepted opportunity with sales before launch. A booked meeting may still be a no-show or an unsuitable buyer. An accepted opportunity should have the evidence your sales process requires, with a named person responsible for accepting it.
Track the chain by account and date:
- Record every account assigned to the pilot.
- Record which accounts passed review and why the others did not.
- Record which contacts received messages and the evidence for their selection.
- Classify replies without merging rejection, curiosity, and purchase interest.
- Record meetings held separately from meetings booked.
- Record sales acceptance, rejection, and later opportunity changes.
Keep a fixed observation date and mark accounts still awaiting a response. Do not remove difficult accounts from the denominator after seeing the results. Report corrections, research cost, operator time, and sales time alongside the commercial outcomes.
Illustrative funnel, not a customer result: suppose a pilot assigns 40 companies. Research approves 30, and outreach reaches people at 24. Six companies reply, three meetings happen, and sales accepts one opportunity. The account reply rate is 6/24, or 25% of reached companies. The accepted-opportunity share is 1/40, or 2.5% of assigned companies. Neither figure is a per-email rate or an expected return for another campaign.
The gaps identify different work. Ten exclusions concern selection. Six approved but unreached companies concern execution. Replies that fail sales acceptance concern qualification or offer fit. Preserve those gaps instead of reducing the pilot to one attractive percentage.
Decide what the pilot leaves unresolved
A small pilot can expose a broken source trail, missing owner, or unusable handoff. It can also produce too few commercial observations to rank two options. Record an inconclusive outcome when the observation period or account count cannot support the proposed conclusion.
A comparable manual workflow can help explain whether automation changes the work. In a narrow market, allocating enough independent accounts for a comparison may be impractical. Use manual review to diagnose failures without claiming a causal lift from an unmatched before-and-after result.
Quantonica's judgment is to keep piloting as the offer, account criteria, and sources change. A successful initial run authorizes only the next bounded operating decision. It does not settle future performance.
Before the next demo, send the provider the evaluation sheet and your definition of an accepted opportunity. Ask them to mark each requested proof as demonstrable now, testable in the pilot, or still unknown. Use the unknowns to write the pilot agreement.