Nine questions to ask any AI vendor before you sign

Demos are designed. They run on clean data, with a cooperative caller, on the happy path, presented by someone who has given this demo two hundred times. You will not learn much from one.

These nine questions will tell you more in fifteen minutes than the demo tells you in an hour. Ask them of us too — we wrote them expecting to be asked.

1. "Will you sign a BAA?"

First question, always, before pricing. If the answer is no, the tool cannot touch patient information and the conversation is over for that use case.

Good answer: "Yes, here's our standard one, send it to your counsel." Immediate, unbothered, already drafted.

Bad answer: anything that begins with "well, technically we don't store…" A vendor explaining why they don't need one is a vendor who hasn't got one.

2. "Draw me the data map. What information goes where?"

Ask them to walk through it: what leaves your building, which companies touch it, where it's stored, who at their company can see it, how long it's kept, how it's deleted.

Good answer: they have a diagram, or they can sketch it confidently in two minutes.

Bad answer: "it's all encrypted, it's very secure." Encryption is a control, not an architecture. If they can't draw the flow, they don't know it, and neither will you when someone asks.

3. "Is our data used to train anything?"

Good answer: "No, and it's in the contract." Then check that it's actually in the contract.

Bad answer: "Only in aggregate, anonymized form to improve the service." That may well be fine, but you need to know exactly what it means, because "anonymized" does a lot of quiet work in that sentence.

4. "What happens when it doesn't know the answer?"

The single most revealing question on this list. Every one of these systems will hit something it wasn't built for, on day one.

Good answer: "It says it doesn't know and hands to a human. Here's the escalation path, here's how you configure what triggers it, let me show you a real transcript of it happening."

Bad answer: "It handles pretty much everything." That means either they haven't tested the edges, or it guesses. Guessing is how a machine tells a patient something wrong in your practice's voice.

5. "Show me a transcript of a call that went badly."

Ask for a real one. Redacted is fine.

Good answer: they have them, they'll show you, and they'll tell you what they changed afterwards. Everyone building these has bad transcripts. Only the honest ones will show you.

Bad answer: deflection, or a "bad" example that's obviously curated to be charming. This question is a character test more than a technical one.

6. "Who owns the configuration, and what happens if we leave?"

Good answer: "You do. Here's the export. Your scripts, your rules, your data, in a usable format."

Bad answer: vagueness, or an export that's technically available in a format nobody can use. Ask specifically what you get on the way out, in what file format. The answer to this question is worth more than any feature on the roadmap, because it determines whether you're a customer or a hostage.

7. "How does it connect to our practice management system, exactly?"

Push past "we integrate with that." There are three very different answers:

• A supported API. Best case. Stable, documented, survives updates.
• A partner or marketplace integration. Usually fine, sometimes limited in what it can write.
• Screen automation — a robot clicking through the interface. Fragile. Breaks when the vendor changes a button. Sometimes it's genuinely the only route and that's an acceptable trade, but you must know you're choosing it, because it changes the maintenance burden completely.

Ask which one. If the answer is fuzzy, assume the third.

8. "What does it cost when we're twice the size?"

Get the pricing model, not the price. Per call, per minute, per user, per provider, per location?

Then run it at double your current volume. Some pricing models are quietly punitive at scale and a fast-growing practice can find its bill has tripled while its revenue went up forty percent.

Also ask what happens at renewal and what the cap on increases is.

9. "Who reads what it says, and how often?"

This is the question nobody asks, and it separates a product from a service.

Somebody must read a sample of its conversations regularly — daily for the first month, weekly after. Wrong answers found in week one are trivial to fix. Found in month six, they've been repeated four hundred times.

Good answer: "We do, and you get a report." Or, honestly: "You do — here's the review interface, budget twenty minutes a week."

Bad answer: silence, or an implication that this isn't necessary. It's always necessary. If nobody's named for it, nobody does it.

---

How to read the answers

You're not looking for perfection. You're looking for two things.

Specificity. Good vendors answer with detail, including detail that's unflattering. Vague, warm answers to precise questions mean either they don't know or they'd rather you didn't.

Willingness to say "no" and "we don't." The most trustworthy sentence any vendor can say is "that's not what this is good for." If you get through all nine questions and they've claimed to be excellent at everything, be suspicious — nobody is excellent at everything, and the ones who say so haven't yet met the edge cases you're about to hand them.

One more, unwritten: notice how they treat your office manager during the demo. If the person who'll actually use this every day is talked over, that's how the whole relationship will go.

---

Print the list. Take it to every demo you sit through, including ours — if we can't answer all nine cleanly, don't hire us. Book the audit or take the scorecard first if you're not sure what you're shopping for yet.