What to ask an artificial intelligence vendor before you sign

Gartner estimates that, of the thousands of vendors now advertising agentic artificial intelligence, only about 130 actually deliver it. It also projects that more than 40% of projects in that category will be canceled before the end of 2027.

You don't need to read much into those two figures. They mean the odds of talking to someone who can't deliver what they sell are high, and that the way to protect yourself is not to understand the technology. It is to ask the right questions.

Here are eight. None requires technical knowledge, and every one is uncomfortable to answer if the vendor has nothing behind it.

1. What happens when the system isn't sure?

It is the first question, and it is the one that separates vendors fastest.

A system that always answers is a system that sometimes makes things up. In an industrial operation that is not a minor flaw. It is the difference between a quote and a warranty problem.

The answer you are looking for describes a concrete behavior: exactly what the system does when it is missing information. If the answer is an accuracy percentage and nothing else, they didn't answer the question.

The wording we think is right, and that you can use as a yardstick: when there is no certainty, the system stops and escalates to a person with what it has already gathered.

2. What data of mine did you test it with?

This is the toughest filter of all, and there is a public test that backs it up.

It is called BEAVER, it is reproducible, and it measures what happens to a generic model when it faces a real company's data. Accuracy drops from 62.9% to 11.4%.

A demo with sample data tells you nothing about your operation. Ask for a test with your catalog, your product names, your equivalences and the odd requests that reach you.

If the vendor resists testing with your data before you sign, you already have your answer.

Read alsoThe evaluation workbook: three steps, with the tables to fill in.

3. How many times in a row does it get it right?

Different from the previous question, and almost never asked.

In customer-service tests, a model that is right 60% of the time on the first try is right 25% of the time when it has to be right eight times in a row.

In a real operation you don't need one correct answer. You need the fortieth one on Tuesday to be correct too.

4. Who else like me is already using it, and can I talk to them?

Don't ask for the logo. Ask for the phone number.

And when you talk to that reference, ask just one thing: what went wrong and how it was fixed. A real customer always has a story. A brochure customer doesn't.

5. How long does it take to reach production, and what does production mean to you?

MIT NANDA found that the best-performing mid-market companies reported an average of 90 days from pilot to full implementation, while large enterprises took nine months or longer.

But the important question is the second half: what counts as production. For some vendors, production means the system is switched on. For you it should mean your people use it every day without anyone chasing them.

Ask for that definition in writing.

6. How is it priced?

Among enterprise software buyers, 43% prefer consumption-based models and 27% prefer outcome-based ones. Fewer than one in five still prefers per-user pricing.

That is not a fad. Per-user pricing charges you for installing, not for serving, and it pulls the vendor out of line with your result.

7. What happens to my people?

If the vendor's answer takes the form of "reduces headcount," be careful, because the data says otherwise.

The U.S. Census Bureau measured, in the field between November 2025 and January 2026, that 66% of companies that use artificial intelligence use it only to extend what their people do, and that AI-related headcount reductions occurred in just 2% of firms.

In Mexico, the Ministry of Economy found that, in manufacturing companies with more than ten employees, for every ten percentage points of adoption, there is an association with 18.8% more output, 5.4% higher wages and 3.3% more people employed. It is association, not causation, and it should be stated that way. But it points in the opposite direction from the fear.

8. What do I have to do?

A vendor who says you don't have to do anything is selling you a disappointment on installments.

RAND interviewed 65 practitioners: 84% pointed to a leadership failure as the primary cause of failure, ahead of data quality and technological maturity. And the first of the five root causes RAND identifies is that decision-makers misunderstand, or miscommunicate, what problem the artificial intelligence is meant to solve. The vendor can't solve that alone.

Ask them to state explicitly what they need from you, from whom and when.

Two public stories worth more than any warning

They are told without naming anyone, because the point is not to point fingers.

One of the four big consulting firms delivered a US$290,000 report prepared with artificial intelligence to a government. It contained invented academic references. The firm refunded part of the payment.

A world-renowned hospital spent US$62 million over five years on an AI-assisted oncology system. In tests it was as accurate as the doctors. The contract expired without it ever having been used on a single patient.

Both organizations had budget, talent and access to the best technology available. What was missing was not technical capacity. It was clarity about what problem was being solved, and for whom.

The short version

If you only take three questions from this text, make them these:

1. What does it do when it isn't sure?
2. Can it be tested with my data before I sign?
3. Who like me can I talk to, and what went wrong there?

A vendor who answers those three well can probably deliver. One who dodges them probably can't, however good their demo.

If you'd like to talk through how these questions apply to your case, that conversation is open.