Visa put 22 AI models through shopping tests on November 5. The Visa Agentic Commerce Benchmark ran 15 scenarios. Each model talked to a customer about buying a TV, a laptop or a personal care appliance. Each model figured out what they needed and decided whether to buy. The models failed 42.8 percent of the test runs.
In 54 percent of failed runs, the agent completed a purchase. It picked a product that missed the customer's requirements. Or it bought before it uncovered all of them. By that math, 23 percent of all runs ended with the wrong purchase.
The agent that cannot walk away
The Visa test was designed to fail a lucky guess. The agent had to uncover enough information to justify its choice. If nothing fit, it had to stop. Half the agents that failed bought something they should not have.
A model got three attempts at each scenario. All three passed only 34.2 percent of the time. When a model passed once, it often failed the same scenario on the next run. The best model passed 82.2 percent of runs. The worst passed 28.9 percent.
The five most expensive models averaged a 67 percent success rate. The five least expensive averaged 50 percent. Cost bought consistency, but no model was reliable across every run.
The review gate and the approval step
Atelier's agent asks me before it sends, posts or drafts an email. The decision to act sits behind a prompt. The Coen ERP has a mandatory human review gate on payroll runs and ADP mapping. I would add the same gate to any agent that can complete a purchase.
A system that stripped its own monitor is one problem. An agent that buys the wrong thing 23 percent of the time is another. The Visa test shows what happens when the agent acts without approval.
Visa's Trust Index from September found that 72 percent of consumers had used an AI assistant. Only 23 percent trusted AI to make payments for them. The gap between use and trust is the gap between browsing and buying.
Who pays when the agent is wrong
American Express announced in early 2026 that it would cover erroneous purchases made by registered AI agents on its network. The policy applies to verified agents that authenticate cardholders before they act. The press release does not mention fraud.
Visa said it plans to extend the benchmark to checkout, payment and order placement. The test so far measures product discovery. The agents it tested could not reliably determine what to buy. The next step is whether they can complete the transaction without breaking something else.
Half the agents that failed still bought something.
Somewhere an agent just ordered six cases of the wrong part. The invoice is already approved.