Agentic AI: Mistakes CIOs Make During Deployment

Business leaders are pushing their management teams to move quickly to deploy and capitalize on agentic AI tools. This CIO.com article notes that the “move fast and break things” approach puts tremendous pressure on CIOs, and discusses the seven mistakes that CIOs make when deploying AI agents.  This excerpt discusses the problem with testing AI agents like traditional software:

Robust regression tests deployed in continuous testing and automated continuous deployment are the goal for applications and APIs. Extend these objectives when building, testing, and deploying AI agents to account for variability in data, models, and real-time inference context.

“The most common mistake is treating an agent like a traditional app: You test it before deployment, sign off, and assume it’s safe in production,” says Sanmi Koyejo, co-founder and head of AI at Virtue AI. “But agents are non-deterministic and stateful, so the same request can trigger a different sequence of tool calls every time. Pre-deployment testing can’t enumerate those paths, and worse, a chain of individually permitted actions can still add up to data exfiltration or an unauthorized transaction.”

Koyejo suggests that testing also needs runtime enforcement, checking every tool call before it executes and either blocking or alerting on risky actions as they occur.

Patrick Phillips, CIO at Vasion, recommends CIOs build four controls before deploying AI agents.

• A kill switch that suspends any agent in seconds.
• A behavioral baseline, so they know what normal activity looks like.
• A post-incident review after every near-miss that asks which control should have stopped it.
• A feedback process for implementing improved controls.

How to avoid the mistake: Blur the lines between testing and monitoring AI agents, as their recommendations and actions should be evaluated consistently across both environments.

Other common deployment mistakes identified by the article include using AI agents where deterministic automation would do, building AI agents with no ownership or decision accountability, planning AI agents without trustworthy data, granting AI agents access to too much information, deploying AI agents without a people strategy, and treating an AI agent’s deployment as the finish line.