AI Risk Management: Flock Fiasco Highlights Third Party AI Risks

A recent Business Insider article says that people may have good reason for hating those Flock cameras that seem to be popping up everywhere. The article reports that an analysis by the Roseville, California Police Department found that Flock cameras misidentified license plates in 71% of the alerts it sent to the Department in 2023 and 2024. This Forrester blog argues that Roseville’s experience with Flock highlights some of the problems inherent in addressing third party AI vendor risk. For example, the blog says there’s often a big gap between vendor claims and real world performance:

Flock reports that its cameras read license plates with 96% accuracy under optimal conditions. Roseville’s experience demonstrates the limitation of relying on that figure alone. Performance measured in controlled testing does not guarantee performance in production.

That gap between 96% accuracy and 71% inaccuracy matters because AI performance is contextual. Accuracy depends on the environment, data quality, operating conditions, and use case. A vendor’s benchmark reflects how a model performed in its testing environment, not your real-life environment. Organizations that treat those numbers as interchangeable create blind spots before deployment even begins.

The blog recommends that companies take the following actions to avoid the same kind of disconnect between vendor claims and actual performance that Roseville encountered:

• Validate vendor claims in production conditions. Test AI against your own data, operating environment, and edge cases before scaling deployment.
• Require human verification for high-consequence decisions. The greater the potential impact, the stronger the validation requirements should be.
• Formalize escalation requirements. Define reportable AI errors, establish review thresholds, and audit whether teams are escalating issues or working around them (or worse, hiding them).
• Expand third-party risk assessments. Ask vendors whether their accuracy claims have been independently audited, require evidence, and treat a refusal or vague answer as a material risk finding.
• Make error reporting a contractual obligation. Define what counts as a reportable error, set a threshold that triggers vendor and internal review, and audit whether your team escalates or quietly works around problems.
• Track state AI verification mandates. Human-in-the-loop requirements for high-stakes AI are emerging state by state, and building the control now costs less than retrofitting it under a deadline.