Tips for Selecting AI Bias Auditors

I’ve harped on the dangers of algorithmic discrimination and bias in many of our previous blogs. My soapbox isn’t without merit; bias in automated decision-making is a leading driver of risk in AI. Failure to test for biased outputs has resulted in lawsuits and settlements, especially in the context of HR and highly regulated industries. Tangential markets are arising with third-party vendors promising they can audit systems for algorithmic bias. AI bias audits are a great tool, but auditor experience and expertise vary. How do you know what to look for in an auditor? A recent Fisher Phillips memo and questionnaire seek to help answer that question:

“You need to know whether your auditor truly understands bias, disparate impact, and the legal landscape. An impressive technical résumé alone isn’t enough – your provider should combine statistical know-how with regulatory awareness.

Sample Questions:

• What expertise does your team have in disparate impact and bias mitigation?
• Which AI-related laws and regulations have you worked with, and how familiar are you with their requirements?
• Have you adjusted your methodologies to account for pending legislation, such as CPRA rules or Illinois’ 2026 AI law?
• Can you share examples of similar projects you’ve completed, including key outcomes?”

These are just a few of the sample questions provided by Fisher Phillips. As with any emerging consultancies, AI bias auditors need to be selected on a case-by-case basis. Companies should take their specific needs into account and tailor their vendor selection.  Asking all the right questions can ensure you select the right vendor for your company.