The Anatomy of AI Audits

Abstract A key element in AI governance is the development of standards and regulatory assessments to evaluate claims of performance—when such evaluations are executed with the objective of accountability, I refer to them as AI audits. The nature of such audits can take on multiple forms, depending on the context and the intent of those designing them. Through a discussion of several audit case studies, I break down the key characteristics and function of an audit within the broader structure of AI governance. In particular, I present the key differences between audits conducted by internal stakeholders and those executed by external stakeholders, and demonstrate how, in each case, the goals, perspectives, and challenges faced by the auditor actively inform audit design and feed into audit processes, outcomes, and consequences.