Auditing of AI: Legal, Ethical and Technical Approaches
0
🏅
Sign in to get full access
Overview
- Auditing of AI systems is a rapidly growing field of research and practice
- This review article provides an overview of previous work in the field of AI auditing
- Three key points emerge from the review:
- AI auditing can learn from how audits have historically been conducted in other domains
- Both policymakers and technology providers have an interest in promoting AI auditing
- AI auditing is an inherently multidisciplinary undertaking
Plain English Explanation
This article takes a look at the growing field of AI auditing. Auditing refers to the process of reviewing and evaluating AI systems to ensure they are working as intended and not causing harm.
The paper highlights three main ideas:
-
AI auditing can learn a lot from how audits have been done in other areas like financial accounting and safety engineering. These established fields have developed rigorous auditing procedures that could be adapted for AI systems.
-
Both government policymakers and the companies developing AI technology have an interest in promoting auditing as a way to ensure AI is being used responsibly. Researchers can help by studying how effective different auditing approaches can be.
-
Auditing AI systems requires expertise from many different disciplines, including computer science, social science, philosophy, law, and industry experience. This diversity of perspectives leads to different auditing approaches, each with their own advantages and limitations.
The article concludes that the next step is to combine these various auditing methods into structured, holistic procedures. This would allow us to audit not just the technical details of AI systems, but also their real-world impacts on people and the environment over time.
Technical Explanation
The paper provides an overview of previous research and approaches in the field of AI auditing. It identifies three key insights:
-
Existing audit practices from domains like financial accounting and safety engineering can inform the development of AI auditing methodologies. These established fields have well-structured audit procedures that could be adapted for AI systems.
-
There is a shared interest among both policymakers and AI technology providers in promoting auditing as a governance mechanism for managing the societal impacts of AI. Researchers can contribute by evaluating the feasibility and effectiveness of different AI auditing approaches.
-
AI auditing is inherently multidisciplinary, drawing on expertise from computer science, social science, philosophy, law, and industry. This diversity of perspectives has led to the emergence of distinct auditing approaches, some focused on the technical properties of AI systems, others on the processes and governance structures of the organizations developing them.
The paper argues that the next step in AI auditing should be to integrate these complementary approaches into comprehensive, end-to-end procedures. This would enable auditing not just the AI systems themselves, but also their real-world impacts on users, societies, and the environment over time.
Critical Analysis
The paper provides a thorough overview of the current state of AI auditing research and practice, highlighting important considerations and directions for future work. However, it does not delve deeply into the specific challenges and limitations of the various auditing approaches discussed.
For example, the paper notes the distinction between technology-oriented and process-oriented audits, but does not explore the trade-offs or practical implications of each approach in depth. It would be valuable to understand the types of insights and assurances that can be reasonably obtained from each, as well as their potential blind spots or areas of weakness.
Additionally, the paper acknowledges the multidisciplinary nature of AI auditing, but does not critically examine how this diversity of perspectives may introduce tensions or challenges in developing cohesive, standardized auditing frameworks. Further research may be needed to address issues of disciplinary silos, differing priorities, and reconciling conflicting audit findings.
Overall, the paper provides a solid foundation for understanding the current state of AI auditing, but could be strengthened by a more nuanced analysis of the practical realities and limitations of the various approaches discussed.
Conclusion
This review article offers a comprehensive overview of the rapidly evolving field of AI auditing. It highlights three key insights: the value of learning from established audit practices in other domains, the shared interest of policymakers and industry in promoting auditing as an AI governance mechanism, and the inherently multidisciplinary nature of this endeavor.
The paper argues that the next step in the development of AI auditing should be the integration of complementary technology-oriented and process-oriented approaches into structured, holistic procedures. This would enable a more complete assessment of AI systems, accounting for not just their technical properties, but also their real-world impacts on users, societies, and the environment over time.
As AI systems become increasingly ubiquitous, the need for robust, reliable, and transparent auditing processes will only grow. The insights offered in this review article provide a valuable roadmap for researchers, policymakers, and industry stakeholders to collaborate in shaping the future of AI auditing and governance.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
🏅
0
Auditing of AI: Legal, Ethical and Technical Approaches
Jakob Mokander
AI auditing is a rapidly growing field of research and practice. This review article, which doubles as an editorial to Digital Societys topical collection on Auditing of AI, provides an overview of previous work in the field. Three key points emerge from the review. First, contemporary attempts to audit AI systems have much to learn from how audits have historically been structured and conducted in areas like financial accounting, safety engineering and the social sciences. Second, both policymakers and technology providers have an interest in promoting auditing as an AI governance mechanism. Academic researchers can thus fill an important role by studying the feasibility and effectiveness of different AI auditing procedures. Third, AI auditing is an inherently multidisciplinary undertaking, to which substantial contributions have been made by computer scientists and engineers as well as social scientists, philosophers, legal scholars and industry practitioners. Reflecting this diversity of perspectives, different approaches to AI auditing have different affordances and constraints. Specifically, a distinction can be made between technology-oriented audits, which focus on the properties and capabilities of AI systems, and process oriented audits, which focus on technology providers governance structures and quality management systems. The next step in the evolution of auditing as an AI governance mechanism, this article concludes, should be the interlinking of these available (and complementary) approaches into structured and holistic procedures to audit not only how AI systems are designed and used but also how they impact users, societies and the natural environment in applied settings over time.
Read more7/10/2024
🤖
0
A Blueprint for Auditing Generative AI
Jakob Mokander, Justin Curl, Mihir Kshirsagar
The widespread use of generative AI systems is coupled with significant ethical and social challenges. As a result, policymakers, academic researchers, and social advocacy groups have all called for such systems to be audited. However, existing auditing procedures fail to address the governance challenges posed by generative AI systems, which display emergent capabilities and are adaptable to a wide range of downstream tasks. In this chapter, we address that gap by outlining a novel blueprint for how to audit such systems. Specifically, we propose a three-layered approach, whereby governance audits (of technology providers that design and disseminate generative AI systems), model audits (of generative AI systems after pre-training but prior to their release), and application audits (of applications based on top of generative AI systems) complement and inform each other. We show how audits on these three levels, when conducted in a structured and coordinated manner, can be a feasible and effective mechanism for identifying and managing some of the ethical and social risks posed by generative AI systems. That said, it is important to remain realistic about what auditing can reasonably be expected to achieve. For this reason, the chapter also discusses the limitations not only of our three-layered approach but also of the prospect of auditing generative AI systems at all. Ultimately, this chapter seeks to expand the methodological toolkit available to technology providers and policymakers who wish to analyse and evaluate generative AI systems from technical, ethical, and legal perspectives.
Read more7/9/2024
0
The Necessity of AI Audit Standards Boards
David Manheim, Sammy Martin, Mark Bailey, Mikhail Samin, Ross Greutzmacher
Auditing of AI systems is a promising way to understand and manage ethical problems and societal risks associated with contemporary AI systems, as well as some anticipated future risks. Efforts to develop standards for auditing Artificial Intelligence (AI) systems have therefore understandably gained momentum. However, we argue that creating auditing standards is not just insufficient, but actively harmful by proliferating unheeded and inconsistent standards, especially in light of the rapid evolution and ethical and safety challenges of AI. Instead, the paper proposes the establishment of an AI Audit Standards Board, responsible for developing and updating auditing methods and standards in line with the evolving nature of AI technologies. Such a body would ensure that auditing practices remain relevant, robust, and responsive to the rapid advancements in AI. The paper argues that such a governance structure would also be helpful for maintaining public trust in AI and for promoting a culture of safety and ethical responsibility within the AI industry. Throughout the paper, we draw parallels with other industries, including safety-critical industries like aviation and nuclear energy, as well as more prosaic ones such as financial accounting and pharmaceuticals. AI auditing should emulate those fields, and extend beyond technical assessments to include ethical considerations and stakeholder engagement, but we explain that this is not enough; emulating other fields' governance mechanisms for these processes, and for audit standards creation, is a necessity. We also emphasize the importance of auditing the entire development process of AI systems, not just the final products...
Read more4/23/2024
🏷️
0
A Framework for Assurance Audits of Algorithmic Systems
Khoa Lam, Benjamin Lange, Borhane Blili-Hamelin, Jovana Davidovic, Shea Brown, Ali Hasan
An increasing number of regulations propose AI audits as a mechanism for achieving transparency and accountability for artificial intelligence (AI) systems. Despite some converging norms around various forms of AI auditing, auditing for the purpose of compliance and assurance currently lacks agreed-upon practices, procedures, taxonomies, and standards. We propose the criterion audit as an operationalizable compliance and assurance external audit framework. We model elements of this approach after financial auditing practices, and argue that AI audits should similarly provide assurance to their stakeholders about AI organizations' ability to govern their algorithms in ways that mitigate harms and uphold human values. We discuss the necessary conditions for the criterion audit and provide a procedural blueprint for performing an audit engagement in practice. We illustrate how this framework can be adapted to current regulations by deriving the criteria on which bias audits can be performed for in-scope hiring algorithms, as required by the recently effective New York City Local Law 144 of 2021. We conclude by offering a critical discussion on the benefits, inherent limitations, and implementation challenges of applying practices of the more mature financial auditing industry to AI auditing where robust guardrails against quality assurance issues are only starting to emerge. Our discussion -- informed by experiences in performing these audits in practice -- highlights the critical role that an audit ecosystem plays in ensuring the effectiveness of audits.
Read more5/29/2024