How to Audit AI Hiring Tools for Bias: A Step-by-Step Guide

A bias audit tells you whether an AI hiring tool treats groups of candidates differently, and why. In New York City it is a legal requirement. Everywhere else, it is fast becoming a basic standard of due diligence.

By the HRight Talks editorial teamUpdated 25 September 20264 minute read

Short answer

To audit an AI hiring tool for bias, inventory the tools that influence candidate decisions, define the stages and demographic groups to analyse, collect outcome data lawfully, calculate selection rates and impact ratios for each group, test whether differences are significant, investigate their causes, check the tool's job relevance and explainability, remediate and document findings, and repeat regularly. In New York City, an independent auditor must complete the audit within one year before the tool is used and a summary must be published.

Key takeaways

Audit outcomes, not intentions: measure what actually happens to each group.
Impact ratios are the core calculation; significance testing adds rigour.
Independence matters: NYC Local Law 144 requires an independent auditor.
An audit is a cycle, not a certificate: repeat after changes and at least annually.

The eight-step audit method

Step 1: Inventory AI tools

List every tool that scores, ranks, filters or recommends candidates, which stage it affects and who owns it.

Step 2: Define scope and groups

Decide which stages and roles to audit and which demographic groups to analyse, in line with legal requirements such as sex, race or ethnicity and intersectional categories.

Step 3: Collect and prepare data

Gather outcome data by stage, with demographic data collected lawfully, securely and separately from decision-making, or use legally permitted test data where historical data is insufficient.

Step 4: Calculate selection rates and impact ratios

For each group, divide the number selected by the number assessed, then divide each group's rate by the highest group's rate.

Step 5: Test significance and investigate

Check whether differences are statistically and practically significant, then identify which inputs or criteria drive any disparity.

Step 6: Review validity and explainability

Confirm the tool measures job-relevant criteria and that its outputs can be explained to candidates and reviewers.

Step 7: Remediate and document

Adjust criteria, configuration or use; strengthen human review; document findings, actions and any required public disclosures.

Step 8: Monitor and repeat

Track outcomes continuously and repeat the full audit at least annually or after significant changes.

Worked example: impact ratio calculation

Category	Assessed	Selected	Selection rate	Impact ratio
Men	500	150	30.0%	1.00
Women	450	117	26.0%	0.87
Category C	200	44	22.0%	0.73

In this illustrative example, Category C's impact ratio of 0.73 is below the 0.8 guideline, so the tool's effect on that group needs investigation. Where a tool produces scores rather than pass or fail outcomes, audits typically compare the rate at which each group scores above the median.

NYC Local Law 144 requirements

Applies to automated employment decision tools used to substantially assist or replace decisions on hiring or promotion for candidates or employees in New York City.
Requires a bias audit by an independent auditor no more than one year before use.
The audit must calculate selection rates and impact ratios by sex, race or ethnicity and intersectional categories.
A summary of results and the distribution date must be made publicly available.
Candidates must receive notice that a tool will be used, with information about the qualifications and characteristics it assesses.

See the NYC Department of Consumer and Worker Protection guidance linked below for the definitive rules.

EU AI Act considerations

For high-risk recruitment AI, providers must apply data governance practices that examine possible biases, and deployers must use the system according to instructions, assign competent human oversight, monitor operation and keep logs. An employer's audit programme supports these duties. See EU AI Act compliance checklist.

Dividing responsibilities with vendors

Activity	Vendor	Employer
Model design and pre-release bias testing	Leads	Reviews evidence
Configuration and criteria	Advises	Decides and documents
Outcome data on your applicants	Provides tooling	Owns and analyses
Independent audit	Cooperates	Commissions where required
Remediation	Fixes model issues	Changes use and oversight

This is general information, not legal advice. Anti-discrimination and AI rules differ by country and state; take qualified advice before relying on any tool for decisions about people.

AI Hiring Bias: Causes, Real Cases, Law and How to Prevent It
Where AI hiring bias comes from, how it is measured, the law, and a prevention framework.
How to Reduce Bias in AI Recruitment: 12 Practical Actions
Twelve actions across the hiring process to reduce AI bias, each with an owner.
EU AI Act Compliance Checklist for Employers and HR Teams
A seven-part checklist ordered by what applies now and what applies from December 2027.
Is AI Resume Screening Accurate? How to Measure and Improve It
What accuracy means for screening, why false negatives matter, and how to measure your own tool.

Frequently asked questions

What is an AI bias audit?

An AI bias audit is an evaluation of whether an AI tool produces different outcomes for demographic groups, usually by calculating selection rates and impact ratios, investigating causes and recommending remediation.

Is a bias audit legally required?

In New York City, employers using automated employment decision tools for hiring or promotion must have an independent bias audit completed within one year before use. Elsewhere it is increasingly expected as due diligence and supports obligations under laws such as the EU AI Act.

How often should AI hiring tools be audited?

At least annually, and whenever the tool, its configuration, the roles it is used for or the applicant population change significantly. Monitoring between audits should be continuous.

Who should conduct an AI bias audit?

An independent auditor with statistical and employment expertise, particularly where the law requires independence. Internal teams should still monitor outcomes continuously.