社会ガイド

Gender Shades and Facial Recognition Bias Audits

Gender Shades is a 2018 study by Joy Buolamwini and Timnit Gebru that audited commercial face analysis systems from IBM, Microsoft and Face++.

4 分で読めます
最終更新日 2026年9月15日

読み始めるクイズに答えてください

このページでは4 分で読めます

概要
ディープダイブ
戦略的影響
The Future of Gender Shades and Facial Recognition Bias Audits
現実世界の実装
リスクとガードレール
実装ロードマップ
探検を続けましょう
よくある質問

概要

It found their gender classification was far less accurate for darker-skinned women than for lighter-skinned men. The study matters because it made intersectional auditing, which breaks results down by combined attributes rather than one at a time, a standard way of exposing hidden AI failures.

ディープダイブ

Joy Buolamwini, then at the MIT Media Lab, noticed that face detection software did not register her face until she put on a white mask. With Timnit Gebru, she designed a structured audit and published it at the 2018 Conference on Fairness, Accountability and Transparency. Existing benchmarks were dominated by lighter-skinned men, so they built their own, the Pilot Parliaments Benchmark. It contains 1,270 images of members of parliament from three African and three European countries, labeled by binary gender and by skin type on the dermatologists' Fitzpatrick scale. They tested the gender classification services of IBM, Microsoft and Face++. All three did better on men than on women and better on lighter skin than on darker skin. The worst results appeared where the two overlapped. Error rates for darker-skinned women reached about 35 percent for the worst system, compared with under 1 percent for lighter-skinned men. A single overall accuracy figure hid this gap. A 2019 follow-up by Inioluwa Deborah Raji and Buolamwini, "Actionable Auditing," found that the audited companies had narrowed their gaps after being publicly named. Vendors that had not been audited, including Amazon, showed similar disparities. Amazon disputed the methodology. A common misconception is that Gender Shades measured the face identification used by police. It measured gender classification. Evidence on identification came from NIST's 2019 demographic effects report, which tested around 200 algorithms. Many had higher false positive rates for some groups, including African and East Asian faces, often by large factors, while the most accurate algorithms showed much smaller differences. Real harm followed. Robert Williams, Nijeer Parks and Porcha Woodruff, all Black, were each wrongly arrested after facial recognition leads. In June 2020, IBM said it would leave the general-purpose facial recognition business, Amazon paused police use of Rekognition, and Microsoft said it would not sell to US police until a federal law was in place.

技術的な洞察

The core technique is disaggregated evaluation: measure errors separately for each combination of attributes, not only for each attribute on its own. Subgroups can be small, so report confidence intervals and balance the test set, as the Pilot Parliaments Benchmark did. For recognition systems, separate the false match rate from the false non-match rate, and separate one-to-one verification (unlocking a phone) from one-to-many identification (searching a mugshot database). A single similarity threshold tuned on a whole population can produce very different false match rates for different groups. One-to-many searches of large galleries amplify small per-comparison differences into many false leads. Skin type labels are themselves a design choice. The Fitzpatrick scale was built to measure how skin responds to sun, and some teams now use broader scales such as the Monk Skin Tone scale. Finally, audit the exact deployed version, because results change with each model update.

戦略的影響

リスクと安全性

AI による壊滅的な被害も日常的な被害も、誰がリスクを理解し、誰が行動できるかにかかっています。

より明確な判決

国民と専門家のリテラシーは、強力な安全政策が政治的に可能かどうかを左右します。

誇大広告を打ち破る

明確な説明は、誇大広告、研究室の PR、曖昧な倫理劇場に囚われることを減らします。

The Future of Gender Shades and Facial Recognition Bias Audits

Face recognition has become more accurate on average, and NIST's continuing tests show demographic differences shrinking for leading algorithms, though not disappearing, and staying large for many others. Policy remains fragmented. Some US cities restrict government use, the EU AI Act sharply limits real-time remote biometric identification in public spaces by police, and several vendors have withdrawn some products. Wrongful arrest cases continue to push police departments to treat matches as leads only. The Gender Shades method, publishing disaggregated results and naming vendors, is now widely used for auditing other AI systems, including language and image generators.

現実世界の実装

A product team reports face verification accuracy separately for darker-skinned women, darker-skinned men, lighter-skinned women and lighter-skinned men, instead of publishing one headline number that can hide a failing subgroup.

Robert Williams, a Black man in Detroit, was arrested in 2020 after facial recognition matched his driver's license photo to shoplifting footage. The charges were dropped, and a later settlement changed Detroit police rules on using such matches.

Before approving a vendor, a city procurement office checks NIST's demographic test results for the specific algorithm version on offer, because error differences vary widely between algorithms.

A researcher builds a test set balanced by skin type and gender, following the Pilot Parliaments Benchmark approach, to check a new face model for gaps before release.

リスクとガードレール

能力が複雑になる一方で、実存的なリスクを SF として扱います。
高度な自律性の下での調整による表面製品の安全性を混乱させる。
英語以外や専門家ではない聴衆には、低品質の情報源しか提供されません。

実装ロードマップ

製品の危害、誤使用、制御不能/調整不良のリスクを分離します。
どのような証拠がタイムラインと重大度についてのあなたの見方を変えるかを尋ねてください。
マーケティング上の主張よりも、一次情報源と具体的な評価を優先します。
意識だけでなく、キャリア、政策、資金、スキルなど、行動経路を 1 つ特定します。

探検を続けましょう

Free newsletter

Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.

One email each weekday. Unsubscribe in one click. We never sell or share your address.

Test yourself

Take the Gender Shades and Facial Recognition Bias Audits quiz

Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.

クイズを開始する

Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation

よくある質問

What is Gender Shades and Facial Recognition Bias Audits?

Gender Shades is a 2018 study by Joy Buolamwini and Timnit Gebru that audited commercial face analysis systems from IBM, Microsoft and Face++. It found their gender classification was far less accurate for darker-skinned women than for lighter-skinned men. The study matters because it made intersectional auditing, which breaks results down by combined attributes rather than one at a time, a standard way of exposing hidden AI failures.

Which companies' face analysis services did the original Gender Shades study audit?

The 2018 study tested IBM, Microsoft and Face++. Amazon was examined in the 2019 follow-up.

What was the Pilot Parliaments Benchmark made of?

The benchmark used 1,270 images of parliamentarians from six countries, chosen to balance gender and skin type.

Which scale did the researchers use to label skin type?

They used the Fitzpatrick scale from dermatology. Some later work uses broader scales such as the Monk scale.

What task did Gender Shades actually measure?

The study measured gender classification. A common misconception is that it measured the identification systems used by police.

Which subgroup had the highest error rates in Gender Shades?

Errors peaked where the two attributes overlapped, at darker-skinned women, reaching about 35 percent for the worst system.

学び続ける