视频内容审核中的人工智能

Video content moderation is the work of finding and acting on videos that break a platform's rules, such as graphic violence, child sexual abuse material, terrorist propaganda, nudity, harassment or dangerous acts. AI in this context means automated systems that inspect videos and their audio, text and metadata, then either take action directly or send the video to human reviewers. Platforms rely on automation because the volume is far beyond human capacity. In its October to December 2023 report filed under California law AB 587, YouTube said more than 500 hours of content are uploaded every minute and that the vast majority of content it reviews and acts on is first detected by automated systems.

There are two main families of technique. The first is matching against known content. A system computes a hash, a short digital fingerprint, of a video or its frames and compares it with a database of fingerprints from material already judged violating. Cryptographic hashes such as MD5 only match exact copies. Perceptual hashes such as Meta's PDQ for images and TMK+PDQF for video are designed so that visually similar files produce similar fingerprints, which catches copies that have been resized, re-encoded or lightly edited. Because only fingerprints are shared, organizations can pool them without exchanging the harmful files themselves.

The second family is classification of new content. Machine learning classifiers are trained on labeled examples to estimate how likely a video is to contain a given kind of violation. A pipeline can sample frames, runs image models on them, analyzes the audio and any speech, reads on-screen text and the title or description, and combines these signals into scores per policy area. Each score is compared with thresholds chosen by the platform: very high-confidence cases may be removed or restricted automatically, uncertain cases go to human moderators, and low scores pass. YouTube's report describes this as a collaborative loop in which human moderators confirm decisions and their feedback is used to retrain the models.

The final stage is the action and the appeal. Actions are not just removal. YouTube's report lists restricting a video, for example age-restricting content unsuitable for all audiences, as well as removal and channel strikes, with three strikes in 90 days able to end a channel. Users are notified and can appeal, and YouTube says appeals go to human review. The results show how early automation acts: in that quarter, YouTube removed 9,012,232 videos, 96.1% of them first flagged by automated systems, 3.4% by ordinary users and 0.6% by organizations in its Priority Flagger program. Just over half of removed videos, 51.51%, had zero views, and another 26.43% had between 1 and 10 views.

Shared databases grew out of specific crises. After a gunman livestreamed the March 2019 attack on two mosques in Christchurch, New Zealand, Facebook reported that the live broadcast was viewed fewer than 200 times and about 4,000 times in total before removal, that the first user report came 29 minutes after the stream began, and that in the following 24 hours it blocked more than 1.2 million copies at upload and removed about 300,000 more. It found more than 800 visually distinct variants and used audio matching to catch edits whose visuals had changed but whose soundtrack had not. That year GIFCT, the Global Internet Forum to Counter Terrorism, expanded its Hash-Sharing Database from content produced by groups on the United Nations Security Council sanctions list to include content from perpetrators who livestream or record attacks, and later added attacker manifestos and URLs. In August 2019 Facebook released PDQ and TMK+PDQF as open source.

Compared with alternatives, each method has a clear role. Human-only review cannot keep pace with hundreds of hours of uploads per minute. User reporting is valuable for context but slow: in the Christchurch case no user reported the live video until after the broadcast ended. Hash matching is precise and cheap but only finds content someone has already identified, and determined uploaders try to evade it with edits. Classifiers can find new material but make more mistakes. In practice, platforms layer all four, using hashes for known material, classifiers for triage, humans for context and appeals, and user reports as an additional signal.

The limitations are significant and documented. Classifiers need many training examples, and Facebook explained that its systems missed the Christchurch livestream partly because such events are rare and partly because similar-looking footage, such as livestreamed video games, would cause floods of false alarms. Classifiers also miss context. In a January 2021 decision, Meta's Oversight Board found that a machine learning classifier had removed a Brazilian user's Instagram post about breast cancer symptoms under the nudity policy, failing to recognize the Portuguese words for breast cancer on the image, even though awareness posts were explicitly allowed. The Board recommended telling users when automation acted, allowing human review of such appeals, and auditing samples of automated removals. Automated systems can also perform unevenly across languages and cultures, and they struggle with satire, news reporting and documentary uses that YouTube's policies explicitly exempt.

In practice, building video moderation requires clear written policies, labeled training data for each policy area, reviewer teams and tooling, appeal processes, and metrics such as the share of removals reversed on appeal. Smaller platforms rarely build all of this themselves. They often join hash-sharing programs such as GIFCT's, which does not dictate how members enforce their own rules, use open-source fingerprinting tools like PDQ and TMK+PDQF, and buy classification from vendors. Livestreams add a time constraint, since harm can occur before review, so platforms combine real-time classifiers with rapid escalation paths. The central design choice is where to set thresholds: lower thresholds catch more violations but remove more legitimate speech, and the right balance differs by harm type, which is why many platforms publish transparency reports so outsiders can examine the trade-offs.