Chuyện gì đã xảy ra
Researchers introduced a Werewolf that measures how observing LLMs update their beliefs after receiving suspicion and accusation messages. Across 1,224 annotated messages and 40 open-weight model configurations, larger models more often identified true wolves from the game history but remained strongly influenced by accusations and the perceived trustworthiness of the accuser.
The paper proposes evaluating belief shifts rather than relying only on the final outcome of a social-deduction game. In its setup, an observing village-side model receives game messages, including suspicions and accusations, and researchers measure how its beliefs change after each message.
The abstract reports results from 40 open-weight LLM configurations and 1,224 annotated messages. Larger models performed better at distinguishing wolves from villagers using the full game history. However, accusations still increased suspicion toward the accused and reduced suspicion toward the accuser, particularly when the accuser was already trusted.
The reported pattern persisted even when the trusted accuser was wolf-aligned. Larger models were better able to resist accusations from accusers they already distrusted, but models up to 120 billion parameters still struggled to integrate the accusation’s content with the reliability of its source. The source provides no detailed per-model scores, statistical uncertainty, or independent replication in the supplied text.
Tại sao nó quan trọng
The study identifies a specific weakness in strategic communication: models may not adequately separate the credibility of a speaker from the evidence contained in that speaker’s claim. That matters for LLM agents operating in settings where messages can be selective, deceptive, or adversarial. The result is a and research finding, not evidence that all deployed AI systems behave this way or that the evaluation generalizes beyond Werewolf.
For researchers evaluating LLM agents, the result suggests that final game success may conceal important weaknesses in intermediate reasoning and belief updating. A model can reach a plausible decision while still overweighting who delivered a claim rather than assessing the claim alongside the available evidence.
The practical implication is limited but useful: evaluations of agents that communicate or make decisions from multiple reports may need to measure belief changes after individual messages, including cases where a credible speaker is misleading. The study does not establish that the same behavior occurs in deployed products or in non-game settings.
Cơ chế tương tác: Nó thực sự hoạt động như thế nào
Khám phá công nghệ cơ bản đằng sau sự phát triển này một cách tương tác.
Which component of an AI application is the machine-learning model itself?
Xem gì tiếp theo
The and code are available through the project website, but the source does not state licensing, hosting details, or whether the evaluated models are available for public use. Further work should test whether the finding transfers to other communication tasks, whether training improves source-content reasoning, and how much results depend on game design and choices.
The authors say the and code are available at the linked project page. The supplied source does not document access requirements, licensing, pricing, supported frameworks, or whether the benchmark includes model outputs beyond the annotated messages.
Important unknowns include how the messages were generated and annotated, how trust was established, whether the results are statistically robust across individual models, and whether larger models’ improved game-history reasoning translates into better resistance to manipulation.
Replication across other strategic communication tasks would help determine whether this is a Werewolf-specific effect or a broader limitation in how LLM agents combine source credibility with accusation content.