发生了什么
Tech Xplore 报道了 LLMScholarBench，这是由复杂性科学中心相关研究人员开发的基准，用于评估大型语言模型如何推荐专家。报告称，该基准测试了 22 个涵盖技术质量和社会代表性指标的模型，发现推荐结果往往偏向于高引用率、资深、男性、美国白人学者。检索增强生成提高了事实准确性，而提示可以引导代表性，但这两个目标仍然存在紧张关系。
Tech Xplore 报道了 LLMScholarBench，该基准由复杂性科学中心相关研究人员开发，用于测试 22 个模型的专家建议：20 个开放权重模型和来自 Gemini、GPT、Llama、Qwen、DeepSeek、Grok、Mistral 和 Gemma 等系列的专有模型。它衡量了五个技术指标——事实准确性、一致性、有效性、拒绝和重复推荐——以及四个社会指标：连通性、文献计量相似性、多样性和均等性。
研究人员首先评估了 5 个物理推荐任务的 6 个开放权重模型，其中包括对前 5 名和前 100 名有影响力专家的请求。该参考数据库包含超过 450,000 名在 1893 年至 2020 年间在美国物理学会期刊上发表过论文的科学家。Tech Xplore 表示，大约 80% 的推荐都是模型在该数据库中命名的科学家，而领域和资历不匹配的情况则更加困难；在初始测试中，物理子领域的不匹配平均约为 40%。
该报告描述了人口和地理方面的偏差。在参考记录中，女性占研究人员的 14% 到 32%，具体取决于子领域和时期，但模型通常推荐较少的女性或不推荐女性。亚洲学者是最大的人口群体，但白人学者的比例往往过高，而黑人和拉丁裔研究人员往往缺席。建议集中于发表文章和引用较多的学者，而较小的模型则将建议集中在较少的国家/地区。
Tech Xplore 报告的事实性得分为 0.63 至 0.82，多样性得分为 0.44 至 0.69，均等得分为 54% 至 60%。 DeepSeek 和 Gemini 在真实性方面最强，DeepSeek 在多样性方面领先，Gemma 在平等性方面领先。对 22 个模型的四项干预表明，检索增强生成提高了技术质量，尤其是准确性，同时提示工程改进了代表性；将它们结合起来仍然不能立即改善每一项措施。
A further study examined whether specified role, language or geographic location changed recommendations across six academic disciplines. Location affected results, while language and role did not. Tests of newer proprietary Gemini 2.5 Pro and Flash models with web retrieval reportedly increased factual accuracy but reduced diversity and parity. The source presents LLMScholarBench as an ongoing auditing tool rather than a definitive ranking, noting that some models may have been updated since the research.
为什么这很重要
人工智能系统越来越多地用于寻找专业机会的人员，包括研究人员、医生和律师。如果推荐反复偏向那些已经非常引人注目的人，那么系统可能会缩小获得机会的机会，同时将其输出呈现为中性。研究结果还表明，为什么准确性本身并不是推荐系统的完整衡量标准：列表可以包含真正的专家，但仍然会重现或加剧人口和地理的不平衡。
Expert recommendation can be a gatekeeping step: conference organizers may find keynote speakers, employers may assemble candidate pools, and patients may seek doctors through an LLM. Tech Xplore reports that researchers see these risks beyond academia. Repeatedly naming highly visible people can direct attention and opportunities toward the same group while leaving less-visible qualified people undiscovered.
The findings separate factuality from fairness. A recommendation may be factually valid because the person exists and works in the field, yet socially unbalanced if it excludes groups present in the relevant professional population. The report’s distinction between technical quality and social representation offers a more useful framework than checking only whether names are real or whether citations are available.
The intervention results complicate the assumption that web search solves recommendation . Tech Xplore says retrieval improved factual accuracy but, in tests of newer proprietary models, lowered diversity and parity. Researchers attribute that risk to underrepresentation on the web; the source presents this as their interpretation, not an independently established causal finding. Grounding a system externally therefore does not automatically make its information representative.
The geographic effect matters for international users. If location changes which scholars are recommended, a system may encode assumptions about local relevance or visibility rather than expertise alone. The source says language and role did not change results in the reported tests, but that does not show they never matter in other disciplines, languages or models. The result applies to the study’s tested conditions.
The source does not establish that any model caused someone to lose a job, invitation or opportunity. It also lacks enough methodological detail to determine demographic assignments, the number of runs behind each score, or uncertainty calculations. Scores can vary with prompts, model versions, retrieval sources and sampling. The work’s public value is therefore exposing a measurable risk and proposing a repeatable audit structure, not proving every deployment behaves identically.
互动机制：它实际上是如何运作的
以交互方式探索这一发展背后的基础技术。
What is a common training objective for an autoregressive language model?
接下来看什么
该消息来源引用了 2026 年 ACM SIGKDD 出版物和两项 arXiv 研究，但这些主要材料并未在此处进行独立审查。重要的开放性问题包括结果如何随提示、抽样、模型更新和人口统计分类方法而变化，以及基准是否预测实际招聘、招生或会议选择系统中的结果。未来的评估应该测试更广泛的参考数据和结构化的人工审查是否可以共同提高事实性和代表性。
Independent examination of the cited primary research is the first priority. Tech Xplore identifies a 2026 ACM SIGKDD paper on benchmarking and intervention-based auditing, plus arXiv papers on the initial audit and persona-prompting effects. Those materials should clarify prompts, model versions, trial counts, statistical uncertainty, demographic-labeling procedures, reference populations, refusals and duplicates. These details should not be inferred from the secondary report alone.
Researchers and deployers should test whether LLMScholarBench findings extend beyond physics and the six disciplines described. APS publication records may not represent fields with different publication patterns, geographic structures or demographic data, and may miss expertise documented outside academic publishing. Testing medicine, law, engineering, public service and skilled trades would show whether the risk generalizes to settings where people already use AI recommendations.
Model updates and retrieval sources require continuous monitoring. The report says some evaluated models have changed and describes newer Gemini tests with web retrieval that produced a different balance of results. A single run can become outdated. Follow-up should compare releases over time, document retrieved web sources, and measure whether changes improve accuracy and representation together rather than shifting performance between dimensions.
Organizations using AI to recommend people should require transparent criteria, human review and a way for qualified individuals to be considered when omitted. They should record the prompt, model version, retrieval setting and output, and evaluate more than whether names are real. The source reports neither a regulatory requirement nor a confirmed industry response, so these are practical safeguards suggested by the risks, not measures already adopted because of the .
It remains unresolved whether more representative data can overcome the reported trade-off. Tech Xplore says the team plans to build a broader scholarly knowledge base, but provides no evidence that it is complete or improves performance. Future results should show reproducible gains in factuality, diversity and parity, plus persistence in realistic recommendation tasks without merely optimizing the benchmark.