AI 환각

Generated answers can contain false statements even when their wording is clear and confident. NIST's Generative AI Profile discusses this risk as confabulation, including erroneous content and outputs that contradict their inputs. A next-token prediction objective does not itself constitute a factual verification process. Models can produce plausible continuations without adequate evidence for a specific name, number, or reference. This is one reason to examine the content rather than infer reliability from style. It is also incorrect to assume that a model never acknowledges uncertainty: some systems can abstain or qualify responses, although such behavior is not a guarantee of accuracy. Retrieval-augmented generation adds information from retrieved material, but the model may still use it incorrectly or generate unsupported details. Verification therefore has two parts. Check whether the cited source exists, then check whether it supports the particular claim. A real URL can lead to a genuine page that says something different. For a supplied document, compare relevant details directly. In an illustrative example, the document says delivery takes five business days, while a summary says two; that discrepancy is a source-faithfulness error even if the summary reads naturally. Long answers may mix supported and unsupported claims. FActScore studies evaluation of fine-grained factual statements against a knowledge source, illustrating why an answer-wide impression can miss individual errors. Evaluation and mitigation should use defined tasks and evidence rather than treating confidence, citations, or a product announcement as proof of universal reliability.