표제어 분석 및 형태소 분석

Stemming and lemmatization are distinct normalization methods. A stemmer applies rules such as suffix removal and can produce a fragment that is not a dictionary word; for example, the Porter stemmer maps “studies” to “studi”. A lemmatizer aims to return a dictionary lemma using vocabulary and morphological analysis, often informed by part of speech. Context matters: a system may map comparative adjective “better” to “good”, while the appropriate normalization depends on the token’s role and the tool’s rules. Both methods can reduce variation for indexing, but can also collapse distinctions. Subword tokenization is a separate technique: byte-pair encoding merges frequent adjacent symbols to build subword units, which neural translation systems can use to represent rare or unknown words. WordNet’s Morphy documentation describes exception lists and part-of-speech-specific morphology rules for mapping inflected forms to base forms. With the comparative adjective reading, “better” may map to “good” in a lexicon-based lemmatizer; behavior depends on the resource, POS, and context.