NLP 기초

Text must be represented in a form a computational system can process. Tokenization splits it into units such as words or word pieces; numerical representations then support rules, statistical models, or neural networks. Token boundaries are a modeling choice and do not always align with what a reader considers one word. Some tasks return a label for a whole document. Others identify spans inside it or produce a new sequence. A sentiment classifier, an entity recognizer, and a summarizer therefore solve different problems even if all use the same underlying language model. Context matters. The meaning of a word can change across sentences, domains, and communities. Negation, ambiguous references, sarcasm, spelling variation, and mixed languages can challenge a system that appears accurate on tidy examples. Build evaluation material from the conditions the application actually encounters. A working NLP application also needs rules for input length, document boundaries, and uncertainty. Check whether truncation silently removes important sections. Preserve the original passage next to extracted information so a reader can confirm the result. Compare against a simple rule or keyword baseline when the task is narrow enough for one.