自然语言处理基础知识

Tasks include classifying documents, finding named entities, translating text, retrieving information, and generating responses. Different tasks require different outputs and evaluation methods.

主要要点

• Define the language task precisely.
• Retain context and source passages.
• Evaluate realistic language variation.

Text must be represented in a form a computational system can process. Tokenization splits it into units such as words or word pieces; numerical representations then support rules, statistical models, or neural networks. Token boundaries are a modeling choice and do not always align with what a reader considers one word. Some tasks return a label for a whole document. Others identify spans inside it or produce a new sequence. A sentiment classifier, an entity recognizer, and a summarizer therefore solve different problems even if all use the same underlying language model. Context matters. The meaning of a word can change across sentences, domains, and communities. Negation, ambiguous references, sarcasm, spelling variation, and mixed languages can challenge a system that appears accurate on tidy examples. Build evaluation material from the conditions the application actually encounters. A working NLP application also needs rules for input length, document boundaries, and uncertainty. Check whether truncation silently removes important sections. Preserve the original passage next to extracted information so a reader can confirm the result. Compare against a simple rule or keyword baseline when the task is narrow enough for one.

1. Use the invented sentence “Mina at Northstar Labs said the delayed launch was disappointing.”
2. An entity task could mark Mina as a person and Northstar Labs as an organization. A sentiment task could classify the expressed reaction as negative.
3. A summary might state that Mina criticized a launch delay. Check that it does not invent the reason for the delay.

The same sentence supports different outputs; each needs its own correctness criteria.