Language AI GUIDE
Named Entity Recognition
Named entity recognition, or NER, identifies spans of text that refer to categories such as people, organizations, and places.
Listen to this guideAI-narrated · ~2 min
On this page2 min read
Overview
It finds mentions under a chosen schema. Linking a mention to a particular real-world record is a separate entity-linking task.
Key takeaways
- Define types and span boundaries.
- Preserve offsets into the original text.
- Keep recognition separate from identity linking.
Deep Dive
Define the entity types and span rules before training or evaluation. Should a company suffix be included? Is a product an organization, a separate type, or outside the schema? Inconsistent annotation rules can make a dataset internally contradictory.
NER systems may assign token-level labels and combine adjacent tokens into spans. Subword tokenization requires care when aligning labels with the original text. Preserve character offsets so applications can show exactly which passage produced an extracted value.
Evaluate both boundaries and types. Identifying only “Northstar” when the annotated organization is “Northstar Research Labs” may count as a span error even if the general category is correct. Report the matching convention with precision and recall so scores can be interpreted.
Context can change the label. “Jordan” might identify a person, country, or organization in different passages. A recognized name is not verified identity information. When using extraction for redaction, search, or record matching, test the downstream outcome and handle ambiguous or missed mentions explicitly.
04Worked example
Recognize a mention without inventing an identity
Use the invented sentence “Jordan joined Northstar Research Labs in June.”
Mark Jordan as a person mention and Northstar Research Labs as an organization mention under a documented schema.
Do not attach a particular biography or company registration unless a separate linking step has evidence for that match.
What it shows
The constructed example separates locating a name from resolving who or what it identifies.
Real-World Implementation
Highlight organizations mentioned in a news article with original text offsets.
Build a review queue for possible names before approving a redacted document.
Putting it into practice
Why it matters
- Speed and scale. Language workflows can move faster without sacrificing consistency.
- Access and reach. It expands access across languages and communication styles.
- Clearer decisions. Teams can spend more time on judgment while automation handles repetition.
Watch out for
- Hallucinated facts can quietly enter reports, support flows, or research outputs.
- Prompt sensitivity can create inconsistent results across similar requests.
- Sensitive text data may be exposed if access controls are weak.
Next steps
- Define output format, tone, and quality standards before rollout.
- Ground responses with trusted sources whenever accuracy matters.
- Keep a human review checkpoint for high-stakes outputs.
- Track failure patterns and retrain prompts or workflows regularly.
Sources and further reading
- Hugging FaceToken classification
Keep Exploring
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Named Entity Recognition quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Frequently asked questions
Does finding a name prove who the person is?
No. A text mention can be ambiguous. Resolving it to a particular person requires additional evidence and a separate linking process.
Keep learning
Related guides
More guides picked for this topic