Skip to content
AI360Xpert
Core ML

Named Entity Recognition

The task of scanning unstructured text to locate and classify proper nouns, such as people, organizations, dates, and locations.

NER acts like a highlighter, classifying individual tokens within a sequence to extract structured entities from raw text.
NER acts like a highlighter, classifying individual tokens within a sequence to extract structured entities from raw text.

Why Does This Exist?

Text classification can tell you that a news article is about "Business", but it cannot tell you which businesses were mentioned. If a financial analyst wants to build a dashboard tracking every time "Apple" or "Microsoft" is mentioned in the news, they need a system that extracts those specific entities from the text.

Named Entity Recognition (NER) bridges the gap between unstructured text and structured databases. It scans sentences and tags tokens that represent predefined categories, most commonly:

  • PER: Person (e.g., "Albert Einstein")
  • ORG: Organization (e.g., "Google", "United Nations")
  • LOC: Location (e.g., "Paris", "Mount Everest")
  • DATE: Temporal expressions (e.g., "July 4th", "tomorrow")

Think of It Like This

Think of It Like This

Imagine you are reading a long contract. You take out three highlighters: a yellow one for people's names, a blue one for company names, and a green one for dates and deadlines.

You read through the document, highlighting the specific words. NER is simply the automated version of those highlighters.

How It Actually Works

Unlike text classification, which assigns one label to an entire document, NER is a Token Classification task. It assigns a label to every single word in the sentence.

The BIO Tagging Format

Because entities often span multiple words (like "New York City"), the model needs a way to signify where an entity starts and ends. The industry standard is the BIO format (Begin, Inside, Outside).

If the sentence is "Tim Cook visited New York.", the tags would be:

  • Tim →\rightarrow B-PER (Begin Person)
  • Cook →\rightarrow I-PER (Inside Person)
  • visited →\rightarrow O (Outside any entity)
  • New →\rightarrow B-LOC (Begin Location)
  • York →\rightarrow I-LOC (Inside Location)
  • . →\rightarrow O

The Architecture

Historically, NER was solved using statistical sequence models like Conditional Random Fields (CRFs), which excel at understanding the dependencies between adjacent tags (e.g., an I-PER is extremely likely to follow a B-PER, but impossible to follow an O).

Today, Transformers (like BERT) dominate NER. The sentence is passed through the Transformer, yielding a contextual embedding for every token. A simple linear classification layer sits on top of each token's embedding, outputting the final BIO tag.

Show Me the Code

In Python, the spaCy library is the industry standard for fast, production-ready Named Entity Recognition.

import spacy
# Load the small English pipeline (contains a pre-trained NER model)# You must run `python -m spacy download en_core_web_sm` firstnlp = spacy.load("en_core_web_sm")
text = "Apple is looking at buying U.K. startup for $1 billion next Tuesday."
# Process the textdoc = nlp(text)
# Iterate over the extracted entitiesfor ent in doc.ents:    print(f"{ent.text:<12} | {ent.label_:<6} | {spacy.explain(ent.label_)}")
# -> Apple        | ORG    | Companies, agencies, institutions, etc.# -> U.K.         | GPE    | Countries, cities, states# -> $1 billion   | MONEY  | Monetary values, including unit# -> next Tuesday | DATE   | Absolute or relative dates or periods

Watch Out For

Watch Out For

Ambiguity and Context. Is "Apple" a fruit or a company? Is "Washington" a person, a state, or a city? Classical keyword-matching systems fail here. Modern NER models succeed because they look at the surrounding context (e.g., "Apple announced..." implies the company). If a sentence lacks context, the model will struggle.

Watch Out For

Capitalization Dependency. Many NER models rely heavily on capitalization as a feature. If you feed them lowercase social media text or OCR outputs where capitalization is broken, their accuracy drops significantly. You may need to fine-tune a model specifically on lowercase data if your domain requires it.

The Quick Version

  • Named Entity Recognition (NER) extracts structured entities (People, Organizations, Locations) from unstructured text.
  • It is framed mathematically as a Token Classification problem, where every word receives a label.
  • Multi-word entities are handled using the BIO (Begin, Inside, Outside) tagging scheme.
  • It is the foundational step for building Knowledge Graphs and populating databases from text documents.

Related concepts