Skip to content
AI360Xpert
Glossary
Definition

BERT

A bidirectional encoder-only transformer pretrained by masked language modeling: hide some tokens and predict them from context on both sides at once.

BERT reads a sentence with no causal mask, so every position attends freely left and right, and it trains by guessing tokens hidden behind a [MASK] symbol using that full bidirectional context. That's genuinely different from a left-to-right model, which can never see what comes after the word it's trying to predict.

It's still the right tool for classification, extraction, and any task that needs a representation of a complete input, as opposed to generating new text.