Core ML

Sequential Recommendation

Predicting what a user wants next by analyzing the strict chronological order of their past actions, recognizing that buying a phone case usually happens *after* buying a phone, not before.

Traditional recommenders treat history as a bag of items. Sequential recommenders treat history as a timeline, understanding cause and effect.

Why Does This Exist?

Classical algorithms like Matrix Factorization view a user's history as an unordered bucket. If John bought a Phone, a Screen Protector, and a Phone Case, the algorithm just sees [Phone, Protector, Case]. It doesn't care what order they happened in.

This causes a massive problem. If John just bought a Phone Case, what should we recommend next? A classical model will say: "People who buy Phone Cases also buy Phones. Recommend a Phone!" This is stupid. John already bought the phone. That's why he bought the case.

Sequential Recommendation models treat user history as a strict chronological timeline. They understand that $A \rightarrow B \rightarrow C$ is fundamentally different from $C \rightarrow B \rightarrow A$ .

Think of It Like This

Imagine predicting the next word in a sentence.

Classical Model: The sentence contains the words [The, Barked, Dog]. Based on those three words, the next word is probably [Loudly]. It completely ignores grammar and order.

Sequential Model (LLM): The sentence is strictly "The Dog Barked". Therefore, the next word is probably "Loudly".

Sequential Recommendation is literally just applying ChatGPT-style Natural Language Processing to e-commerce. Instead of sentences made of words, we process sequences made of products.

How It Actually Works

For years, sequential recommendation was done using Recurrent Neural Networks (RNNs) and Markov Chains. Today, it is almost entirely dominated by Transformers (the exact same architecture used in Large Language Models).

SASRec (Self-Attentive Sequential Recommendation)

Introduced in 2018, SASRec treats the user's click history as a sentence. It uses the Attention Mechanism to look back at the user's history and figure out which past clicks matter right now.

If you are looking at running shoes, SASRec will pay "attention" to the fact that you bought a water bottle 3 days ago, but it will ignore the fact that you bought a TV 6 months ago.

BERT4Rec

Inspired by Google's BERT model, BERT4Rec takes the user's history and randomly "masks" (hides) some of the items. It then forces the neural network to guess the hidden items. By doing this thousands of times, the model develops a deep, bidirectional understanding of how items relate to each other over time.

Session-Based vs. User-Based

Sequential recommenders are often used to solve the Session-Based Recommendation problem. If you visit a website without logging in, the website has no idea who you are. They don't have a User Profile or an Embedding for you. All they have is the 4 things you clicked on during your current 5-minute session. A sequential model can look at those 4 clicks in order, deduce your immediate intent, and predict your 5th click, without ever needing to know your name.

Show Me the Code

Training a Transformer for recommendation is complex, but here is the conceptual flow using PyTorch.

1import torch2import torch.nn as nn3
4# 1. Define a simple Transformer for Sequences5class SASRec(nn.Module):6    def __init__(self, item_count, max_sequence_length, embed_size):7        super().__init__()8        # Convert item IDs into dense vectors9        self.item_embedding = nn.Embedding(item_count, embed_size)10        # Give the model a sense of time (position in the sequence)11        self.position_embedding = nn.Embedding(max_sequence_length, embed_size)12        13        # The core Transformer block14        self.transformer = nn.TransformerEncoderLayer(d_model=embed_size, nhead=2)15        16    def forward(self, user_history_sequence):17        # user_history_sequence: [Batch Size, Sequence Length] (e.g. 50 past clicks)18        positions = torch.arange(user_history_sequence.size(1))19        20        # Add the item vector and the time vector together21        x = self.item_embedding(user_history_sequence) + self.position_embedding(positions)22        23        # Pass through the transformer to learn the sequential patterns24        output = self.transformer(x)25        26        # Predict the next item27        return output28
29# 2. To use it, you feed in a chronological list of item IDs30# E.g., User clicked Item 4, then 12, then 99. Predict the next one.31history = torch.tensor([[4, 12, 99]])

Watch Out For

Extremely high computational cost. Transformers scale quadratically with sequence length. If you try to feed a user's entire 10-year purchase history (10,000 items) into a Transformer, your servers will melt. You must truncate the sequences (e.g., only look at the last 50 clicks) to keep the system performant in real-time.

The Quick Version

Classical recommenders ignore time. They assume buying a case and then a phone is the same as buying a phone and then a case.
Sequential Recommendation treats history as a chronological timeline, allowing the model to deduce immediate, short-term intent.
The industry standard architectures are Transformers (SASRec, BERT4Rec), effectively treating e-commerce exactly like a language translation problem.
Sequential models are especially useful for Session-Based recommendations where the user is anonymous.

Related concepts

The Cold Start Problem