Contextual Bandits

Why Does This Exist?

In classical E-Commerce, if you want to test two different recommendation algorithms, you run an A/B Test. You send 50% of your users to Algorithm A, and 50% to Algorithm B. You wait 2 weeks, look at the dashboard, realize Algorithm B is terrible, and turn it off.

This is incredibly wasteful. For 2 weeks, you intentionally gave 50% of your users a terrible experience just to prove a point. This lost you money.

A Multi-Armed Bandit (MAB) solves this. It doesn't wait 2 weeks. On Day 1, it starts at 50/50. By Day 3, it notices Algorithm A is winning, so it shifts traffic to 70/30. By Day 7, it's 95/5. It learns while it tests, minimizing your losses.

A Contextual Bandit goes one step further. It realizes that there is no single "best" algorithm. Algorithm A might be the best for Mobile users, but Algorithm B is the best for Desktop users. It uses the user's context (features) to decide which arm to pull.

Think of It Like This

The Explore-Exploit Dilemma

All Bandits are built on the fundamental tension of Reinforcement Learning:

• Exploitation: Doing what you currently believe is the best thing, to maximize short-term profit.
• Exploration: Trying something you think is sub-optimal, just to gather more data and prove yourself right or wrong.

If you only Exploit, you might get stuck in a local minimum forever (e.g., "I know showing the user Batman gets clicks, so I will only ever show them Batman"). If you only Explore, you are just showing random items and ruining the user experience.

How to balance it:

1. Epsilon-Greedy ($\epsilon$-greedy): The simplest approach. 90% of the time, show the best item. 10% of the time ($\epsilon = 0.1$), pick a completely random item.
2. Upper Confidence Bound (UCB): Pick items that have high scores, or items that you are highly uncertain about. Uncertainty is treated as a bonus.
3. Thompson Sampling: A Bayesian approach. You maintain a probability distribution for every item. You sample from the distribution, which naturally balances exploration (wide distributions) and exploitation (tall, narrow distributions).

Contextual Bandits in Recommender Systems

In modern recommenders (like Netflix or Spotify), the homepage is almost entirely driven by Contextual Bandits.

They don't use Bandits to pick individual movies. The action space (10,000 movies) is too large. Instead, they use Contextual Bandits as the Routing Layer.

• Arm 1: The "Because you watched Matrix" carousel.
• Arm 2: The "Trending Now" carousel.
• Arm 3: The "New Releases" carousel.
• Context: The User is a 25-year-old male who logged in at 2 AM on a Tuesday.

The Contextual Bandit looks at the Context, and predicts which of the 3 carousels has the highest probability of getting a click right now. It puts that carousel at the very top of the homepage.

Show Me the Code

Implementing a simple $\epsilon$-greedy Contextual Bandit.

1import numpy as np2
3class ContextualBandit:4    def __init__(self, n_arms, n_features, epsilon=0.1):5        self.epsilon = epsilon6        # We maintain a separate linear regression model for EVERY arm7        # weights shape: [n_arms, n_features]8        self.weights = np.zeros((n_arms, n_features))9        10    def select_arm(self, context_vector):11        """Decide which carousel to show the user."""12        13        # 1. EXPLORE: 10% of the time, pick a random carousel14        if np.random.random() < self.epsilon:15            return np.random.randint(len(self.weights))16            17        # 2. EXPLOIT: 90% of the time, predict the best carousel18        # We calculate the expected payout (dot product) for all arms based on this specific context19        expected_payouts = np.dot(self.weights, context_vector)20        21        # Pick the arm with the highest expected payout22        return np.argmax(expected_payouts)23        24    def update(self, arm_pulled, context_vector, reward):25        """Learn from the result (0 for ignore, 1 for click)."""26        # A highly simplified gradient descent step27        learning_rate = 0.0128        prediction = np.dot(self.weights[arm_pulled], context_vector)29        error = reward - prediction30        31        # Update the weights for the arm we actually pulled32        self.weights[arm_pulled] += learning_rate * error * context_vector33
34# Usage35bandit = ContextualBandit(n_arms=3, n_features=5)36user_context = np.array([1, 0, 0, 24, 1]) # e.g. [IsMobile, IsTablet, IsDesktop, Age, IsWeekend]37
38# The bandit decides which carousel to show39chosen_carousel = bandit.select_arm(user_context)40
41# The user either clicks it (1) or ignores it (0)42user_clicked = 1 43
44# The bandit learns from the interaction instantly45bandit.update(chosen_carousel, user_context, reward=user_clicked)

Watch Out For

The Quick Version

• A/B Tests are static, wasteful, and assume there is a single global winner.
• Multi-Armed Bandits learn dynamically, shifting traffic to the winner during the test to minimize wasted traffic.
• Contextual Bandits use user features to realize that different arms are optimal for different users.
• They constantly balance Exploitation (showing what works) with Exploration (trying new things to gather data).
• They are heavily used to rank UI components (like carousels) rather than individual items.

What to Read Next

• Off-Policy Evaluation
• A/B Testing for ML (Coming soon)