Core ML

Off-Policy Evaluation

Guessing how much money a new algorithm will make, using only the historical data generated by an old algorithm, without actually deploying the new algorithm to real users.

In OPE, you look at historical clicks and ask: 'Would my NEW model have shown this exact same item?' If yes, you count the click. If no, you ignore it.

Why Does This Exist?

In Reinforcement Learning and Recommender Systems, we have a "Policy" (the algorithm that decides what to show the user).

On-Policy Evaluation: You deploy the new Policy to production (A/B testing). You watch real users interact with it, and you measure the revenue. This is 100% accurate, but it is slow, expensive, and risks angering users if the new Policy is bad.
Off-Policy Evaluation (OPE): You test the new Policy offline using historical data gathered by an old Policy. This is instant and free, but mathematically extremely difficult.

Why is OPE difficult? Because of counterfactuals. Suppose the Old Policy showed the user the movie Batman, and the user clicked it. Now you test the New Policy. The New Policy says, "I would have shown the user Superman." Did the user click Superman? You have no idea. The user was never shown Superman. The historical data cannot tell you what would have happened in an alternate universe.

Think of It Like This

Imagine you are evaluating a new coach for a basketball team.

On-Policy (A/B Test): You hire the coach, let them coach 10 real games, and see if they win. (High risk, high accuracy).

Off-Policy (OPE): You make the new coach watch a tape of last year's games (coached by the old coach). Every time the old coach makes a substitution, you pause the tape and ask the new coach, "What would you do?" If the new coach says the exact same thing the old coach did, you press play and see what happened. If the new coach says something different, you have to throw the tape away, because you can't know what would have happened.

How It Actually Works: Inverse Propensity Scoring (IPS)

The most common way to solve OPE is Inverse Propensity Scoring (IPS).

If we want to estimate how good the New Policy ( $\pi_{new}$ ) is using data from the Old Policy ( $\pi_{old}$ ), we look at the historical logs. Every time the Old Policy showed an item and got a click (Reward = 1), we check what the New Policy would have done.

$\text{Estimated Value} = \text{Reward} \times \frac{\text{Probability New Policy shows it}}{\text{Probability Old Policy showed it}}$

The Math in Action

The Agreement: The Old Policy showed Batman (10% chance). The user clicked it. The New Policy loves Batman, and would have shown it with a 90% chance.
- Score = $1 \times (0.90 / 0.10) = 9$ .
- The New Policy gets a massive bonus because it aggressively recommended an item that we know the user likes.
The Disagreement: The Old Policy showed Superman (80% chance). The user clicked it. The New Policy hates Superman, and would have shown it with a 5% chance.
- Score = $1 \times (0.05 / 0.80) = 0.06$ .
- The New Policy gets almost no credit, because it basically refused to show an item we know the user likes.

Show Me the Code

Here is a simplified calculation of the IPS estimator.

1import numpy as np2
3# Historical Log Data (generated by the OLD policy)4# Format: [Item_Shown, Probability_Old_Policy, User_Clicked]5logs = [6    {"item": "Batman",   "prob_old": 0.2, "reward": 1},7    {"item": "Superman", "prob_old": 0.8, "reward": 0},8    {"item": "Barbie",   "prob_old": 0.5, "reward": 1},9]10
11# Our NEW Policy. What probability does it assign to these items?12new_policy_probs = {13    "Batman": 0.9,   # The new policy loves Batman14    "Superman": 0.1, # The new policy hates Superman15    "Barbie": 0.5    # The new policy is neutral on Barbie16}17
18# Calculate the IPS estimate19ips_estimates = []20for log in logs:21    prob_new = new_policy_probs[log["item"]]22    prob_old = log["prob_old"]23    reward = log["reward"]24    25    # Importance Sampling Weight26    weight = prob_new / prob_old27    ips_estimates.append(reward * weight)28
29# The final estimated value of the New Policy30estimated_value = np.mean(ips_estimates)31print(f"Estimated Value of New Policy: {estimated_value:.2f}")32# Output: Estimated Value of New Policy: 1.8333# Since the Old Policy average reward was 0.66 (2 clicks out of 3), 34# the New Policy is estimated to be nearly 3x better!

Watch Out For

High Variance and Exploding Weights. If the Old Policy almost never showed an item ( $Prob \approx 0.001$ ), but the New Policy loves it ( $Prob = 1.0$ ), the IPS weight becomes $1.0 / 0.001 = 1000$ . A single lucky click in the historical data will be multiplied by 1000, completely destroying your estimate. To fix this, in production you must use Clipped IPS (cap the maximum weight at 10) or Doubly Robust (DR) estimators, which combine IPS with a machine learning model to smooth out the variance.

The Quick Version

On-Policy Evaluation (A/B Testing) is accurate but expensive and slow.
Off-Policy Evaluation (OPE) attempts to evaluate a new algorithm instantly using historical logs generated by an old algorithm.
The biggest challenge is counterfactuals: we don't know what the user would have done if we showed them a different item.
Inverse Propensity Scoring (IPS) solves this by re-weighting the historical data. It rewards the new policy if it agrees with the old policy on successful items.
Standard IPS is highly unstable, so production systems use Clipped IPS or Doubly Robust estimators.