Core ML

LIME

You cannot explain a complex model globally, but if you zoom in close enough on a single prediction, the boundary looks like a straight line. LIME builds a simple model to explain that local area.

Why Does This Exist?

Techniques like permutation-importance and partial-dependence-and-ice are great for answering global questions like, "Does Age matter?" But in the real world, explainability is often requested for local, specific events: "Why was my specific mortgage application denied?"

If the underlying model is a massive neural network, its decision boundary is a highly complex, multi-dimensional curve. There is no simple equation to explain that curve. However, calculus teaches us that if you zoom in close enough on any complex curve, it eventually looks like a straight, flat line.

LIME (Local Interpretable Model-agnostic Explanations) uses this concept to explain individual predictions. Instead of trying to explain the entire complex model, LIME zooms in on one specific prediction, generates a bunch of fake data points around it, and trains a simple, readable linear model (like Logistic Regression) just for that tiny neighborhood. You can then read the coefficients of that simple model to explain why the complex model made its decision.

Think of It Like This

Think of the global model like the surface of the Earth.

If you ask someone to describe the shape of the Earth globally, they have to use complex geometry to describe a sphere with mountains and valleys.

But if you zoom in on a single person standing in a parking lot, the ground beneath their feet is perfectly flat. If you ask them to describe the Earth locally, they can just draw a straight line. LIME ignores the mountains and valleys of the global model; it just maps the flat parking lot underneath the one specific prediction you care about.

How It Actually Works

LIME is a post-hoc, model-agnostic surrogate explainer. It never looks inside the original model; it only interacts with its inputs and outputs.

1. Select the Target

You select the single prediction you want to explain. For example, John Doe, who was denied a loan.

2. Generate Neighborhood Data (Perturbation)

LIME takes John Doe's data and tweaks it slightly thousands of times to create fake "neighborhood" data. It might change his age from 30 to 32, or his income from $50k to$ 49k.

LIME then feeds all 1,000 of these fake, perturbed data points into the original Black Box model and records the predictions.

3. Weight the Samples

LIME calculates how far each fake data point is from the original John Doe. Fake data points that are very close to John Doe (e.g., age 31) are given a high weight. Fake data points that drifted far away (e.g., age 80) are given a very low weight.

4. Train the Surrogate Model

LIME trains a completely new, simple model (usually a linear regression or a shallow decision tree) on this fake dataset. The simple model is forced to pay the most attention to the heavily weighted points closest to John Doe.

Because it is a linear model, you can instantly read its weights. If the linear model says that Debt has a massive negative coefficient in this specific neighborhood, you can confidently tell John Doe: "Your loan was denied because your debt is too high."

Show Me the Code

The lime Python package makes this process incredibly straightforward, wrapping the perturbation and surrogate training into a single call.

1import lime2import lime.lime_tabular3
4def explain_prediction_with_lime(5    training_data: np.ndarray, 6    feature_names: list[str], 7    black_box_model, 8    target_instance: np.ndarray9):10    """11    Uses LIME to explain a single prediction from a black-box model.12    """13    # 1. Initialize the explainer with the training distribution14    explainer = lime.lime_tabular.LimeTabularExplainer(15        training_data=training_data,16        feature_names=feature_names,17        class_names=['Denied', 'Approved'],18        mode='classification'19    )20    21    # 2. Explain the specific instance22    # We pass the black_box_model's predict_proba function to LIME23    explanation = explainer.explain_instance(24        data_row=target_instance, 25        predict_fn=black_box_model.predict_proba,26        num_features=5 # Show me the top 5 reasons27    )28    29    # 3. Print the human-readable explanation30    print("Local Explanation for this specific user:")31    for feature, weight in explanation.as_list():32        print(f"{feature}: {weight:.3f}")33
34# Output might look like:35# Debt > 40k: -0.420 (Pushed toward Denied)36# Income < 50k: -0.150 (Pushed toward Denied)37# Missed_Payments <= 0: +0.080 (Pushed toward Approved)

Watch Out For

Neighborhood Instability

Because LIME relies on random sampling to generate the fake neighborhood data, it is non-deterministic. If you run LIME on John Doe today, it might say Debt is the biggest factor. If you run it on John Doe tomorrow, it might say Income is the biggest factor, simply because the random sampling shifted slightly. You must configure the number of samples (num_samples) high enough to ensure the surrogate model is stable.

The Linearity Assumption

LIME fundamentally assumes that if you zoom in close enough, the model's decision boundary is linear. For highly chaotic neural networks, this is not always true. If the boundary is a sharp zigzag even at the microscopic level, LIME's linear surrogate model will fail to fit the fake data accurately, resulting in a garbage explanation.

The Quick Version

While global explainability tells you what the model cares about on average, local explainability tells you why the model made a specific prediction.
LIME explains a single prediction by zooming in on it and pretending the complex model is actually a simple linear model in that tiny neighborhood.
It achieves this by generating thousands of slightly modified fake data points, passing them through the black box, and training a linear regression on the results.
The weights of that simple linear regression serve as the explanation for the user.
LIME is fast and model-agnostic, but its reliance on random sampling can make its explanations unstable if not configured correctly.