Logistic Regression & Decision Boundaries

In many machine learning tutorials, a decision boundary is drawn as a hard, thin line separating two classes. This is a useful lie, but it hides the true nature of models like Logistic Regression.

A logistic regression model does not output discrete classes (e.g., "cat" or "dog"). It outputs a continuous probability field. The boundary you see is just a human-selected threshold placed on top of that field.

The Probability Field

At its core, Logistic Regression is simply linear regression passed through a sigmoid function. The linear part ($w_1x_1 + w_2x_2 + b$) produces any real number from $-\infty$ to $+\infty$. The sigmoid function squashes that number into a range between 0 and 1:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

This produces a smooth gradient of probabilities. In the interactive visual above, this is represented by the continuous heatmap. Deep blue represents areas where the model is highly confident in the negative class ($p \approx 0$), and deep red represents areas where it is highly confident in the positive class ($p \approx 1$).

The Decision Contour

If the model outputs probabilities, where does the decision boundary come from? It comes from you.

By choosing a threshold (commonly 0.5), you declare: "Anything with a probability $\ge 0.5$ will be predicted as positive." The decision boundary is simply the contour line where the probability field exactly equals your chosen threshold.

Try moving the threshold slider in the simulation. Notice how the boundary moves, even though the underlying probability field and the model's weights have not changed at all!

Evaluation and Trade-offs

Because the boundary is a choice, metrics like Precision and Recall are properties of your threshold, not properties of your model.

• Precision: Of the points you flagged as positive, how many actually were?
• Recall: Of all the actual positive points, how many did you find?

By sliding the threshold, you trade one for the other. A high threshold means you only flag the points you are most confident about (high precision), but you will miss many true positives (low recall).

To evaluate the model independently of any specific threshold, we use the ROC Curve (Receiver Operating Characteristic) and the AUC (Area Under the Curve). The ROC curve plots the True Positive Rate against the False Positive Rate for every possible threshold simultaneously.