Polynomial Regression

Linear regression draws straight lines. If the true relationship in the data is a curve, a line will systematically miss the points, consistently under-predicting the middle and over-predicting the edges.

Bending the Line

We don't need a new algorithm to fit curves. We just add new columns to the data. By taking our existing $x$ and adding $x^2$ (degree 2) or $x^5$ (degree 5) as independent features, linear regression can draw parabolas and wavy lines without changing its underlying maths.

The Overfitting Trap

Give the model enough degrees and it will perfectly thread the needle through every single training point. It memorises the random noise rather than the underlying pattern, creating wild oscillations that will fail completely when predicting on unseen data.

Taming the Oscillations

If we apply regularisation (like Ridge or Lasso), we penalise the model for using large coefficients. This forces the wavy curve to smooth out, allowing us to include high-degree features safely without letting the model oscillate out of control.

Where It Breaks

You cannot ask the model which degree is best by checking its training error. Every time you add a degree, the training error drops, pulling you directly into overfitting. The only way to find the optimal degree is by measuring performance on a hold-out test set.

The Quick Version

• The problem: Straight lines underfit curved patterns.
• The fix: Add powers of $x$ ($x^2, x^3$) as new features.
• The trap: Too many degrees leads to wild overfitting.
• The remedy: Regularisation shrinks coefficients to smooth the curve.
• Failure: Training error always says "more degrees are better".

What to Read Next

• Pooling Layers
• Random Forests