Parametric vs Non-Parametric

The Short Answer

The terms can be confusing because "non-parametric" doesn't mean the model has no parameters. It means the number of parameters is not fixed in advance; it depends entirely on the size of the training data.

Parametric Models have a fixed number of parameters (weights). Once you define the model architecture (e.g., a neural network with 1 million parameters, or a linear regression with 5 weights), the model size stays the same whether you train it on 10 examples or 10 billion examples.

Non-Parametric Models do not have a fixed set of parameters. Instead, their complexity grows as the data grows. In many cases, the training data is the model. When you add more data, the model literally gets bigger.

Where They Differ

Feature	Parametric	Non-Parametric
Model Size	Fixed in advance.	Grows with the data.
Training Speed	Usually slower (requires optimization loops).	Usually faster (often just storing data).
Inference Speed (Prediction)	Fast (just run the input through the fixed weights).	Slow (must compare the input against the entire stored dataset).
Flexibility	Limited by the chosen architecture. Can underfit if the true pattern is too complex.	Highly flexible. Can mold to any shape given enough data.
Examples	Linear/Logistic Regression, Neural Networks, Naive Bayes.	K-Nearest Neighbors (KNN), Decision Trees, Support Vector Machines (RBF kernel).

Architectural Consequences

The Parametric Workflow

During training, a parametric model looks at a piece of data, adjusts its fixed internal weights ($\theta$) slightly to reduce error, and then throws the data away. The data is distilled into the weights. By the time it's deployed, the model contains zero actual training examples, just the mathematical "essence" of what it learned.

The Non-Parametric Workflow

During training, a non-parametric model like K-Nearest Neighbors doesn't adjust mathematical weights. It just memorizes the data. When it's time to make a prediction, it looks at the new input, searches through its entire memory of training data to find the most similar examples, and bases its prediction on them.

Choose A When

(When to use Parametric Models)

• You have massive datasets: Distilling 1 billion rows into a 10MB model of weights makes prediction lightning fast and memory efficient.
• Inference latency is critical: A neural network (parametric) takes the same amount of time to predict regardless of how much data it was trained on.
• You have strong domain knowledge: You know roughly the shape the function should take and can design an architecture to match it.

Choose B When

(When to use Non-Parametric Models)

• You have a smaller dataset: Non-parametric models won't buckle under massive memory requirements if the data is small.
• You have no idea what the data distribution looks like: Non-parametric models make very few assumptions about the shape of the data; they are infinitely flexible.
• Explainability is paramount: With KNN (non-parametric), you can literally point to the 5 historical examples that caused the model to make its current prediction.

What People Get Wrong

Thinking Neural Networks are non-parametric

Because deep learning models are incredibly flexible, people assume they are non-parametric. They are strictly parametric. A ResNet-50 model always has ~25 million parameters. If you train it on 1 image or 1 trillion images, it still has exactly 25 million parameters.

Assuming non-parametric means simple

Non-parametric models can be incredibly complex. A deep Decision Tree can carve up feature space into highly intricate, jagged boundaries that a simple parametric Logistic Regression could never learn.