Core Machine Learning

Problem framing, generalization, classic supervised and unsupervised models, features, validation, and leakage.

Problem Framing

Supervised: Learn a mapping from features X to labels y (classification or regression).
Unsupervised: Find structure without labels (clustering, density, dimensionality reduction).
Self-supervised: Create labels from the data itself, such as masked or next-token prediction.
Reinforcement learning: Learn a policy that maximizes reward through interaction.
Baseline first: A constant, rule, or linear model sets the bar every model must beat.

Define target
Collect data
Split
Baseline
Train + tune
Evaluate
Deploy + monitor

The ML workflow; most failures trace back to the first two steps

Bias, Variance, Generalization

E[(y − ŷ)²] = Bias² + Variance + σ²

σ² is irreducible noise; model choice trades the first two

Symptom	Diagnosis	Remedies
High train and val error	Underfitting (bias)	Richer model, better features
Low train, high val error	Overfitting (variance)	More data, regularize, simplify
Val good, prod bad	Distribution shift or leakage	Re-check splits and features

Read the learning curve

If train and validation error converge high, add capacity. If they stay far apart, add data or regularization.

Linear Regression

ŷ = w^T x + b

w = (X^T X)⁻¹ X^T y

Ordinary least squares: the normal equation minimizes squared error

Assumptions: Linear relation, independent errors, constant variance, little multicollinearity.
Coefficients: Change in ŷ per unit of a feature, holding the others fixed.
Scaling: Not needed for OLS itself; required for regularized or gradient-based fits.
Outliers: Squared loss is sensitive; consider Huber loss or robust regression.

Regularization

Ridge: min ‖y − Xw‖² + λ‖w‖₂²

Lasso: min ‖y − Xw‖² + λ‖w‖₁

λ controls strength; standardize features first

Ridge (L2): Shrinks all weights smoothly; stable with correlated features.
Lasso (L1): Drives some weights to exactly zero, giving feature selection.
Elastic Net: Mixes L1 and L2; keeps groups of correlated features together.

Logistic Regression

P(y = 1 | x) = σ(w^T x + b), σ(z) = 1 / (1 + e^(−z))

Loss = −[y log p + (1 − y) log(1 − p)]

Output: A calibrated-ish probability; pick the decision threshold from business costs.
Odds ratio: e^(wᵢ) is the multiplicative change in odds per unit of feature i.
Multiclass: Softmax regression or one-vs-rest.

Distance and Kernel Models

k-NN: Predicts from the k closest training points; no training step, slow inference.
Scaling matters: Distance-based models need standardized features.
SVM: Maximizes the margin between classes; only support vectors define the boundary.
Kernel trick: RBF or polynomial kernels fit nonlinear boundaries without explicit features.
C parameter: Large C fits training points tightly; small C widens the margin.

Trees and Ensembles

Decision tree: Greedy splits on Gini or entropy; interpretable, high variance alone.
Random forest: Bagged trees on bootstrap samples with random feature subsets.
Gradient boosting: Trees fit sequentially to the residuals (gradients) of the previous ensemble.
XGBoost / LightGBM / CatBoost: Fast, regularized boosting; strong default for tabular data.
Stacking: A meta-model learns how to combine base model predictions.

Gini = 1 − Σ pₖ²

Fₘ(x) = Fₘ₋₁(x) + η · hₘ(x)

η = learning rate; hₘ = tree fit to the current gradients

Naive Bayes

P(y | x) = P(x | y) · P(y) / P(x)

Naive assumption: Features are conditionally independent given the class.
Strengths: Very fast, works with little data, strong text baseline.
Smoothing: Laplace smoothing avoids zero probabilities for unseen words.

Clustering

Method	Shape assumption	Needs k?	Notes
k-means	Spherical, similar size	Yes	Fast; sensitive to scaling and init
DBSCAN	Density-connected	No	Finds noise; tune eps and min samples
Hierarchical	Any (by linkage)	No	Dendrogram; O(n²) memory
GMM	Elliptical	Yes	Soft assignments, probabilistic

Choosing k: Elbow on inertia, silhouette score, or downstream usefulness.

Dimensionality Reduction

PCA: Linear projection onto directions of maximum variance; standardize first.
Explained variance: Keep enough components to retain about 90–95% of variance.
t-SNE / UMAP: Nonlinear embeddings for visualization; distances between clusters are not meaningful.
Autoencoders: Learn a compressed representation with a neural network.

Feature Engineering

Numeric: Standardize, log-transform skewed values, clip extreme outliers.
Categorical: One-hot for low cardinality; target or hash encoding for high cardinality.
Dates: Extract day of week, month, recency, and time since an event.
Text: TF-IDF for classic models; embeddings for semantic tasks.
Interactions: Ratios and products often beat raw features for linear models.
Missing values: Impute and add an "is missing" indicator when absence is informative.

Class Imbalance

Use the right metric: PR-AUC, recall at fixed precision, or cost-weighted error, not accuracy.
Class weights: Penalize minority errors more inside the loss.
Resampling: Undersample the majority or oversample the minority (SMOTE) on train only.
Threshold tuning: Move the decision threshold instead of retraining.

Resample after splitting

Oversampling before the train/validation split copies minority rows into both sets and inflates scores.

Cross-Validation

k-fold: Rotate k validation folds; average the scores.
Stratified k-fold: Preserves class ratios per fold; default for classification.
Group k-fold: Keeps all rows of a user or entity in one fold.
Time series split: Train on the past, validate on the future, never shuffle.
Nested CV: Inner loop tunes, outer loop estimates the tuned model's performance.

Data Leakage

Target leakage: A feature is computed using information from after the prediction time.
Preprocessing leakage: Scalers or imputers fit on the full dataset instead of train only.
Duplicate leakage: The same entity appears in both train and test.
Temporal leakage: Random splits on time-ordered data.

Too good to be true usually is

A sudden jump in validation score is a leakage signal. Inspect the top features first.

Model Selection at a Glance

Situation	Good first choice
Small tabular data	Logistic / linear regression, random forest
Large tabular data	Gradient boosting (LightGBM, XGBoost)
Text classification	TF-IDF + linear model, then fine-tuned transformer
Images, audio	Pretrained CNN or transformer
Need explanations	Linear models, shallow trees, SHAP

Hyperparameter Intuition

Learning rate: Too high diverges, too low is slow; tune on a log scale.
Tree depth: Deeper fits interactions but overfits; boosting prefers shallow trees.
n_estimators: More trees help random forests; boosting needs early stopping.
Regularization λ / C: Trade training fit for generalization.
Search: Random or Bayesian search beats grid search in high dimensions.

Final Checklist

The target and prediction time are defined precisely.
The split matches how the model will be used (time, group).
A simple baseline is recorded.
All preprocessing is fit on the training fold only.
The metric reflects business costs, not just accuracy.
Results include uncertainty (CV spread or confidence interval).