ReLU
Rectified Linear Unit is the most widely used activation function in deep learning, which outputs the input directly if positive, and exactly zero otherwise.
Think of It Like This
Like a strict bouncer at a club who lets everyone with money in normally, but instantly kicks out anyone carrying a negative balance.
Mathematically defined as f(x) = max(0, x), ReLU revolutionized deep learning by effectively solving the vanishing gradient problem that plagued Sigmoid and Tanh functions. Because its derivative is a constant 1 for positive values, gradients flow easily through deep networks. However, it can suffer from the 'dying ReLU' problem where neurons become permanently inactive.