Weighted Sum¶
$$ z=x^{T}w$$
Heaviside Step Function¶
$$ heaviside(z) = \begin{cases}
0 & \text{if $z \lt 0$} \\
1 & \text{if $z \ge 0$}
\end{cases}$$
Sign Function¶
$$ sgn(z) = \begin{cases}
-1 & \text{if $z < 0$}\\
0 & \text{if $z = 0$} \\
+1 & \text{if $z > 0$}
\end{cases}$$
Logistic Function, Sigmoid Function¶
$$\sigma(z) = \frac{1}{1+exp(-z)}$$
Hyperbolic Tangent Function¶
- Output value range from -1 to 1
- Make each layer's output more or less centered aound 0 at the beginning of training, which helps speed up convergence
$$tanh(z) = 2\sigma(2z)-1$$
Rectified Linear Unit Function¶
- Not differentiable at z = 0
- Fast to compute
- Does not have a maximum ouput value which helps reduce some issues during Gradient Descent
$$ ReLU(z) = max(0, z)$$