Choosing Activation Functions for a Neural Network

ByJu Yeon Eum

When building a neural network, one important design decision is choosing the activation function for each layer. The hidden layers and output layer do not necessarily need to use the same function.

Although the sigmoid function is a familiar choice, other activation functions—particularly tanh and ReLU—often allow neural networks to learn more effectively.

Activation Functions in Forward Propagation

For a layer \(l\), forward propagation consists of two steps. First, the network performs a linear calculation:\[ Z^{[l]} = W^{[l]}A^{[l-1]} + b^{[l]} \]

It then applies an activation function:\[ A^{[l]} = g^{[l]}(Z^{[l]}) \]

Here, \(g^{[l]}\) represents the activation function used in layer \(l\). Different layers can use different activation functions, so \(g^{[1]}\) does not have to be the same as \(g^{[2]}\).

The activation function is generally nonlinear. This nonlinearity is essential because it allows the network to learn relationships that cannot be represented by a purely linear model.

The Sigmoid Activation Function

The sigmoid function is defined as:\[ a = \sigma(z) = \frac{1}{1+e^{-z}} \]

Its output ranges from 0 to 1:\[ 0 < \sigma(z) < 1 \]

This range makes sigmoid a natural choice for the output layer of a binary classification model. If the target \(y\) is either 0 or 1, the prediction \(\hat{y}\) can be interpreted as a probability:\[ \hat{y} = P(y=1 \mid x) \]

For example, the output layer might use:\[ A^{[2]} = \sigma(Z^{[2]}) \]

The sigmoid function, however, is generally not the best choice for hidden layers.

Limitations of Sigmoid

Sigmoid activations are always positive and usually do not have a mean close to zero. Their values may instead be centered around approximately \(0.5\). This can make optimization in the following layer more difficult.

A more significant problem occurs when \(z\) is very large or very small. At both ends of the function, its slope becomes close to zero:\[ \sigma'(z) = \sigma(z)(1-\sigma(z)) \]

When the derivative is very small, the gradients propagated through the network also become small. Gradient descent then makes only tiny parameter updates, slowing the learning process.

For these reasons, sigmoid is usually avoided in hidden layers. Its most important use is the output layer of a binary classification model.

The Hyperbolic Tangent Function

The hyperbolic tangent, or tanh, activation function is:\[ a = \tanh(z) \]

Its formula is:\[ \tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}} \]

Unlike sigmoid, tanh produces values between \(-1\) and \(1\):\[ -1 < \tanh(z) < 1 \]

The tanh function can be viewed as a shifted and rescaled version of sigmoid. While sigmoid is centered above zero, tanh passes through the origin:\[ \tanh(0)=0 \]

Why Tanh Can Work Better Than Sigmoid

Because tanh produces both positive and negative values, the activations in a hidden layer are more likely to have a mean close to zero.

This has an effect similar to centering input data before training. A more zero-centered activation distribution can make learning easier for the next layer.

For hidden units, tanh therefore almost always works better than sigmoid:\[ A^{[1]} = \tanh(Z^{[1]}) \]

A network used for binary classification might combine tanh and sigmoid:\[ A^{[1]} = \tanh(Z^{[1]}) \]\[ A^{[2]} = \sigma(Z^{[2]}) \]

In this example, tanh is used in the hidden layer, while sigmoid is used in the output layer to produce a value between 0 and 1.

Limitation of Tanh

Although tanh is generally better than sigmoid for hidden layers, it has a similar saturation problem. When \(z\) becomes very positive or very negative, the slope of tanh approaches zero:\[ \frac{d}{dz}\tanh(z)=1-\tanh^2(z) \]

As a result, tanh can also produce very small gradients and slow down gradient descent.

The ReLU Activation Function

The Rectified Linear Unit, commonly called ReLU, is defined as:\[ a = \operatorname{ReLU}(z)=\max(0,z) \]

This means:\[ \operatorname{ReLU}(z)= \begin{cases} z, & z>0 \\ 0, & z\leq 0 \end{cases} \]

Its derivative is:\[ \operatorname{ReLU}'(z)= \begin{cases} 1, & z>0 \\ 0, & z<0 \end{cases} \]

The derivative is technically undefined at \(z=0\). In practice, this does not cause a meaningful problem because the probability of obtaining exactly \(z=0\) is extremely small. An implementation can define the derivative at zero as either 0 or 1.

Why ReLU Is a Common Default

ReLU is one of the most widely used activation functions for hidden layers. Its primary advantage is that its derivative remains equal to 1 throughout the positive region.

Unlike sigmoid and tanh, ReLU does not saturate when \(z\) becomes strongly positive. This reduces the problem of gradients approaching zero and often allows neural networks to learn much faster.

It is true that the ReLU derivative is zero whenever \(z\) is negative. Nevertheless, enough hidden units generally receive positive values for learning to continue effectively across most training examples.

If there is no strong reason to choose another hidden-layer activation, ReLU is usually the recommended default:\[ A^{[l]} = \operatorname{ReLU}(Z^{[l]}) \]

The Leaky ReLU Activation Function

One disadvantage of ReLU is that its output and derivative are both zero for negative inputs. Leaky ReLU modifies the negative part of the function by giving it a small nonzero slope:\[ a = \operatorname{LeakyReLU}(z)=\max(0.01z,z) \]

Equivalently:\[ \operatorname{LeakyReLU}(z)= \begin{cases} z, & z>0 \\ 0.01z, & z\leq 0 \end{cases} \]

Its derivative is:\[ \operatorname{LeakyReLU}'(z)= \begin{cases} 1, & z>0 \\ 0.01, & z<0 \end{cases} \]

Because the negative side has a small slope, gradients can continue flowing even when \(z\) is negative.

Leaky ReLU sometimes performs better than standard ReLU, although it is used less frequently. In many applications, either choice can work well.

The value \(0.01\) is not a universal constant. It can be replaced with another small number or even treated as a parameter that the learning algorithm determines.

Comparing the Main Activation Functions

Sigmoid

\[ g(z)=\frac{1}{1+e^{-z}} \]

  • Produces values between 0 and 1.
  • Is a natural output activation for binary classification.
  • Is generally not recommended for hidden layers.
  • Can produce very small gradients for large positive or negative inputs.
  • Does not produce zero-centered activations.

Tanh

\[ g(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}} \]

  • Produces values between \(-1\) and \(1\).
  • Usually works better than sigmoid in hidden layers.
  • Produces activations that are more closely centered around zero.
  • Can still suffer from very small gradients when the input magnitude is large.

ReLU

\[ g(z)=\max(0,z) \]

  • Is the most common default choice for hidden layers.
  • Has a derivative of 1 for positive inputs.
  • Often enables faster learning than sigmoid or tanh.
  • Has a zero derivative for negative inputs.

Leaky ReLU

\[ g(z)=\max(0.01z,z) \]

  • Preserves a small gradient for negative inputs.
  • Can sometimes outperform standard ReLU.
  • Is a reasonable alternative when ordinary ReLU does not work well.

Practical Rules of Thumb

For binary classification, sigmoid is a natural choice for the output layer:\[ g^{[L]}(z)=\sigma(z) \]

For hidden layers, ReLU is generally the best starting point:\[ g^{[l]}(z)=\operatorname{ReLU}(z) \]

Tanh remains a reasonable hidden-layer alternative, especially when zero-centered activations are desirable. Leaky ReLU can also be tested when the zero-gradient region of standard ReLU appears problematic.

A typical binary classification network might therefore use:\[ A^{[1]} = \operatorname{ReLU}(Z^{[1]}) \]\[ A^{[2]} = \sigma(Z^{[2]}) \]

The hidden layer uses ReLU to support efficient learning, while the output layer uses sigmoid to produce a probability between 0 and 1.

The Best Choice Depends on the Problem

Neural network design involves many choices, including:

  • The number of hidden layers
  • The number of units in each layer
  • The activation function for each layer
  • The method used to initialize the weights
  • Other optimization and architectural settings

It is difficult to know in advance which combination will perform best for a particular application. General guidelines provide a useful starting point, but each dataset and problem has its own characteristics.

If the best activation function is unclear, a practical approach is to test multiple choices—such as ReLU, Leaky ReLU, and tanh—and compare their performance on a development or validation set.

The activation function that performs best on this held-out data is usually the most appropriate choice for the application. Testing alternatives also makes the architecture less dependent on rigid rules that may not apply to every problem or remain optimal as deep-learning techniques evolve.

Key Takeaway

Activation functions can differ between layers, and their selection can have a major effect on training speed and model performance.

The general recommendations are:

  • Use sigmoid primarily in the output layer for binary classification.
  • Use ReLU as the default activation function for hidden layers.
  • Consider tanh when zero-centered hidden activations may be helpful.
  • Try Leaky ReLU when maintaining a gradient for negative inputs is beneficial.
  • Compare multiple options on a development set when the best choice is uncertain.

Sigmoid, tanh, ReLU, and Leaky ReLU are all nonlinear activation functions. Their nonlinearity allows a neural network to learn complex relationships—a capability that would be lost if activation functions were removed entirely.

Similar Posts

Leave a Reply