Why Neural Networks Need Nonlinear Activation Functions

A neural network needs nonlinear activation functions to learn complex relationships between its inputs and outputs. If every layer uses only a linear—or identity—activation function, adding more hidden layers does not make the model more expressive. No matter how deep the network becomes, it remains equivalent to a single linear model.

The Identity Activation Function

The identity activation function simply returns its input:\[ g(z)=z \]

It is also commonly called a linear activation function. When a layer uses this function, its activation is equal to its linear output:\[ A^{[l]}=g(Z^{[l]})=Z^{[l]} \]

Consider a two-layer neural network with the following forward-propagation equations:\[ Z^{[1]}=W^{[1]}X+b^{[1]} \]\[ A^{[1]}=g^{[1]}(Z^{[1]}) \]\[ Z^{[2]}=W^{[2]}A^{[1]}+b^{[2]} \]\[ A^{[2]}=g^{[2]}(Z^{[2]}) \]

Suppose both layers use the identity activation function. We then have:\[ A^{[1]}=Z^{[1]}=W^{[1]}X+b^{[1]} \]

and:\[ A^{[2]}=Z^{[2]}=W^{[2]}A^{[1]}+b^{[2]} \]

Substituting the first-layer activation into the second-layer equation gives:\[ A^{[2]} = W^{[2]}\left(W^{[1]}X+b^{[1]}\right)+b^{[2]} \]

Expanding the expression:\[ A^{[2]} = W^{[2]}W^{[1]}X + W^{[2]}b^{[1]} + b^{[2]} \]

We can define:\[ W’=W^{[2]}W^{[1]} \]

and:\[ b’=W^{[2]}b^{[1]}+b^{[2]} \]

The entire network then reduces to:\[ A^{[2]}=W’X+b’ \]

This is simply another linear function of the input \(X\).

Why Additional Linear Layers Do Not Help

A composition of linear functions is still a linear function. Therefore, stacking multiple linear layers does not allow the network to represent more complicated relationships.

For example, suppose a deeper network uses identity activations throughout:\[ A^{[1]}=W^{[1]}X+b^{[1]} \]\[ A^{[2]}=W^{[2]}A^{[1]}+b^{[2]} \]\[ A^{[3]}=W^{[3]}A^{[2]}+b^{[3]} \]

Even though the network contains several layers, repeated substitution will reduce the entire calculation to:\[ A^{[3]}=W’X+b’ \]

for some combined weight matrix \(W’\) and bias vector \(b’\).

The same result applies regardless of how many linear hidden layers the network contains. A network with ten linear layers is still equivalent to a model with one linear layer.

Consequently, linear hidden layers do not add meaningful expressive power. If every layer is linear, there is little reason to include hidden layers at all.

A Linear Hidden Layer Followed by Sigmoid

Suppose the hidden layer uses an identity activation, while the output layer uses sigmoid:\[ A^{[1]}=W^{[1]}X+b^{[1]} \]\[ A^{[2]}=\sigma\left(W^{[2]}A^{[1]}+b^{[2]}\right) \]

Substituting \(A^{[1]}\) gives:\[ A^{[2]} = \sigma\left( W^{[2]}W^{[1]}X + W^{[2]}b^{[1]} + b^{[2]} \right) \]

Combining the parameters again produces:\[ A^{[2]}=\sigma(W’X+b’) \]

This has the same basic form as logistic regression. Therefore, adding a linear hidden layer before a sigmoid output does not make the model more expressive than standard logistic regression without a hidden layer.

The problem is not the number of layers. The problem is that nothing nonlinear has been introduced between them.

How Nonlinearity Makes Neural Networks More Powerful

A hidden layer with a nonlinear activation performs:\[ A^{[1]}=g^{[1]}(W^{[1]}X+b^{[1]}) \]

where \(g^{[1]}\) might be ReLU, tanh, or Leaky ReLU.

The next layer receives this nonlinear transformation:\[ A^{[2]} = g^{[2]}\left(W^{[2]}A^{[1]}+b^{[2]}\right) \]

Because \(A^{[1]}\) is no longer simply a linear function of \(X\), the layers cannot be collapsed into a single weight matrix and bias vector.

This allows the network to represent complex nonlinear relationships. As more nonlinear layers are added, the network can build increasingly sophisticated transformations of the original input.

Common nonlinear activation functions include:

ReLU

\[ g(z)=\max(0,z) \]

Tanh

\[ g(z)=\tanh(z) \]

Leaky ReLU

\[ g(z)=\max(0.01z,z) \]

These functions prevent a multilayer network from collapsing into a single linear equation.

When a Linear Activation Function Is Appropriate

Although linear activations are generally not useful in hidden layers, they can be appropriate in the output layer of a regression model.

In regression, the target \(y\) may be any real number. For example, a model predicting a numerical quantity might need to produce values ranging from negative infinity to positive infinity.

A linear output layer can produce this unrestricted range:\[ \hat{y}=Z^{[L]} \]

or equivalently:\[ g^{[L]}(z)=z \]

This makes a linear output activation suitable for problems such as predicting temperature changes, financial values, or other continuous quantities that can be either positive or negative.

The hidden layers should still use nonlinear activation functions:\[ A^{[1]}=\operatorname{ReLU}(Z^{[1]}) \]\[ A^{[2]}=\tanh(Z^{[2]}) \]\[ \hat{y}=Z^{[L]} \]

Here, the hidden layers learn nonlinear representations, while the output layer converts the final representation into an unrestricted real-valued prediction.

Predicting Nonnegative Values

Some regression targets cannot be negative. Housing prices, for example, must be greater than or equal to zero.

Although a linear output layer can be used, it may occasionally produce negative predictions. If the output must be nonnegative, ReLU can be used in the output layer:\[ \hat{y}=\operatorname{ReLU}(Z^{[L]}) \]

Because:\[ \operatorname{ReLU}(z)=\max(0,z) \]

the resulting prediction always satisfies:\[ \hat{y}\geq 0 \]

The best output activation therefore depends on the range of the target:

• Use a linear output when \(y\) can be any real number.
• Use ReLU when \(y\) must be nonnegative.
• Use sigmoid when performing binary classification and the output should be between 0 and 1.

Rare Uses of Linear Hidden Layers

Linear activation functions in hidden layers are extremely uncommon because they do not increase the expressive power of the network.

There are some specialized applications related to dimensionality reduction or compression where linear hidden representations may be useful. However, these are exceptions rather than the usual way neural networks are constructed.

For most neural networks, the hidden layers should use nonlinear activations such as ReLU, tanh, or Leaky ReLU.

Key Takeaway

A neural network made entirely from linear layers is still only a linear model:\[ W^{[2]}(W^{[1]}X+b^{[1]})+b^{[2]} = W’X+b’ \]

Adding more linear layers cannot change this result. A deep linear network has no more representational power than a single linear layer.

Nonlinear activation functions prevent the layers from collapsing into one linear transformation. This allows neural networks to learn complex patterns and makes depth useful.

Linear activations still have an important role in regression output layers, especially when predictions may take any real value. In hidden layers, however, nonlinear activation functions are a critical part of an effective neural network.