I knew the core answer, stacked linear layers collapse into one linear transformation, but I fumbled explaining the 'why' clearly at first.
Start by explaining that replacing all activation functions with identity makes the entire network equivalent to a single linear transformation, regardless of depth. Then discuss the loss of non-linear representational power and when identity activations are intentionally used, such as in linear bottlenecks or residual connections.
Pro tip: Mention that identity activations can be useful in specific architectural components like linear layers in transformers or as skip connections, but they must be combined with non-linearities elsewhere to maintain expressive power.
Explain that a composition of linear functions is itself linear, so the deep network reduces to a single linear layer with weight matrix equal to the product of all layer weight matrices.
Since the network is linear, adding more layers does not increase representational power; the effective function is the same as a single-layer linear model.
Without non-linear activations, the network cannot approximate non-linear functions, limiting it to only linearly separable problems.
Identity activations are intentionally used in linear bottlenecks (e.g., in MobileNet), residual connections (to ease gradient flow), and in some attention mechanisms where linear projections are desired.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.