| # FAQs | |
| **Q:** If the weight of a conv layer is zero, the gradient will also be zero, and the network will not learn anything. Why "zero convolution" works? | |
| **A:** This is wrong. Let us consider a very simple | |
| $$y=wx+b$$ | |
| and we have | |
| $$\partial y/\partial w=x, \partial y/\partial x=w, \partial y/\partial b=1$$ | |
| and if $w=0$ and $x \neq 0$, then | |
| $$\partial y/\partial w \neq 0, \partial y/\partial x=0, \partial y/\partial b\neq 0$$ | |
| which means as long as $x \neq 0$, one gradient descent iteration will make $w$ non-zero. Then | |
| $$\partial y/\partial x\neq 0$$ | |
| so that the zero convolutions will progressively become a common conv layer with non-zero weights. | |