The mathematics behind the Neural Network allow the vanish gradient problem to take place. A fix would be to have other activation- and errorfunctions (?)

To allow that, #2 will need to be fixed as well.

See #6, different initializations will also reduce the risk of this problem.