In this post you will learn that 1. You can initialize deep net with shallow net; 2. For such initialization, similar to this paper it can be shown that loss will tend to zero as network grows, without spin glass model; 3. It does not make sense to study local minimum quality alone; 4. You don’t want layers of your network to be too “thin”.

The local minimum problem, associated with the training of deep neural networks, is frequently viewed as their serous drawback. In this post I argue why with a supervised pretraining initialization and popular choices of neuron types this problem does not affect much training of large neural networks. I confirm my results with some experimental evaluation.

I explore some simple approaches for automatic learing of the training algorithms for nerual netwoks (as well as for other predicitve models). I show why this is a hard problem for standard approaches (in particular for deep learning) and what new insights can be learned while solving it. I continue with more dedicated approaches in part 2 of this post.

One popular view on developments in the field of AI is that at some point it would lead to a scary scenario when self improving AI rebels against humanity and takes over the planet in order to allocate more resources for itself. In this post I argue that under common assumptions a self improving AI is rather unlikely to do so.