Very interesting topic. Indeed, one way to interpret the neural networks is to design the deep architecture interpretable at the first place. Namely, one can design the structure and activation functions of each layer to mimic one step of gradient descent. Then, K layer of this tailored DNN resembles K iterations of an optimization algorithm, e.g., see [1-3]. But still, one need to prove the convergence of such networks (empirical results show very promising convergence speed).
[1] https://arxiv.org/abs/1812.00797
[2] https://openreview.net/pdf?id=B1lnzn0ctQ
[3] https://papers.nips.cc/paper/8120-theoretical-linear-converg...