I haven't seen anybody applying something like this approximate sequentially updated consensus ADMM to neural networks. Is it worth the effort & extra complexity ? Would it be much better than SGD with momentum & rmsprep ? I don't know.