Asymptotic Network Independence in Distributed Stochastic Optimization for Machine Learning.

Journal: IEEE signal processing magazine
Published Date: May 6, 2020

Abstract

We provide a discussion of several recent results which, in certain scenarios, are able to overcome a barrier in distributed stochastic optimization for machine learning. Our focus is the so-called asymptotic network independence property, which is achieved whenever a distributed method executed over a network of nodes asymptotically converges to the optimal solution at a comparable rate to a centralized method with the same computational power as the entire network. We explain this property through an example involving the training of ML models and sketch a short mathematical analysis for comparing the performance of distributed stochastic gradient descent (DSGD) with centralized stochastic gradient decent (SGD).

Authors

  • Shi Pu
    Institute for Data and Decision Analytics, The Chinese University of Hong Kong, Shenzhen, China and Shenzhen Research Institute of Big Data. The research was conducted when the author was with Division of Systems Engineering, Boston University, Boston, MA.
  • Alex Olshevsky
    Department of Electrical and Computer Engineering and Division of Systems Engineering, Boston University, Boston, MA.
  • Ioannis Ch Paschalidis
    Department of Electrical and Computer Engineering and Division of Systems Engineering, Boston University, Boston, MA.

Keywords

No keywords available for this article.

External Resources

Popular Topics
Recent Journals