Open Advanced Search

Log in
Enterprise Plans
Sign Up

Log in
Sign Up
Enterprise Plans
Browse
Features
Pricing

DeepDyve

Advanced Search

show/hide query preview

With all of the words or DOIs:

DeepDyve's default query mode: search by keyword or DOI.

With the exact phrase:

Require these words, in this exact order. Do not surround your terms in double-quotes ("") in this field.

With at least one of the words:

Find any of these words, separated by spaces

Without the words:

Exclude each of these words, separated by spaces

With words in the article title:

Search for these terms only in the title of an article

Published between the years:

With the following authors:

Most effective as: LastName, First Name or Lastname, FN

With words in the journal name:

Search for articles published in journals where these words are in the journal name

Reset filters

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

DeepDyve requires Javascript to function. Please enable Javascript on your browser to continue.

Optimization Algorithms for Distributed Machine Learning

Part of the Synthesis Lectures on Learning, Networks, and Algorithms Book Series

DOI: 10.1007/978-3-031-19067-4

ISBN: 978-3-031-19066-7

Publisher: Springer International Publishing - Springer Journals

Nov 26, 2022

1 - 9 of 9 Chapters

LitStream

Chapter 1Distributed Optimization in Machine Learning

Joshi, Gauri

Pages: 1 –12

[Machine learning is revolutionizing data-driven decision-making in a myriad applications including search and recommendations, self-driving cars, robotics, and medical diagnosis.]
LitStream

Chapter 2Calculus, Probability and Order Statistics Review

Joshi, Gauri

Pages: 13 –25

[In this chapter, we will begin by reviewing some concepts from calculus and probability that are essential to understanding the error and runtime analyses of various distributed SGD algorithms that we will study in the rest of the book.]
LitStream

Chapter 3Convergence of SGD and Variance-Reduced Variants

Joshi, Gauri

Pages: 27 –38

[In this chapter we will analyze the convergence of gradient descent (GD) and stochastic gradient descent (SGD) to determine the number of iterations required to reach a target error.]
LitStream

Chapter 4Synchronous SGD and Straggler-Resilient Variants

Joshi, Gauri

Pages: 39 –49

[For large training datasets, it can be prohibitively slow to conduct sequential SGD training at a single node, as we described in Chap. 1.]
LitStream

Chapter 5Asynchronous SGD and Staleness-Reduced Variants

Joshi, Gauri

Pages: 51 –66

[In Chap. we saw that synchronous distributed SGD suffers from tail latency due to straggling workers. Its straggler-resilient variants K-sync and K-batch-sync SGD overcome the straggling delay, but they end up discarding partially completed gradient computation at one or more workers, thus...
LitStream

Chapter 6Local-Update and Overlap SGD

Joshi, Gauri

Pages: 67 –92

[In the last two chapters, we studied synchronous and asynchronous SGD, where the workers compute mini-batch gradients and the parameter server aggregates them and updates the model. Both these algorithms and their variants require constant communication between the workers and the parameter...
LitStream

Chapter 7Quantized and Sparsified Distributed SGD

Joshi, Gauri

Pages: 93 –106

[The communication delay in sending and receiving gradients or model updates between the worker nodes and the parameter server can be significantly, especially in networks with limited bandwidth and high latency on the communication links.]
LitStream

Chapter 8Decentralized SGD and Its Variants

Joshi, Gauri

Pages: 107 –121

[In all the distributed SGD implementations that we studied so far, namely, synchronous SGD.]
LitStream

Chapter 9Beyond Distributed Training in the Cloud

Joshi, Gauri

Pages: 123 –127

[Let us summarize the concepts that we learned through this book, and discuss the future of distributed machine learning beyond cloud-based implementations that we studied in this book.]

results per page

Browse All eBooks

Features

Search and discover articles on DeepDyve, PubMed, and Google Scholar
Read the full-text of open access and premium content
Organize articles with folders and bookmarks
Collaborate on and share articles and folders

Info

Pricing
Enterprise Plans
Browse Journals & Topics
About DeepDyve

Help

Help
Publishers
Contact Us

Sign up for your DeepDyve account now!

Read and print from thousands of top scholarly journals.

System error. Please try again!

Continue with Facebook

By signing up, you agree to DeepDyve’s Terms of Service and Privacy Policy.

Already have an account? Log in

Save Article to Bookmarks

Bookmark this article. You can see your Bookmarks on your DeepDyve Library.

To save an article, log in first, or sign up for a DeepDyve account if you don’t already have one.

Subscribe to Journal Email Alerts

To subscribe to email alerts, please log in first, or sign up for a DeepDyve account if you don’t already have one.

Follow a Journal

To get new article updates from a journal on your personalized homepage, please log in first, or sign up for a DeepDyve account if you don’t already have one.

Our policy towards the use of cookies

All DeepDyve websites use cookies to improve your online experience. They were placed on your computer when you launched this website. You can change your cookie settings through your browser.

cookie policy