Parameter servers and distributed deep networks
Distributed learning divides a training task across workers, but the workers still need an agreed way to handle model parameters. A parameter server separates work on observations from the management of shared parameters. Workers calculate contributions using their data; server nodes hold the model state that those contributions change.
This organisation makes communication and consistency explicit parts of the learning procedure. The system must define what a worker reads, how its update reaches shared state and how independent updates interact. These decisions are especially visible when deep networks and large training collections place pressure on both computation and memory.
Data parallelism separates observations
Wikipedia’s data-parallelism article describes distributing data across processors or nodes that work on those data in parallel. In a learning setting, workers can process different portions of a training collection while contributing to a common model. Each worker’s local work is useful only in relation to an update rule that combines or applies those contributions.
Data parallelism should be distinguished from model parallelism. The first divides the observations processed; the second divides the model’s computation or state. A model can be too large to place conveniently on each worker even when the training data are already partitioned. These are different constraints, so the placement of examples and the placement of parameters should be considered separately.
Workers and parameter nodes
The USENIX paper Scaling Distributed Machine Learning with the Parameter Server proposed a framework in which worker nodes held data and workloads, while server nodes maintained globally shared parameters as dense or sparse vectors and matrices. It described asynchronous communication, flexible consistency models, elastic scalability and continuous fault tolerance.
The word server names a role in that architecture rather than a requirement that one machine contain every parameter. The important separation is between performing local calculations and managing shared state. A worker reads the information needed for its calculation, processes observations and communicates its contribution. The parameter-management side supplies a defined place for updates to meet.
Dense and sparse representations have different communication implications. A dense update can involve a broad portion of the parameter space. A sparse update touches only selected entries. The representation must retain enough information to associate each contribution with the correct parameter. Compact messages help only when the update semantics still match the intended learning algorithm.
Consistency changes the meaning of an update
In a synchronous arrangement, workers meet at a defined boundary before the next stage begins. An asynchronous arrangement allows communication and computation to overlap. A worker may then calculate using parameter values that precede changes already made elsewhere. That possibility makes the freshness of a read part of the algorithm’s practical behaviour.
Flexible consistency is therefore a choice with statistical consequences, not just a transport option. The system needs a rule for the relationship between the state a worker reads and the state its update changes. An analysis of convergence depends on the assumptions made about that relationship. Comparing implementations without describing the update rule can hide an important difference between the procedures being compared.
DistBelief and distributed deep networks
The NIPS 2012 paper Large Scale Distributed Deep Networks described DistBelief as a framework for training large models on computing clusters. Within it, the paper introduced Downpour SGD, an asynchronous procedure supporting model replicas, and Sandblaster, a framework for distributed batch optimisation procedures. These descriptions show that distributed deep learning included several approaches to organising optimisation.
A replica lets a worker perform calculations with a local view of model parameters. Replication does not itself establish how agreement is maintained; that depends on the surrounding communication and consistency rules. Batch optimisation and stochastic updates also ask different things of the system. The optimisation page explains why the expense and frequency of updates matter alongside their mathematical form.
Failure, restart and elastic resources
Fault tolerance concerns how a system continues meaningful work despite faulty components. In distributed training, a worker or parameter node can stop independently of the rest of the job. Restart behaviour must account for model state and work already performed. Repeating an operation may require attention to whether its contribution has already been applied.
Wikipedia’s cloud-computing article describes network access to a scalable, elastic pool of shared resources. A training cluster using such resources still needs a way to respond when capacity changes. Additional workers alter the distribution of work and can change the pattern of updates arriving at parameter nodes. Elastic capacity is useful only when the learning procedure and its state-management rules can accommodate it.
Communication belongs in the model
The complete cost of a learning update includes reading parameters, calculating a contribution and moving that contribution to the appropriate destination. Larger messages, uneven data partitions and frequently accessed parameters can each concentrate work. Counting workers alone does not describe this arrangement; the important question is what those workers require from the shared state.
Collective operations such as AllReduce combine contributions through a different communication pattern, while graph-parallel systems organise dependencies around relationships. A parameter server gives another way to express distributed state. Its value is assessed through the needs of the model, the consistency assumptions of the algorithm and the communication and restart behaviour of the complete system.