We investigate the use of a distributed asynchronous algorithm utilizing infinitesimal perturbation analysis (IPA) gradient estimators for on-line optimization of tandem networks of queues. In our scheme, each queue has a processor that updates a control parameter associated with the queue according to a stochastic gradient algorithm driven by IPA estimates of the gradient of the performance measure. The update times of the processors are not synchronized. The processors also communicate results of computations with each other, and this communication involves delay. We give conditions under which the algorithm converges with probability one. In our proof of convergence we analyze a particular subsequence of the sequence of control parameters, and show that this subsequence behaves like a sequence generated by a centralized synchronous gradient algorithm which updates before the start of certain busy periods of the network, and with gradient estimates that are asymptotically unbiased.