In the paper the problem of using a differential evolution algorithm for feed-forward neural network training is considered. A new parallelization scheme for the computation of the fitness function is proposed. This scheme is based on data decomposition. Both the learning set and the population of the evolutionary algorithm are distributed among processors. The processors form a pipeline using the ring topology. In a single step each processor computes the local fitness of its current subpopulation while sending the previous subpopulation to the successor and receiving next sub-population from the predecessor. Thus it is possible to overlap communication and computation using non-blocking MPI routines. Our approach was applied to several classification and regression learning problems. The scalability of the algorithm was measured on a compute cluster consisting of sixteen two-processor servers connected by a fast infiniband interconnect. The results of initial experiments show that for large datasets the algorithm is capable of obtaining very good, near linear speedup