This work presents a study of parallel evolutionary programming (EP). The paper is divided into two parts. The first part proposes a concept of parallel EP. Four numerical functions are used to compare the performance between the serial algorithm and the parallel algorithm. In the second part, we apply parallel EP to a more complicated problem - an evolving neural networks problem. The results from this problem show that the parallel version is not only faster than the serial version, but the parallel version also more reliably finds optimal solutions.