We propose and experimentally demonstrate an optical processor for a binarized neural network (NN). Implementation of a binarized NN involves multiply-accumulate operations, in which positive and negative weights should be implemented. In the proposed processor, the positive and negative weights are realized by switching the operations of a dual-drive Mach–Zehnder modulator (DD-MZM) between two quadrature points corresponding to two binary weights of +1 and −1, and the multiplication is also performed at the DD-MZM. The accumulation operation is realized by dispersion-induced time delays and detection at a photodetector (PD). A proof-of-concept experiment is performed. A binarized convolutional neural network (CNN) accelerated by the optical processor at a speed of 32 giga floating point operations/s (GFLOPS) is tested on two benchmark image classification tasks. The large bandwidth and parallel processing capability of the processor has high potential for next generation data computing.