Artificial neural networks (ANNs) are widely applied to solve real-world problems. Most of the actions we take and the processes around us are time-varying. ANNs with dynamic properties allow processing time-dependent data and solving tasks such as speech and text processing, prediction models, face and emotion recognition, game strategy development. Dynamics in neural networks can appear in the input data, the architecture of the neural network, and the individual elements of the neural network – synapses and neurons. Unlike static synapses, dynamic synapses can change their connection strength based on incoming information. This is a fundamental principle allows neural networks to perform complex tasks like word processing or face recognition more efficiently. Dynamic synapses play a key role in the ability of artificial neural networks to learn from experience and change over time, which is one of the key aspects of artificial intelligence. The scientific works examined in this article show that there are no literature sources that review and compare dynamic DNTs according to their synapses. To fill this gap, the article reviews and groups DNTs with dynamic synapses. Dynamic neural networks are defined by providing a general mathematical expression. A dynamic synapse is described by specifying its main properties and presenting a general mathematical expression. Also an explanation, how these synapses can be modelled and integrated into 11 different dynamic ANNs is shown. Moreover, structures of dynamic ANNs are compared according to the properties of dynamic synapses.