In this paper, we consider neural networks in the case when the neurons are subject to a certain impulsive state displacement at fixed moments and the duration of this displacement is not negligible small (these are known as non-instantaneous impulses). We examine some stability properties of the equilibrium of the model. Several sufficient conditions for uniform Lipschitz stability of the equilibrium of neural networks with time varying self-regulating parameters of all units and time varying functions of the connection between two neurons in the network are obtained. These sufficient conditions are explicitly expressed in terms of the parameters of the system and hence they are easily verifiable. The case of non-Lipschitz activation functions is also studied. The theory is illustrated on particular nonlinear neural networks.