The time irreversibility of a dynamical process refers to the phenomenon where its behaviour or statistical properties change when it is observed under a time-reversal operation, i.e., backwards in time and indicates the presence of an “arrow of time”. It is an important feature of both synthetic and real-world systems, as it represents a macroscopic property that describes the mechanisms driving the dynamics at a microscale level and that stems from non-linearities and the presence of non-conservative forces within them. While many alternatives have been proposed in recent decades to assess this feature in experimental time series, the evaluation of their performance is hindered by the lack of benchmark time series of known reversibility. To solve this problem, we here propose and evaluate the use of a geometric Brownian motion model with stochastic resetting. We specifically use synthetic time series generated with this model to evaluate eight irreversibility tests in terms of sensitivity with respect to several characteristics, including their degree of irreversibility and length. We show how tests yield at times contradictory results, including the false detection of irreversible dynamics in time-reversible systems with a frequency higher than expected by chance and how most of them detect a multi-scale irreversibility structure that is not present in the underlying data.