Cavitation erosion is one of the most severe problems encountered in hydraulic turbomachinery. When testing the materials, the engineers usually rely on standardized procedures. The most common one being the vibratory ASTM G-32 test, which offers two possibilities of performing the test - the direct, where the specimen is attached to the ultrasonic device and the indirect, where the specimen is stationary and exposed to the ultrasonic horn, positioned just 0.5 mm from it. The erosion rates from the two are significantly different and a question may be asked if they are at all comparable and further on are they comparable to the "real-life" hydrodynamic cavitation which occurs in turbomachinery. In this study we performed erosion tests on a stationary specimen where the gap between the specimen and the horn was varied from 0.3 to 4 mm. In addition, we used high speed visualization to observe the cavitation in the gap. We observed that the cavitation erosion rate strongly depends on the gap. From visualization we see that the cavitation dynamics significantly changes in a small gap, leading to a large, but 2-dimensional cavitation bubbles which collapse very slowly, compared to the small spherical ones in a larger gap. We investigated the probability of shock wave occurrence and derived a very simple model, which gives accurate qualitative predictions of experimental data. Finally, the study puts into question the validity of ASTM G32 test - the most common approach used in engineering today.