To improve the efficiency of hard rock breaking by a pulsed water jet (PWJ), a hydraulically controlled piston-pressurized PWJ (HCPPPWJ) device has been developed, by which the large amplitude pressurization of the jet could be realized through the motion coupling of the piston and the valve core inside the device without requiring additional control or ultra-high-pressure components. Under the continuous injection of low-pressure hydraulic oil, the device has a stable pressurization effect and controllable pulse pressure and pulse frequency. The jet pressure varies periodically with the alternation of high and low pressures; in the rising stage of the pulse pressure, the jet morphology presents an umbrella-like thin-layer structure, which ensures an effective initial impact force of the jet in contact with the target. With the addition of high-frequency stress waves and water wedge pressure, local flaky exfoliation was observed when the granite surface was eroded, and the maximum radius and volume of the erosion pit were greater than those in the case of employing a continuous water jet. Compared with the interrupted PWJ, the HCPPPWJ efficiently utilizes the jet energy during the erosion process, and the specific energy is lower. The results prove that the HCPPPWJ device is an advanced tool in the field of hard rock breaking.