In this paper, we collected the seismic phase arrival data of 14,033 local natural earthquakes above magnitude 3.0 recorded by 435 seismograph stations in the study area provided by the International Seismological Centre (ISC), covering the period from January 2011 to April 2020. We selected the first arrival P- and S-wave arrival time data and obtained 281,859 P- and 112,926 S-wave absolute arrival times and 528,250 P-differential and 207,968 S-differential arrival times. Then we determined 3-D P and S wave velocity structures from the Japan Trench to the back arc area under the Japan Islands by using double-difference tomography method. The results show strong lateral heterogeneities under the forearc region. The subducting Pacific slab is imaged clearly as a high-velocity (high-V) faster than the surrounding mantle. Low-velocity (low-V) zones are imaged in the mantle wedge with significant along arc variations under the volcanic front. The vertical section shows that the low velocity zone extends to a depth of approximately 70–150 km below the arc. The crust and mantle wedges beneath the front and back arcs of the volcanoes reveal a low-V anomaly, the likely main source of which is the partial melting of plate and mantle wedge material. Referring to the previous results of plate dehydration, mineral composition and thermal state, it is found that fluids play a crucial role in the arc magmatism and plate melting of mantle wedge behind Japan Trench. The fluids brought down by the Pacific subducting plate are released into the mantle wedge by dehydration and subsequently transported upward by the upwelling flow in the mantle wedge. The present results obtained using new and advanced imaging methods enrich the understanding of the velocity structure beneath Japan Islands, which may improve the understanding of the dynamic processes of subduction zones and mantle upwelling.