We highlight the difference between the service sector and the manufacturing sector in regard to the determinants for a firm to start FDI and the productivity growth it achieves. This paper analyzes two questions: (1) whether a certain level of productivity explains a Japanese firm's choice to be a multinational firm (by starting FDI), and (2) how the productivity of such a multinational firm changes over time after FDI. Using the longitudinal panel data on Japanese firms from 1980 to 2005, We trace some firm-level decisions over several decades. This research contributes to the discussions where empirical evidence is not yet profoundly available: how the TFP of the service and that of manufacturing sectors present difference for the choice of overseas activity, and how much productivity gain firms may achieve by intrafirm and cross-border reallocation of firm resources. We have found the following results: (1) compared by year and by industry, the TFP in manufacturing does not explain a firm's choice for starting FDI, but the TFP in the service sector does, then a low level of productivity deters a firm from pursuing FDI; (2) in the manufacturing sector, the size and profitability of firms are positive factors for their future choice in FDI, but these do not matter in the service sector; (3) after FDI, entrants in the service sector show 1.4 times higher annual productivity growth than those in the manufacturing sector. The productivity in service is also on average higher than that of selected domestic firms for counterfactuals.