In the collaborative filtering algorithm, the matrix factorization method based on rating data has been widely applied and developed, but the data sparsity problem affects the method recommendation quality. In view of this problem, a recommendation method (RHAOR) is proposed to integrate the review text hierarchical attention and outer product. Two parallel networks are used to process user review sets and item review sets, respectively. This paper applies aspect-level attention mechanism to the review text content, marks multiple words (or phrases) with aspect information, applies review-level attention mechanism to the review set, and marks valid reviews. The outer product is used to establish an outer product interaction matrix for user preferences and item features, and the multi-layer convolutional neural network is used to extract the outer product interaction feature. The outer product interaction feature is introduced into the improved latent factor model (LFM) for rating prediction. The experimental results show that the proposed method consistently outperforms traditional rating score and review based methods in root mean square error (RMSE) on Amazon and Yelp datasets.