Prediction of abdominal aortic aneurysm (AAA) growth is of essential importance for the early treatment and surgical intervention of AAA. Capturing key features of vascular growth, such as blood flow and intraluminal thrombus (ILT) accumulation play a crucial role in uncovering the intricated mechanism of vascular adaptation, which can ultimately enhance AAA growth prediction capabilities. However, local correlations between hemodynamic metrics, biological and morphological characteristics, and AAA growth rates present high inter-patient variability that results in that the temporal-spatial biochemical and mechanical processes are still not fully understood. Hence, this study aims to integrate the physics-based knowledge with deep learning with a patch-based convolutional neural network (CNN) approach by incorporating important multiphysical features relating to its pathogenesis for validating its impact on AAA growth prediction. For this task, we observe that the unstructured multiphysical features cannot be directly employed in the kernel-based CNN. To tackle this issue, we propose a parameterization of features to leverage the spatio-temporal relations between multiphysical features. The proposed architecture was tested on different combinations of four features including radius, intraluminal thrombus thickness, time-average wall shear stress, and growth rate from 54 patients with 5-fold cross-validation with two metrics, a root mean squared error (RMSE) and relative error (RE). We conduct extensive experiments on AAA patients, the results show the effect of leveraging multiphysical features and demonstrate the superiority of the presented architecture to previous state-of-the-art methods in AAA growth prediction.