Existing inertial measurement unit (IMU) based human activity recognition (HAR) approaches still face a major challenge when adopted across users in practice. The severe heterogeneity in IMU data significantly undermines model generalizability in wild adoption. This paper presents UniHAR, a universal HAR framework for mobile devices. To address the challenge of data heterogeneity, we thoroughly study augmenting data with the physics of the IMU sensing process and present a novel adoption of data augmentations for exploiting both unlabeled and labeled data. We consider two application scenarios of UniHAR, which can further integrate federated learning and adversarial training for improved generalization. UniHAR is fully prototyped on the mobile platform and introduces low overhead to mobile devices. Extensive experiments demonstrate its superior performance in adapting HAR models across four open datasets.