The modern sedentary lifestyle is negatively influencing human health, and the current guidelines recommend at least 150 min of moderate activity per week. However, the challenge is how to measure human activity in a practical way. While accelerometers are the most common tools to measure activity, current activity classification methods require calibration studies or labelled datasets—requirements that slow the research progress. Therefore, there is a pressing need to classify and quantify human activity efficiently. In this work, we propose an unsupervised approach to classify activities from accelerometer data using hidden semi-Markov models. We tune and infer the model parameters on accelerometer data from the UK Biobank and select the optimal model based on features used and informativeness of the prior. The best model achieves an average correlation of 0.4 between the inferred activities and the reference ones, with the overall physical activity obtaining a correlation of 0.8. Additionally, to prove the clinical significance of the method, we validate it by performing a linear regression between the inferred activities and anthropometric measures such as BMI and waist circumference. We show that for a sedentary behaviour and total physical activity, the proposed method achieves comparable regression coefficients to the reference labelled dataset. Moreover, the proposed method achieves a good agreement with a labelled dataset for daily time spent in a sedentary behaviour and total physical activity. The unsupervised nature of the method allows for a data-driven classification that does not require calibration studies or labelled datasets and can thus facilitate both clinical research as well as lifestyle recommendations.