Abstract In many problems involving multivariate time series, hidden Markov models (HMMs) are often employed for modeling complex behavior over time. HMMs can, however, require large number of states, what can lead to poor problem insight and model overfitting, especially when limited data is available. In this paper, we further investigate the family of asymmetric hidden Markov models (HMM-As), which generalize the emission distributions to arbitrary Bayesian-network distributions, allowing for state-specific graphical structures in the feature space. As a consequence, HMM-As are able to render more compact state spaces, thus from a learning perspective HMM-As can better handle the complexity-overfitting trade-off. In this paper, we study representation properties of asymmetric and symmetric HMMs, as well as provide a learning algorithm for HMM-As. We provide empirical results based on simulations for comparing HMM-As with symmetric and other asymmetry-aware models, showing that modeling more general asymmetries can be very effective. We also consider real-world datasets from several domains, aiming to show that multiple graphical structures underlying data can be identified and are able to provide additional problem insight. Although learning HMM-As can be more complex, it is shown that it is feasible in practice due to their ability to maintain compact state spaces, yet more expressive ones.