The primary purpose of this study was to investigate whether an indirect or a direct comparison method is more sensitive for comparing hearing aid programs in daily life. The secondary purpose was to investigate whether adding a longer transition time (TT) to the direct comparison method affects the results and whether there is a difference in experienced burden between the methods. Nine adults with hearing impairment were included in this randomized cross-over study. The participants were fitted with a set of hearing aids including a high directionality (HD) and low directionality (LD) programs. The participants answered questionnaires each day via an Ecological Momentary Assessment app. For sound quality and hearing aid satisfaction, the indirect method resulted in the highest contrast between the two programs. All three methods pointed towards a preference of the LD program. Approximately 40% of all rated situations contained speech and a rating for speech understanding. Here the direct comparison yielded a clear preference for the HD program, while the other two methods showed a trend in the opposite direction. Burden ratings were the lowest for indirect comparison and highest for direct comparison with TT. The methods provided results at different sensitivity for different outcomes, hence it should be carefully considered which method to use for a given research question. The higher burden when using direct comparison with TT coupled with no significant program preference suggests that direct comparison method should be used without the additional TT time.