Parameters in ARMA models are only locally identified. It is shown that the use of diffuse priors in these models leads to a preference for locally nonidentified parameter values. We therefore suggest to use likelihood based priors like the Jeffreys' priors which overcome these problems. An algorithm involving Importance Sampling for calculating the posteriors of ARMA parameters using Jeffreys' priors is constructed. This algorithm is based on the implied AR specification of ARMA models and shows good performance in our applications. As a byproduct the algorithm allows for the computation of the posteriors of diagnostic parameters which show the identifiability of the MA parameters. As a general to specific modeling approach to ARMA models suffers heavily from the previous mentioned identification problems, we derive posterior odds ratios which are suited for comparing (nonnested) parsimonious (low order) ARMA models. These procedures are applied to two datasets, the (extended) Nelson-Plosser data and monthly observations of US 3-month and 10 year interest rates. For approximately 50% of the series in these two datasets an ARMA model is favored above an AR model which has important consequences for especially the long run parameters