When expressing preferences in qualitative setting, several possible linguistic terms with different weights (represented by probabilities) may be considered at the same time. The probabilistic distribution is usually hard to be provided completely and ignorance may exist. In this paper, we first propose a novel concept called probabilistic linguistic term set (PLTS) to serve as an extension of the existing tools. Then we put forward some basic operational laws and aggregation operators for PLTSs. After that, we develop an extended TOPSIS method and an aggregation-based method respectively for multi-attribute group decision making (MAGDM) with probabilistic linguistic information, and apply them to a practical case concerning strategy initiatives. Finally, the strengths and weaknesses of our methods are clarified by comparing them with some similar techniques.