Ascent sequences are sequences of nonnegative integers with restrictions on the size of each letter, depending on the number of ascents preceding it in the sequence. Ascent sequences have recently been related to $(2+2)$-free posets and various other combinatorial structures. We study pattern avoidance in ascent sequences, giving several results for patterns of lengths up to 4, for Wilf equivalence and for growth rates. We establish bijective connections between pattern avoiding ascent sequences and various other combinatorial objects, in particular with set partitions. We also make a number of conjectures related to all of these aspects.