Shallow permutations were defined in 1977 to be those that satisfy the lower bound of the Diaconis-Graham inequality. Recently, there has been renewed interest in these permutations. In particular, Berman and Tenner showed they satisfy certain pattern avoidance conditions in their cycle form and Woo showed they are exactly those whose cycle diagrams are unlinked. Shallow permutations that avoid 321 have appeared in many contexts; they are those permutations for which depth equals the reflection length, they have unimodal cycles, and they have been called Boolean permutations. Motivated by this interest in 321-avoiding shallow permutations, we investigate $\sigma$-avoiding shallow permutations for all $\sigma \in \mathcal{S}_3$. To do this, we develop more general structural results about shallow permutations, and apply them to enumerate shallow permutations avoiding any pattern of length 3.