We show that infinitesimal automorphisms and infinitesimal deformations of parabolic geometries can be nicely described in terms of the twisted de Rham sequence associated to a certain linear connection on the adjoint tractor bundle. For regular normal geometries, this description can be related to the underlying geometric structure using the machinery of BGG sequences. In the locally flat case, this leads to a deformation complex, which generalizes the well known complex for locally conformally flat manifolds. Recently, a theory of subcomplexes in BGG sequences has been developed. This applies to certain types of torsion free parabolic geometries including quaternionic structures, quaternionic contact structures and CR structures. We show that for these structures one of the subcomplexes in the adjoint BGG sequence leads (even in the curved case) to a complex governing deformations in the subcategory of torsion free geometries. For quaternionic structures, this deformation complex is elliptic.