Difference sets have been studied extensively. The known examples fall into several families together with some sporadic exceptions. Supplementary difference systems are generalizations of difference sets. The author considers such generalizations of the residue difference sets. He establishes the existence of several new classes of supplementary difference systems which can be seen as corresponding to the family of difference sets. The major result is the existence of several new infinite families of supplementary difference systems which are unions of residue classes. More precisely, the multiplicative groups \(F^*_{q^2}\), where \(q= 4m+3\) is a prime power, are considered. Further it is proven that the union of certain cosets of the \((q+1)\)st powers in \(F^*_{q^2}\) form a supplementary difference system in the additive group of \(F_{q^2}\). Finally an application to both coding theory and Hadamard matrices is given.