We consider two pairs: the standard unknotted $n$-sphere in $S^{n+2}$, and the product of two $p$-spheres trivially embedded in $S^{2p+2}$, and study orientation preserving diffeomorphisms of these pairs. Pseudo-isotopy classes of such diffeomorphisms form subgroups of the mapping class groups of $S^n$ and $S^p\times S^p$ respectively and we determine the algebraic structure of such subgroups when $n>4$ and $p>1$.