Sparse Bundle Adjustment (SBA) is a method for simultaneously optimizing a set of camera poses and visible points. It exploits the sparse primary structure of the problem, where connections exist just between points and cameras. In this paper, we implement an efficient version of SBA for systems where the secondary structure (relations among cameras) is also sparse. The method, which we call Sparse SBA (sSBA), integrates an efficient method for setting up the linear subproblem with recent advances in direct sparse Cholesky solvers. sSBA outperforms the current SBA standard implementation on datasets with sparse secondary structure by at least an order of magnitude, while also being more efficient on dense datasets.