Coded computation is a framework for providing redundancy in distributed computing systems to make them robust to slower nodes, or stragglers. In a recent work of Lee et al., the authors propose a coded computation scheme for distributedly computing A x x in the presence of stragglers. The proposed algorithm first encodes the data matrix A to obtain an encoded matrix F. It then computes F x x using distributed processors, waits for some subset of the processors to finish their computations, and decodes A x x from the partial computation results. In another recent work, Dutta et al. explore a new tradeoff between the sparsity of the encoded matrix F and the number of processors to wait to compute A x x. They show that one can introduce a large number of zeros into F to reduce the computational overheads while maintaining the number of processors to wait relatively low. Hence, one can potentially further speed up the distributed computation. In this work, motivated by this observation, we study the sparsity of the encoded matrix for coded computation. Our goal is to characterize the fundamental limits on the sparsity level. We first show that the Short-Dot scheme is optimal if an Maximum Distance Separable (MDS) matrix is fixed. Further, by also designing this MDS matrix, we propose a new encoding scheme that can achieve a strictly larger sparsity than the existing schemes. We also provide an information-theoretic upper bound on the sparsity.