We consider a recovery problem for an unknown matrix $A$ lying in a known subspace of real $m\times n$ matrices. Examples of such subspaces or ``linear families'' include Toeplitz, Hankel, circulant, and tridiagonal matrices. With fixed nonnegative integers $q_R$ and $q_L$, the goal of the recovery problem is to choose a real $n\times q_R$ matrix $X_R$ and a real $m\times q_L$ matrix $X_L$ so that $A$ is uniquely determined within its family when $Y_R = A X_R$ and $Y_L = A^T X_L$ are known. The columns of $Y_R$ and $Y_L$ are formed by matrix-vector products of $A$ and $A^T$ with the columns of $X_R$ and $X_L$ to be determined. We show that if this problem has a solution, then the pairs of matrices $(X_R, X_L)$ that have the stated dimensions and do not solve the recovery problem form a set of Lebesgue measure zero. It follows that square Toeplitz, Hankel, circulant, and tridiagonal matrices are uniquely determined by their matrix-vector products with almost any $q_R = 2,2,1,$ and $3$ vectors respectively and $q_L=0$.