We present a method for simultaneously learning linear models of multiple high dimensional data sets and the dependencies between them. For example, we learn asymmetrically coupled linear models for the faces of two different people and show how these models can be used to animate one face given a video sequence of the other. We pose the problem as a form of Asymmetric Coupled Component Analysis (ACCA) in which we simultaneously learn the subspaces for reducing the dimensionality of each dataset while coupling the parameters of the low dimensional representations. Additionally, a dynamic form of ACCA is proposed, that extends this work to model temporal dependencies in the data sets. To account for outliers and missing data, we formulate the problem in a statistically robust estimation framework. We review connections with previous work and illustrate the method with examples of synthesized dancing and the animation of facial avatars.