In this paper, we propose a mathematical framework to model activities with both motion and context information for activity recognition. This is motivated from the observations that an activity does not only depend on the motion of the objects of interest but the surrounding objects also provide useful cues for an understanding of the activity. Thus the surrounding objects can serve as context for the concerned activity. Given training data, our model aims to automatically capture and weigh motion and context patterns for each activity class, from sets of predefined attributes, during the learning process. Then, the learned model is used to generate optimum labels for activities in the testing videos based on the motion and context features of these activities. We show how to learn the model parameters via an unconstrained convex optimization methodology and how to predict the correct label for a testing instance. We show promising results on the publicly available VIRAT Ground Dataset that demonstrates the benefit of modeling the surrounding context in recognizing activities in a wide-area scene.