Networks are one of the basic structures in many physical phenomena pertaining to engineering applications. As a network can be represented by a graph which is isomorphic to its adjacency matrix, the study of analysis of networks involving rate of change with respect to time reduces to the study of graph differential equations or equivalently matrix differential equations. In this paper, we develop the basic infrastructure to study the IVP of a graph differential equation and the corresponding matrix differential equation. Criteria are obtained to guarantee the existence of a solution and an iterative technique for convergence to the solution of a matrix differential equation is developed.