We present a general graph-based modeling abstraction for optimization that we call an OptiGraph. Under this abstraction, any optimization problem is treated as a hierarchical hypergraph in which nodes represent optimization subproblems and edges represent connectivity between such subproblems. The abstraction enables the modular construction of highly complex models in an intuitive manner, facilitates the use of graph analysis tools (to perform partitioning, aggregation, and visualization tasks), and facilitates communication of structures to decomposition algorithms. We provide an open-source implementation of the abstraction in the Julia-based package Plasmo.jl. We provide tutorial examples and large application case studies to illustrate the capabilities.