In this paper, we consider memristors, meminductors, and memcapacitors and their properties as port-Hamiltonian systems. The port-Hamiltonian formalism naturally arises from network modeling of physical systems in a variety of domains. Exposing the relation between the energy storage, dissipation, and interconnection structure,this framework underscores the physics of the system. One of the strong aspects of the port-Hamiltonian formalism is that a power-preserving interconnection between port-Hamiltonian systems results in another port-Hamiltonian system with composite energy, dissipation, and interconnection structure. This feature can advantageously be used to model, analyze, and simulate networks consisting of complex interconnections of both conventional and memory circuit elements. Furthermore, the port-Hamiltonian formalism naturally extends the fundamental properties of the memory elements beyond the realm of electrical circuits.