This paper presents the design of the teleoperation system for a gantry crane, entirely using tools from passivity theory. Gantry cranes still pose a control challenge, mainly due to the problem of load swing suppression, in underactuated conditions. In the teleoperation setup, the operator-master dynamics and the transmission delay complicate the overall system design. Fortunately, the framework of passivity theory offers a lot of resources to address the problem. This paper describes the design choices for the various subsystems, which lead finally to a reasonably satisfactory solution. Some relevant aspects treated here are: the use of polar coordinates for the swing dynamics of the crane; the proposal of an arrangement for connecting an IDA-PBC controlled site to a teleoperation line; and the recasting of the fundamental tradeoff between swing suppression and performance, in terms of energy shaping.