We propose a notion of passivity for hybrid systems. Our work is motivated by problems in haptics and teleoperation where several computer controlled mechanical systems are connected through a communication channel. To account for time delays and to better react to user actions it is desirable to design controllers that can switch between different operating modes. Each of the interacting systems can be therefore naturally modeled as a hybrid system. A traditional passivity definition requires that a storage function exists that is common to all operating modes. We show that stability of the system can be guaranteed even if different storage function is found for each of the modes, provided appropriate conditions are satisfied when the system switches.