Overlay multicast networks are used by service providers to distribute contents such as Web pages, static and streaming multimedia data, or security updates to a large number of users. However, such networks are extremely vulnerable to message-dropping attacks by malicious or selfish nodes that intentionally drop the packets they are required to forward to others. It is difficult to detect such attacks both efficiently and effectively and to further identify the attackers, especially when members in the overlay switch between online/offline statuses frequently. In this article, we consider various attacking strategies of an attacker and propose an optimal sampling-based scheme to detect such attacks in the overlay network. We analyze the detection problem from a game-theoretical viewpoint and show that our scheme outperforms a random sampling-based scheme in terms of detection rate. In addition, based on a reputation system, we propose a sampling-based path-resolving scheme to identify compromised or selfish nodes. Unlike other existing approaches, our schemes do not assume global knowledge of the overlay hierarchy and work for dynamic overlay networks as well. Extensive analysis and simulation results show that besides being band width efficient, our schemes have high detection and identification rates and low false-positive rates.