VoIP networks are in a major deployment phase and are becoming widely spread out due to their extended functionality and cost efficiency. Meanwhile, as VoIP traffic is transported over the Internet, it is the target of a range of attacks that can jeopardize its proper functionality. In this paper we describe our work in a VoIP specific security assessment framework. Such an assessment is automated with integrated discovery actions, data management and security attacks allowing to perform VoIP specific penetration tests. These tests are important because they permit to search and detect existing vulnerabilities or misconfigured devices and services. Our main contributions consist in an elaborated network information model capable to be used in VoIP assessment, an extensible assessment architecture and its implementation, as well as in a comprehensive framework for defining and composing VoIP specific attacks.