An empirical study conducted on primary malware dataset for investigating behavioural analysis. This study propose a novel reverse engineering approach incorporated with CNN. There is no study that has analyzed malware behavior using reverse engineering techniques incorporating CNN. Hence, We developed novel approach to overcome this issue. The results revealed that our study able to analysed malware behavior effectively with 96% accuracy. We includes the investigation of malware behavior targeting windows operating system and excludes malware that affects other operating systems, encrypted malware and limits its focus to static analysis without employing dynamic analysis. This study demonstrates the successful outcomes and the developed approach achieved a high detection accuracy, highlighting its potential to enhance exiting malware detection approach.