SummaryThe security of digital images is an essential and challenging task on shared communication Model. Generally, high secure working environment and data are also secured with an encryption and decryption method by using secret and public keys. In this paper, the innovative encryption technique for image security, ie, Multiple key‐based Homomorphic Encryption (MHE) technique is proposed. For increasing the security level of encryption and decryption processes, the optimal key is selected using Adaptive Whale Optimization (AWO) algorithm. Fitness function was considered for optimization as PSNR of plain and cipher images. The original image was transformed into blocks and then rearranged utilizing encryption process, this work achieved maximum security, much better than other encryption techniques. From the outcomes, one can achieve incredible quality of the proposed model, the maximum PSNR, and the minimum MSE contrasted with other encryption schemes.