Protecting digital data, especially digital images, from unauthorized access and malicious activities is crucial in today’s digital era. This paper introduces a novel approach to enhance image encryption by combining the strengths of the RSA algorithm, homomorphic encryption, and chaotic maps, specifically the sine and logistic map, alongside the self-similar properties of the fractal Sierpinski triangle. The proposed fractal-based hybrid cryptosystem leverages Paillier encryption for maintaining security and privacy, while the chaotic maps introduce randomness, periodicity, and robustness. Simultaneously, the fractal Sierpinski triangle generates intricate shapes at different scales, resulting in a substantially expanded key space and heightened sensitivity through randomly selected initial points. The secret keys derived from the chaotic maps and Sierpinski triangle are employed for image encryption. The proposed scheme offers simplicity, efficiency, and robust security, effectively safeguarding against statistical, differential, and brute-force attacks. Through comprehensive experimental evaluations, we demonstrate the superior performance of the proposed scheme compared to existing methods in terms of both security and efficiency. This paper makes a significant contribution to the field of digital image encryption, paving the way for further exploration and optimization in the future.