Steganalysis is the reasonable method to detect whether the transmitted media content contains secret messages (e.g. business secrecy). This paper proposes two steganalysis methods to estimate the modification ratio of F5 steganography and its improved version that are popularly used to hide secrecy in images. The proposed methods measure the distance between the coefficient histogram of a given image and that of an estimated stego image. The distance is measured based on relative entropy that has the superiority of measuring the distance between two distributions. The estimated modification ratio can be used to distinguish the stego images marked by F5 steganography or its improved version from the original images. Experimental results are given to show that the proposed methods outperform the existing quantitative steganalysis methods against F5 steganography and its improved version.