This paper deals with the security of chaos-based "true" random number generators (RNG)s. An attack method is proposed to analyze the security weaknesses of chaos-based RNGs and its convergence is proved using a master slave synchronization scheme. Attack on a RNG based on a double-scroll attractor is also presented as an example. All secret parameters of the RNG are revealed where the only information available is the structure of the RNG and a scalar time series observed from the double-scroll attractor. Simulation and numerical results of the proposed attack method are given such that the RNG doesn't fulfill NIST-800-22 statistical test suite, not only the next bit but also the same output bit stream of the RNG can be reproduced.