System security has emerged as a premier design requirement. While there has been an enormous body of impressive work on testing integrated circuits (ICs) desiderata such as manufacturing correctness, delay, and power, there is no reported effort to systematically test IC security in hardware. Our goal is to provide an impetus for this line of research and development by introducing techniques and methodology for rigorous testing of physically unclonable functions (PUFs). Recently, PUFs received a great deal of attention as security mechanisms due to their flexibility to form numerous security protocols and intrinsic resiliency against physical and side channels attacks. We study three classes of PUFs properties to design pertinent test methods: (i) predictability, (ii) sensitivity to component accuracy, and (iii) susceptibility to reverse engineering. As our case studies, we analyze two popular PUF structures, linear and feed-forward, and show that their security is not adequate from several points of view. The technical highlights of the paper are the first non-destructive technique for PUF reverse engineering and a new PUF structure that is capable of passing our security tests.