Ferroelectric materials have remained one of the major focal points of condensed matter physics and materials science for over 50 years. In the last 20 years, the development of voltage-modulated scanning probe microscopy techniques, exemplified by Piezoresponse force microscopy (PFM) and associated time- and voltage spectroscopies, opened a pathway to explore these materials on a single-digit nanometer level. Consequently, domain structures and walls and polarization dynamics can now be imaged in real space. More generally, PFM has allowed studying electromechanical coupling in a broad variety of materials ranging from ionics to biological systems. It can also be anticipated that the recent Nobel prize [“The Nobel Prize in Chemistry 2016,” http://www.nobelprize.org/nobel_prizes/chemistry/laureates/2016/ (Nobel Media, 2016)] in molecular electromechanical machines will result in rapid growth in interest in PFM as a method to probe their behavior on single device and device assembly levels. However, the broad introduction of PFM also resulted in a growing number of reports on the nearly ubiquitous presence of ferroelectric-like phenomena including remnant polar states and electromechanical hysteresis loops in materials which are non-ferroelectric in the bulk or in cases where size effects are expected to suppress ferroelectricity. While in certain cases plausible physical mechanisms can be suggested, there is remarkable similarity in observed behaviors, irrespective of the materials system. In this review, we summarize the basic principles of PFM, briefly discuss the features of ferroelectric surfaces salient to PFM imaging and spectroscopy, and summarize existing reports on ferroelectric-like responses in non-classical ferroelectric materials. We further discuss possible mechanisms behind observed behaviors and possible experimental strategies for their identification.