Number representation systems establish ways in which numbers are mapped to computer architectures, and how operations over the numbers are translated into computer instructions. The efficiency of public-key cryptography is strongly affected by the used number representations, as these systems are constructed from mathematically inspired problems to ensure security, and thus rely on operations over large integers. In this paper, unconventional representations systems, including the Residue Number System (RNS) and stochastic number representations, are systematically reviewed. Homomorphic representations, which allow for parties to operate on data without having access to their plaintext representation, are also considered. The main goal of this survey is to introduce the reader to key aspects of non-traditional number representations that may be exploited for public-key cryptography, without delving too much into the details. Examples of the methods and algorithms herein surveyed include subquadratic modular multiplication for isogeny-based cryptography, the acceleration of Goldreich-Goldwasser-Halevi (GGH) decryption by an order of magnitude, the improvement of the Direct Anonymous Attestation (DAA) protocol both in terms of storage requirements and the time taken to execute it, and efficient algorithm-hiding Fully Homomorphic Encryption (FHE). The implementation of this type of systems in both sequential and parallel platforms is analysed, and their performance is compared with traditional approaches. We hope this work sows the seed of further research on the application of non-positional number arithmetic to other cryptographic use-cases.