How touch and hearing influence visual processing in sensory substitution, synaesthesia and cross-modal correspondences

Doctoral thesis English OPEN
Hamilton-Fletcher, Giles (2015)
  • Subject: QP0431 | BF0231

Sensory substitution devices (SSDs) systematically turn visual dimensions into patterns of tactile or auditory stimulation. After training, a user of these devices learns to translate these audio or tactile sensations back into a mental visual picture. Most previous SSDs translate greyscale images using intuitive cross-sensory mappings to help users learn the devices. However more recent SSDs have started to incorporate additional colour dimensions such as saturation and hue.\ud Chapter two examines how previous SSDs have translated the complexities of colour into hearing or touch. The chapter explores if colour is useful for SSD users, how SSD and veridical colour perception differ and how optimal cross-sensory mappings might be considered.\ud After long-term training, some blind users of SSDs report visual sensations from tactile or auditory stimulation. A related phenomena is that of synaesthesia, a condition where stimulation of one modality (i.e. touch) produces an automatic, consistent and vivid sensation in another modality (i.e. vision). Tactile-visual synaesthesia is an extremely rare variant that can shed light on how the tactile-visual system is altered when touch can elicit visual sensations. Chapter three reports a series of investigations on the tactile discrimination abilities and phenomenology of tactile-vision synaesthetes, alongside questionnaire data from synaesthetes unavailable for testing.\ud Chapter four introduces a new SSD to test if the presentation of colour information in sensory substitution affects object and colour discrimination.\ud Chapter five presents experiments on intuitive auditory-colour mappings across a wide variety of sounds. These findings are used to predict the reported colour hallucinations resulting from LSD use while listening to these sounds.\ud Chapter six uses a new sensory substitution device designed to test the utility of these intuitive sound-colour links for visual processing. These findings are discussed with reference to how cross-sensory links, LSD and synaesthesia can inform optimal SSD design for visual processing.
  • References (277)
    277 references, page 1 of 28

    Hollins, M., Goble, A., Whitsel, B., & Tommerdahl, M. (1990). Time course and action spectrum of vibrotactile adaptation. Somatosensory and Motor Research, 7, 205-221. doi: 10.3109/08990229009144707

    Howells, T. H. (1944). The experimental development of color-tone synesthesia. Journal of Experimental Psychology, 34(2), 87. doi: 10.1037/h0054424

    Hribar, V., & Pawluk, D. (2011). A tactile-thermal display for haptic exploration of virtual paintings. The Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility, 221-222. doi: 10.1145/2049536.2049577

    Hubbard, T. L. (1996). Synesthesia-like mappings of lightness, pitch, and melodic interval. American Journal of Psychology, 109(2), 219. doi: 10.2307/1423274

    Hubbard, E. M., Arman, A. C., Ramachandran, V. S., & Boynton, G. M. (2005). Individual differences among grapheme-color synesthetes: brain-behavior correlations. Neuron, 45(6), 975-985. doi: 10.1016/j.neuron.2005.02.008

    Hubbard, E. M., Brang, D., & Ramachandran, V. S. (2011). The cross‐activation theory at 10. Journal of Neuropsychology, 5(2), 152-177. doi: 10.1111/j.1748-6653.2011.02014.x

    Hubbard, E. M., Manohar, S., & Ramachandran, V. S. (2006). Contrast affects the strength of synesthetic colors. Cortex, 42(2), 184-194. doi: 10.1016/S0010-9452(08)70343-5

    Hunt, R. W. G., & Pointer, M. R. (2011). Measuring Colour (4th ed.). Chichester, UK: John Wiley & Sons.

    Huopaniemi, J., Zacharov, N., & Karjalainen, M. (1999). Objective and subjective evaluation of headrelated transfer function filter design. Journal of the Audio Engineering Society, 47(4), 218-239.

    Hurvich, L., & Jameson, D. (1955). Some quantitative aspects of an opponent-colors theory: II. Brightness, saturation and hue in normal and dichromatic vision. Journal of the Optical Society of America, 45, 602-616. doi: 10.1364/JOSA.45.000602

  • Related Research Results (3)
  • Metrics
    No metrics available
Share - Bookmark