Have 'Quasi-Inverted Spectrum' Individuals Fallen into Our World, Unbeknownst to Us?
A Tentative Perspective on Explaining Synesthesia
Today, when you open Wikipedia, type "synesthesia," and press Enter, you will get a definition of synesthesia like this:
Synesthesia or synaesthesia is a perceptual phenomenon in which stimulation of one sensory or cognitive pathway leads to involuntary experiences in a second sensory or cognitive pathway.
For anyone who, like me, has read the works of scholars like V.S. Ramachandran and Richard Cytowic, who pioneered modern synesthesia research, this definition will be very familiar. In fact, this is precisely their theory and core hypothesis:
The theories of both Ramachandran and Cytowic make a core assumption:
Synesthetes claim they "hear colors" or "taste shapes" because the activation of different brain parts is a necessary condition for forming different subjective experiences. When synesthetes perceive, their brains activate more areas than those of non-synesthetes, and these extra activated areas are exactly the areas responsible for sensations of other modalities. This causes synesthetes, upon receiving a stimulus in one sensory modality, to not only have an experience corresponding to that modality but also an accompanying experience of another sensory modality. This is why they claim to "hear colors" or "taste shapes."
For example, in the most common grapheme-color synesthesia, when letters are presented, we can expect the color center in a synesthete's brain to be activated, whereas this is not the case in typical subjects.
Obviously, this is a great theory because we can expect it to be testable with modern brain imaging techniques:
If the theory holds, we would expect to see that when a stimulus (e.g., the inducer) is presented, a synesthete's brain would show more activated areas compared to a non-synesthete's. Furthermore, the sensory modalities corresponding to these extra activated areas should align with the "additional sensations" (e.g., more professionally, their concurrent) experienced by the synesthete.
The problem is, to this day, it remains difficult to judge the correctness of this theory from neuroscience experiments. A 2015 review1 summarized the relevant research findings. The differences between experiments are vast, with almost no common brain regions found across studies. More importantly, in the experiments analyzed in this paper, almost none observed activation in the brain regions of interest.
On one hand, this can certainly be attributed to the large errors in brain imaging technology and poor experimental reproducibility. But on the other hand, I have begun to wonder: is the aforementioned assumption correct? Have we truly listened to and understood what synesthetes are telling us?
So I reread Cytowic's book2. I selected some self-reports from synesthetes in the first chapter:
MN I remember most accurately scents. We were preparing to move into the house I grew up in. I remember at age 2 my father was on a ladder painting the left side of the wall. The paint smelled blue, although he was painting it white. I remember to this day thinking why the paint was white, when it smelled blue
MM I see shapes and colors in response to sounds. I enjoy electronic music because it evokes such wonderful shapes and colors in my visual perception area. I feel for the first time that I am not nuts! The colored shape is seen as if I were looking through a plastic transparency which is in front of my eyes. If I shut my eyes, or if it is at night in the dark, then the shapes are the only thing in the field and are therefore more intense.
DS When I listen to music, I see the shapes on an externalized area about 12 inches in front of my face and about one foot high onto which the music is visually projected. Sounds are most easily likened to oscilloscope configurations—lines moving in color, often metallic with height, width and, most importantly, depth. My favorite music has lines that extend horizontally beyond the "screen" area.
Upon reading these accounts, what struck me as most odd is that synesthetes do not seem to explicitly state that when receiving a stimulus in one sense, they experience the sensation associated with that first sense along with an involuntary experience of a second sense. Instead, they claim to "smell blue" or "see shapes and colours in response to sounds." They seem to treat the second sensory experience as the direct experience of the first sensory stimulus. Otherwise, why wouldn't they report that they feel the color blue while smelling the pungent odor of gasoline, instead of saying "gasoline smells blue"?
Of course, the above interpretation may seem a bit forced. Due to the slow progress in theories of consciousness, it is difficult to determine directly from these vague descriptions whether their use of "sound" or "smell" refers to the physical stimulus received by the senses, or to an actual subjective experience of "sound" and "smell" that is distinct from the experience of color. Nevertheless, this has made me suspicious: have we interpreted synesthetes' accounts with theoretical preconceptions? And have we selectively reported their self-descriptions? Therefore, I looked into research on synesthesia from the early 20th century. I went back to an era without neuroimaging and without established theories of synesthesia, to carefully examine psychologists' reports on synesthetes' own accounts, free from the "light pollution" of modern theories.
The material I found shocked me; it confirmed that my concerns were more than just paranoia. In Charles S. Myers's paper3, "Two cases of synaesthesia", he reported on two cases of sound-color synesthesia:
As will have been gathered by now, Scriabin's chromaesthesia refers to the tonality of the music. As the tonality changes in a piece, so the colour changes. Scriabin explains that "the colour underlines the tonality; it makes the tonality more evident." The colour or a change of colour sometimes appears to him before he is aware of the tonality or of a change of tonality. For such reasons he believes that the musical effects are enhanced by the simultaneous presentation to the eye of the appropriate colour. In general, when listening to music, he has only a 'feeling' of colour; only in cases where the feeling is very intense does it pass over to give an 'image' of colour.
…
Like the case (Subject A) described in my previous paper, this subject has very poor visual imagery, and the colours which she obtains vary with the pitch and with the timbre of the tone. A visual image only comes to her when she has taken special care to attend to the previous perception of the object. Thus, when painting, she can visualise her 'sitter' if she has attended particularly to the pose. But she cannot get a visual image of her breakfast table, "because she has never attended to it." Nor does she 'see' colours in sounds; she explains her coloured hearing on the ground that sounds "give her the same mental sensations" as colours.
In June E. Downey's paper4, "A Case of Colored Gustation," a similar account is described:
The young man (S.), who reports the present case, is a senior in college, has had some practice in psychological experimentation, and is an excellent observer. He has been under observation a year. So far as he can remember, he has always experienced tastes as colored. He reports that, as children, he and his sister employed color-names in describing their tastes. His taste-colors are located in the mouth; and they are intensified by closing his eyes.
I also asked synesthetes on Reddit5, and one of them gave me this reply (admittedly, this is not typically considered strong evidence on psychology):
Experientially, I don't "see" sound separately than I hear it. Sounds, for anyone, have different aspects, volume, timbre, direction, pitch. For me sounds also have, color, shape, texture. But it is just one sound. So intuitively, there isn't a real difference in what the word see or hear means to me. Objectively, I of course understand that. But I almost have to remind myself that other people don't see sounds all the time.
I wonder, have we really found individuals similar to those in John Locke's 'inverted spectrum' thought experiment (though different from the original, as this is not a symmetrical swap but rather one modality replacing another)? Imagine if, from birth, our auditory qualia disappeared and were replaced by visual qualia, changing the experienced qualia just as in the original inverted spectrum experiment. How would we describe the world? Naturally, we would use visual elements to name auditory elements, starting from the very day we learned to speak. As for the concepts described by typical people, like pitch, timbre, and intensity, we would need to learn them carefully to cautiously map these concepts to the visual qualia we "hear." Perhaps synesthetes also find us strange, wondering why we give such vastly different names to two such similar experiences?
Let's now look at one final study6, an excerpt from a research report using introspection by a synesthete. I find it to be the most detailed description I could find of how a sound-color synesthete perceives their world and let us just dive into what he has described.
Stimulus: The tapping of a bell with instructions to A to anticipate the sound. "Tried to anticipate the sound in terms of leaning my head in the direction of the experimenter; called up detached visual images of familiar sounds; said 'sound' to myself several times; found myself totally unable to obtain auditory imagery of any sound for all I could obtain was a series of visualized sounds. When the bell was tapped the first experience I had was a visual image of a small, silvery cloud; could not possibly have said that it was an auditory stimulus until the visual image floated into my field of vision from the direction of the experimenter. All this happened very quickly, with exceedingly rapid shifts of attention. Then by means of eye-movement I shifted my attention vigorously in the direction of the stimulus. Then for an instant I thought I heard the sound itself as such but I noticed definitely that it was non-focal; the visual cloud momentarily vanished but far quicker than it takes to say it, it reappeared and I found that with the reappearance I was again aware of the sound. In attending to the sound I find myself focused upon visual rather than upon auditory qualities. I tried it again, while the sound was still lingering; this time I said 'sound' in vocal-motor fashion and had intensive strains in my neck of leaning toward the source of the stimulus. Here I thought I succeeded in attending to the sound itself, focally, but as before, discovered that what meant the sound to me was in reality this persisting, silvery, visual image. I was conscious of the metallic, mellow ring of the bell in terms of the silvery quality of the visual image; the intensity of the bell-sound was the brightness of the visual image; the changing quality and intensity of the bell came to me only in terms of changing quality and brightness of the silvery image. At no time was I getting away from visual qualities; they followed me no matter where I turned or what I did. The sound was present to consciousness but never with sufficient distinctness to describe. It was merely 'something' and was always a 'step behind' attention. I could tell you when the sound stopped and when it began or when it changed, but always in terms of its visual associate."
Stimulus: prolonged blast upon a small, shrill whistle. A was instructed to do his best to attend to the sound qualities as such. "I find that I can attend to the sound as long as I do not attempt to analyze it; all the sound of which I am aware is an undifferentiated 'something'; it may be 'raw' sound; it is meaningless; I interpret the experience as sound in terms of the vocal-motor image, 'sound,' or my attention shifts to motor relaxation and I find myself merely responding to a 'sound' situation. I simply cannot focalize the sound as such in the absence of visual imagery; my alertness to the sound invariably appears in terms of its visual associate. All of the above I have interpreted from what actually happened. The instant the stimulus was given I became conscious of something; but this something was at once followed by a focusing of visual attention upon color and brightness. (Then a further attempt to shift from visual to auditory qualities resulted only in shifting from one visual quality to another.) As I tried, with a vocal-motor aufgabe to single out auditory qualities I found myself attending to single visual qualities; the intensity of the sound was present in terms of the sharpness and brightness of the visual image; a quality of the whiteness of the visual cloud meant the timbre of the sound; upon each attempt to attend to auditory qualities I found that the sound itself lingered in consciousness as an undifferentiated bit of awareness; the combination of sound plus visual image constituted the sound of a certain pitch, quality and intensity. I could attend to the sound itself as long as I was not listening to anything in particular and did not try to label the sound."
So what is going on with all of this? One possibility is that the factors determining a quale are not confined to a local cortical area but are the result of network interactions. Another possibility is that qualia are learned through specific stimuli during a "critical period," and thus synesthetes process external stimuli differently from us, but have no memory of this learning process. However, after analyzing cases of sight restoration in the blind, I have found that both of the above hypotheses are likely wrong. The original "cross-wiring" hypothesis must be right about at least one thing: the activation of different brain regions is a necessary condition for the generation of different subjective experiences.
Richard L. Gregory introduced a patient7, S.B., whose sight was restored through surgery. This patient's experience is legendary. After the surgery, he did not experience the traditionally expected "Mary in the black-and-white room" scenario. Unexpectedly, he was not surprised by what he saw and could even recognize objects by sight on first viewing that he had previously learned only by touch. Although his vision was not perfect and differed from that of a typical person in some visual tests, such as distance judgment, this was still astonishing enough.
Another reported case of sight restoration in 2003 reproduced similar results8. Some neuroimaging findings also indicate that the visual cortex of congenitally blind individuals is taken over by input from other senses9, yet certain functions of the visual cortex remain the same as in sighted individuals, regardless of the modality of the information received. Studies have shown that the brain area for recognizing Braille through touch is located very close to the area used by sighted people for visually encoding written forms10.
I know this material is complex and lacks a tightly-linked chain of evidence to connect it all, but these faint connections all point towards one possibility: certain properties of the cortex are stubborn, both in terms of the conscious experiences it generates and its brain functions.
In fact, the classic experiment of swapping the auditory and visual afferent nerves in ferrets shows that although the ferret can still develop "normal" visual and auditory functions, these functions still differ from those of normally developed ferret. At the cellular level, the way neurons in the experimental group of ferret respond to stimuli also differs from the control group. It seems that the auditory cortex of the ferret does not become completely identical to a normal ferret's visual cortex in its response to visual stimuli, even if it has been exposed to them from early in life11.
These differences are likely determined by genes. In fact, the expression of nerve cell genes and receptors varies greatly across different regions of the human brain(Even within the visual cortex, there are still differences)12. Whether these inconsistencies determine what kind of subjective experience will be produced when a cortical area is stimulated later in life is a question for which there is currently no experimental evidence.
However, if this hypothesis is true, it leads to some very interesting conclusions. It would mean that the type of subjective experience generated by stimulating any given area of the human brain is determined very early in development. Later learning processes do not change this property but only modulate neural firing to better process information. Then, a person could use visual qualia to represent the properties of sound (pitch, timbre, intensity), and these representations could also be linked to the physical properties of sound.
What's more, since human development is not a completely deterministic and stable process, it is indeed possible that during development, some cortical areas that were supposed to express certain genes or receptors made an error, expressing other genes and receptors instead. If these genes are unrelated to those involved in building large-scale cortical connections and structures, it would result in a synesthete's brain structure appearing no different from a non-synesthete's on a neuroimage. This would explain why we can hardly detect robust differences in activated regions between synesthetes and non-synesthetes in many neuroimaging trials, because the difference between synesthetes and non-synesthetes lie no where in the activation location in their brains when they face same stimulations. This theory could also explain another problem that is difficult for the cross-wiring theory to address: for some synesthetes, the brain systems for processing the inducer and the concurrent are located far apart in a typical brain, making them hard to explain via cross-wiring. But if there were a genetic expression variation in the cortical area processing that sensation, then distance would no longer be an explanatory hurdle.
Einstein once said: "Subtle is the Lord, but malicious He is not." The origin and nature of qualia are perhaps the most perplexing and vexing problems in the natural world. However, I believe that nature, not being malicious, will always leave a door ajar for humanity to peek at the secrets behind it, as long as we have the wisdom and courage to push it open. If synesthetes are indeed "inverted spectrum" individuals who have fallen among us, then regardless of whether my theory is correct, they are the very people who can best provide the help we need to push open that door, allowing us to further probe the mysteries of the brain's workings and consciousness.
P.S.
This theory is very difficult to prove correct directly, but its inspirational point is that the biochemical properties of the cortex likely influence qualia. In that case, in neuromodulation, would it be possible to find a stimulus that acts on the traditional visual cortex (areas that currently only produce visual experiences with electrical stimulation) to generate a stable auditory experience, or vice versa? Or, could there be a drug that temporarily alters neural activity, thereby briefly changing the subjective experience produced by a cortical region in response to external stimuli (for example, from visual to auditory)?
If we could truly find such a thing, we could then begin to use experiments to narrow down the range of physicochemical factors that affect qualia.
Hupé, J. M., & Dojat, M. (2015). A critical review of the neuroimaging literature on synesthesia. Frontiers in human neuroscience, 9, 103.
Cytowic, R. E. (2002). Synesthesia: A union of the senses. MIT press.
Myers, C. S. (1914). Two cases of synaesthesia. British Journal of Psychology, 7(1), 112.
Downey, J. E. (1911). A case of colored gustation. The American Journal of Psychology, 22(4), 528-539.
After some consideration, I have decided not to include the link to the original post. It seems rather impolite to post someone else's content elsewhere. However, if anyone is genuinely interested, I will contact the original poster.
Wheeler, R. H., & Cutsforth, T. D. (1922). Synaesthesia, a Form of Perception. Psychological Review, 29(3), 212.
Gregory, R. L. (2015). Eye and brain: The psychology of seeing.
Gregory, R. L. (2003). Seeing after blindness. Nature neuroscience, 6(9), 909-910.
Sadato, N., Pascual-Leone, A., Grafman, J., Ibañez, V., Deiber, M. P., Dold, G., & Hallett, M. (1996). Activation of the primary visual cortex by Braille reading in blind subjects. Nature, 380(6574), 526-528.
Dehaene, S., & Cohen, L. (2011). The unique role of the visual word form area in reading. Trends in cognitive sciences, 15(6), 254-262.
Sharma, J., Angelucci, A., & Sur, M. (2000). Induction of visual orientation modules in auditory cortex. Nature, 404(6780), 841-847.
Amunts, K., & Zilles, K. (2015). Architectonic mapping of the human brain beyond Brodmann. Neuron, 88(6), 1086-1107.