What Is Vision and Art

The field of vision and art has two broad goals: (a) to inform an understanding of art and artistic practice in terms of fundamental neural mechanisms of vision and (b) to supplement knowledge of visual information processing through a close reading of art works. As Gregory et al. (1995) remarked, “The scientist can learn at least as much from the artist as the other way round.” The field of vision and art does not seek to use science to adjudicate works of art or experiences of them but rather to provide an alternative lens to consider art and brain, to deepen the mystery and wonder of both. The field has roots in gestalt psychology (Arnheim, 1974) and formal analysis, and it leverages contemporary approaches in neuroanatomy, neurophysiology, functional brain imaging, psychophysics, artificial intelligence, cognitive science, and machine learning.

An enterprise with similar origins to vision and art is the blossoming field of neuroaesthetics, which aims to uncover the biological bases of aesthetic experiences (Chatterjee & Vartanian, 2014; Pearce et al., 2016). Neuroaesthetics is rapidly expanding, encompassing domains beyond visual art to include dance, architecture, theater, literature, and art therapy (Magsamen, 2019). A preoccupation of neuroaesthetics has been the identification of brain regions that are engaged during aesthetic experiences (Iigaya et al., 2020); these same brain regions are engaged in many behaviors associated with reward and decision-making, and they are not constrained to an appreciation of art (Skov & Nadal, 2018). Skepticism toward neuroaesthetics derives from its use of science as an instrument to legitimize (or delegitimize) subjective experiences (Zeki et al., 2020). The field of vision and art has more circumscribed objectives compared to neuroaesthetics, endeavoring to connect biological mechanisms of vision in health and disease with the strategies that artists use to make artwork (Cavanagh, 2005; Conway & Livingstone, 2007; Livingstone, 2002; Marmor & Ravin, 2009). Vision and art sidesteps questions of aesthetics because there can be no consensus about whether “aesthetics” concerns perception, beauty, or art (Conway & Rehding, 2013, Skov & Nadal, 2021), and ideas of “beauty” do not generalize (Germine et al., 2015).

The visual system of primates evolved to look at things, not to make visual representations of things. Indeed, making art is a peculiar behavior, perhaps unique to humans. The field of vision and art considers not only finished artworks and how they are seen but also the process of making a work, for it is through the act of art production that artists uncover how vision works (Cavanagh et al., 2013). Because of space constraints, the current discussion focuses on mechanisms of vision in the healthy visual system. For the interested reader, there is considerable work connected with the field of vision and art that takes up various disease states and what they reveal about how vision works and the impact of aberrations in visual function on art practice—for example, in strabismus (Livingstone & Conway, 2004, 2005; Livingstone et al., 2011; Tyler, 2019), neurological damage (Chatterjee, 2004; Mell et al., 2003; Miller et al., 1998), mental illness (Henemann et al., 2017; Huddleston & Russell, 2015; Thys et al., 2014; Turkheimer et al., 2020), autism (O’Connor & Hermelin, 1990; Pring & Hermelin, 1993; Snyder & Thomas, 1997), pharmacological intoxication (Dan, 2003; Jones, 2007; Kempley, 2019; Ladisich, 2018; Ten Berge, 1999), other eye diseases (Elliott & Skaff, 1993), and aging (Ravin, 1994; Werner, 1998).

Retinal Images, Vision, and Art

The images on the retina are not what we see. Instead, vision depends on extensive networks in the brain that interpret retinal images. Perdreau and Cavanagh (2012) illustrate this point with the example of size constancy: The area on the retina occupied by an object changes dramatically with changes in distance between the object and the viewer, yet we do not see the object as growing or shrinking. Instead, the visual system uses the retinal images as a prompt for vision, interpreting the retinal signals in light of contextual information, some of which is not visual. The brain generates vision of an advancing object or person by incorporating shape, perspective, and motion cues, together with knowledge that objects and people do not typically grow or shrink, to generate a representation that is constant in size.

We see objects not only as constant in size but also as whole, even if in the retinal image the objects are occluded. Images made by artists bear hallmarks of visual processing, and they provide clues to cognitive goals of vision. For example, renderings of scenes, especially by children, will often show objects and people separated from each other, not overlapping as they would likely exist in the retinal image of the scene. Depicting the items as isolated reflects the culmination of visual processing, which generates for each item a complete visual idea, a symbol that reflects both the retinal input and its behavioral relevance. The drawing therefore reveals what it is about the retinal image that holds meaning for the artist.

Starting very early in development, what we see and what we draw are influenced by what we know about the world, which presumably reflects what aspects of the world are important (Dillon, 2021; Vivaldi & Salsa, 2021). Figure 1 shows a typical drawing by a boy aged 4 years of a giraffe (the boy is my son, Benjamin). Figures drawn by children at approximately this age do not typically include necks, perhaps because the relevance of human necks lags that of other features such as eyes and limbs. At age 4 years, the boy leans on his limited experience in his depiction of the giraffe, omitting the attribute of the giraffe that, with more experience, we come to appreciate as a defining feature. Visual works made by artists, even art expressly created to be realistic impressions such as work by Claude Monet, are not veridical representations of retinal images. They are much richer windows into vision, brain, mind, and cognition.

Open in new tab

Figure 1. Giraffe, by Benjamin (age 4).

According to his friend and confidant Lilla Cabot Perry, Monet “said he wished he had been born blind and then had suddenly gained his sight so that he could have begun to paint . . . without knowing what the objects were that he saw before him” (Perry, 1927, p. 120). Monet’s fantasy reveals a deep misunderstanding about the relationship between retinal images and vision. In the 21st century, Monet’s paintings might be regarded as accurate representations of scenes. But they were not always appraised this way. The appreciation of Monet’s work—what critics see in his pictures—has itself changed because of cultural exposure and knowledge.

Artists who are interested in rendering visual images must wrestle with the brain operations that turn retinal images into knowledge. As Monet advised (as cited in Perry, 1927, p. 120),

When you go out to paint, try to forget what objects you have before you, a tree, a house, a field or whatever. Merely think, here is a little square of blue, here an oblong of pink, here a streak of yellow, and paint it just as it looks to you, the exact color and shape, until it gives your own naïve impression of the scene before you.

The strategies that artists use to undo the impact of knowledge in vision provide valuable insight into how vision works. Throughout this article, I return to this theme: the ways in which artists attempt to circumvent the normal operations of vision, such as the proscription by Monet to “merely” paint patches of color rather than objects, and the evidence in artworks that vision is not a process of truthfully encoding retinal images, whatever “truthful” might mean.

The Brain as Medium

Oil paint, charcoal, tempera, watercolors, spray paint, ink, graphite, and video. These are some of the many media that visual artists might work with. Each medium represents a choice for the way information will be encoded, stored, and transmitted. Oil paint is a serious medium. Oil paint is stuff that takes up three dimensions, and oil paintings command attention. Consider the monumental abstract paintings of Joan Mitchell or the expansive water lily paintings by Claude Monet, in which the images are created by overlapping layers of opaque marks created by dabbing, smearing, scraping, and pushing the paint around. During making, each mark might be mixed with, or cover, the color of the underlying mark, while retaining something of the history of brushwork texture. The painting encodes the battle of its making. Oil painting is a formal lecture. Watercolor, meanwhile, retains the immediacy, transparency, and intimacy of dribbling water. Think of John Singer-Sargent’s Corfu Lights and Shadows, in which the color at many locations in the finished work derives from overlaid transparent gestures, or from paint bleeding between adjacent areas, and where the texture of the painting exists on a separate dimension from the paint, determined solely by the grain of the paper. The painting is fragile, informal, conversational. By comparison, spray paint is coarse and aggressive, reflecting the gesture of the arm swinging irreverently through space, sticking to almost anything, a yelled command. Video is light produced, rather than reflected, and obligates the viewer to stick around to watch in a darkened room, meditating often in silence. These examples show that the choice of medium privileges communication of certain information while being constrained by specific limitations. There is a synergy, in which the message dictates the medium, and the medium helps shape how the message is encoded. For the viewer—the consumer of the information—an analysis of the medium can reveal much about what information the artist wants to convey: a lecture, a conversation, a command, a meditation.

There is a medium common to all art making: the brain. The field of vision and art contends that an analysis of the brain can inform our understanding of art and, in a parallel synergy, that an analysis of art can shed light on neural operations. Art does not come about in an instant but rather reflects a sequence of decisions. Art, like the brain mechanisms that give rise to perception and cognition, is inevitably a process that unfolds with temporal contingencies. Neuroscientists sometimes bemoan conclusions based on anatomy alone because the anatomy is silent on function; anatomy can provide clues to physiology, but it cannot reveal physiological operations. By the same token, studies of finished works of art are limited. To understand art, we need to consider the “physiology” of art—how the art came to be, the dynamic interaction of vision and visual–motor feedback engaged by the artist in the process of making work, the sequence of decisions engaged by the artist, as well as the network of experiences leading up to the art-making act. Much of these data are not available from a close reading of the finished work, which imposes a challenge to a comprehensive study of vision and art. But clues from artists’ statements, and observations of artists at work, allow us to speculate about the processes of art production—the creative events, often private, that give birth to a work of art.

A Heuristic for Vision and Art

A key tenet of the field of vision and art is that both visual neuroscience and art involve empirical activities directed at understanding visual experience. Artists and vision scientists go about their business in many different ways, but a common theme is experimentation: an exploration through trial and error to determine what works. David Hockney (1993) says, “I keep pictures I have done around the studio . . . it takes a while to realize what I really did there, how it works; then I may use that in something else” (pp. 130–131). The goal of a vision scientist is an understanding how vision works; the goal of the artist is a product that inspires engagement. But for both, the process is not arbitrary or capricious, but involves deliberation in pursuit of reproducible behavior. The central hypothesis of the field of vision and art is that artistic output provides data about brain function. The decisions that artists make in creating their work and the artworks that people choose to look at provide evidence of underlying brain processes.

From the perspective of the vision and art field, the artistic process implicitly involves breaking the computations in people’s heads that are often taken for granted—shattering the veneer that people’s experience is whole, complete, and unified. This “breaking” of the mechanisms of perception is an important step toward making something new, and it is an example of a general principle of creativity—it is seen in the toddler smashing a tower of blocks to figure out how it is built; the jazz ensemble breaking rules to make new music; the scientist knocking out genes to determine what they do; and the artist cutting up the world to deepen people’s understanding of it.

The brain is the most complicated machine in the universe, capable of a hundred million giga floating point operations per second—that is more than 10 million times the capacity of an iPhone and still far exceeds the world’s largest computers. This complexity makes neuroscience difficult, but the difficulty is exacerbated because, paradoxically, the operations of the brain seem so effortless. The distance between the workings of the 2.7 pounds of jelly in one’s head and one’s experience of any image is so great that it seems like the two are unrelated. It is as if evolution has built the brain to protect people from knowing how it works. Ironically, it is the brain’s complexity that enables people’s visual experience to be so simple. But the simplicity is an illusion, just as the simplicity of the iPhone is an illusion. The designers at Apple have obscured how the iPhone works because that knowledge is not relevant to people’s use of the device and might even interfere with the experience of it. It is conceivable that selective pressures have similarly made interrogation of brain function challenging. Nonetheless, the artist must deconstruct the brain’s inscrutable computations to create art. Consider a painter. She typically starts with a blank canvas and must build up an image using a sequence of marks. Each gesture represents a decision that is evidence of many neural events, reflecting an interaction between memory (about what visual marks might be effective for desired visual objective) and vision (Are the marks that have been made effective?).

One heuristic for approaching vision and art is provided by the operation of vision and the organization of the visual pathways. For each stage in the visual-processing hierarchy—from eye movements to retinal encoding, through cortical mechanisms in primary visual cortex and extrastriate cortex, culminating in executive functions and memory—one can ask how the biology informs one’s understanding of visual art, and how visual artists have exploited aspects of how the brain processes visual information. In some cases, the art is evidence of a visual-processing mechanism. In other cases, the art suggests that the artist is leveraging their understanding of how visual processes work to achieve a creative objective. The strategies exploited by artists often serve to undo the operations of the visual system that give rise to perceptual constancies (Perdreau & Cavanagh, 2011). This article provides examples of low-level and high-level vision to illustrate both cases and to sketch out the usefulness and breadth of the field of vision and art. But first a note of caution.

Limitations of Vision and Art

My daughter was four years old when she discovered that yellow paint mixed with blue paint looks green. Her expression was one of confusion, surprise, and delight. Without prior experience, there is nothing from the appearance of blue and yellow that by itself predicts the color of the mixture. But as compelling as the experience of color mixing appears, it does not shed any light on brain function. Yet color mixing has often been used to support an argument about how color is encoded by the retina. For example, it has been said that the existence of three classes of cone photoreceptors follows from the convention of three primary paints in the artist’s kit. The link with neural processing is spurious: Color mixing does not provide any evidence whatsoever about how the brain captures spectral information. The magic of color mixing uncovers the predictable way in which light is absorbed, reflected, and filtered depending on the physical properties of pigments and the wavelength of light. The yellow paint absorbs all wavelengths except those in the middle of the visible spectrum, whereas the blue paint absorbs all except very short wavelengths. When mixed, the filtering properties of the two pigments combine to absorb light over almost all the visible spectrum except for a relatively narrow region sandwiched between the blue and yellow filters, centered on what is called green. The explanation for color mixing therefore derives entirely from the laws of physics. Other examples from art could be misapplied to generate or evaluate hypotheses about brain function. But one must be on guard.

Given the complexity of the brain, and of the artistic process, it is naive to expect direct and comprehensive relationships across the disciplines of visual neuroscience and art. Indeed, scientific explanations may end up limiting the appreciation of what artists are doing. Again, caution is warranted. In addition, success in the field depends on a mutual respect for the traditions of the respective disciplines. Terms may be used differently in different fields, and terms borrowed from one field may be misapplied in another. For example, the neurophysiological term “receptive field” is defined as the part of the visual field to which a given neuron is responsive. In panel discussions about art, I have heard the term misapplied to mean the experiences to which an observer is sensitive.

Since we cannot typically obtain objective data on how most artists work or on the neural mechanisms in their brains when they work cannot be obtained, the analysis of artists’ practices is necessarily indirect. As such, it is useful to recognize that the primary benefit of the field of vision and art is the generation of hypotheses, not universal or reductive truths about art.

Eye Movements and Looking Behavior

Leonardo da Vinci aptly described the process of vision thusly:

The organ of sight is one of the quickest, and takes in at a single glance an infinite variety of forms; notwithstanding which, it cannot perfectly comprehend more than one object at a time. For example, the reader, at one look over this page, immediately perceives it full of different characters; but he cannot at the same moment distinguish each letter, much less can he comprehend their meaning. He must consider it word by word, and line by line, if he be desirous of forming a just notion of these characters. In like manner, if we wish to ascend to the top of an edifice, we must be content to advance step by step, otherwise we shall never be able to attain it.

Vision does not operate by having the eyes uniformly, systematically, scan a scene like a dot-matrix printer. What one sees depends instead on where the eyes are directed, a process that reflects goals and attention. The retina is not homogeneous (Figure 2, left). The portion of the retina that captures the center of gaze is defined by the highest density of cone photoreceptors; moreover, each retinal ganglion cell at the fovea samples a relatively low number of photoreceptors. The consequence of these anatomical adaptations is that the fovea has higher visual acuity than the visual periphery (Figure 2, right). But peripheral vision is not just a poor version of foveal vision. Instead, peripheral vision is adapted to encode low spatial frequency information that is invisible to the fovea.

Open in new tab

Figure 2. (Top left) Fundoscopic image of the healthy human retina. (Top right) A chart of letters scaled to equate for visual acuity across eccentricity. (Bottom) Histological cross section of the peripheral and fovea regions of the human retina.

Source: Top right figure from Anstis (1974).

Emotional expressions are conveyed predominantly through low spatial frequency signals that accompany deep muscle contractions during smiling and frowning. Makeup artists exploit this by applying rouge under the cheek bones. As an aspiring actor in high school, I recall one of the girls criticizing how I had applied my makeup—she pointed out that one should not be able to “see” the makeup. Her point was that the makeup is effective when it is only visible to the peripheral retina, away from the focus of overt attention. Hybrid images composed of low spatial frequency filtered versions of a scene under one condition merged with high spatial frequency versions of the scene under a different condition illustrate the extent to which foveal and peripheral signals can be dissociated (Oliva et al., 2006). One intriguing idea is that da Vinci exploited the different adaptations of peripheral and foveal vision in his Mona Lisa, capturing her smile with a low spatial frequency filtered version of a smiling face superimposed on a high spatial frequency filtered version of a neutral face (Livingstone, 2000).

Patterns of spontaneous fixations are fairly consistent across people (Mannan et al., 1997). The behavior exploits the adaptations of the fovea so as to register on the fovea the portion of the visual scene that is likely to engage overt attention (Eckert & Buchsbaum, 1993, Reinagel & Zador, 1999). Areas of visual salience, such as defined by high-contrast and high spatial frequency content, compete for fixations. Moreover, the pattern of eye movements that sample an image unfold in time. Artists can leverage the biology of eye movements to guide a viewer’s looking behavior by creating structure across the image that instructs the viewer where to look.

Unsurprisingly, knowledge and task impact eye movement patterns. An expert bird watcher will likely spend more time searching for birds in a photograph, even if the birds are tiny specks in the scene. Similarly, artists might track images differently from non-artists, presumably because they have different goals (Vogt & Magnussen, 2007). Nonetheless, certain image statistics recruit fixation. Consider the two paintings in Figure 3. The picture by Matisse (Figure 3, left) comprises a range of spatial frequencies. Near the center of the image is an area of higher spatial frequency (and high local contrast) created by a diagonal grate painted with thin black lines. This region is surrounded by areas of increasingly large spatial frequency: the small white parallelogram and white triangle to the left of the grate, followed even further left by a larger area of yellow-ochre, and even further by a swath of turquoise, and finally at the left margin of the painting by a broad band of blue. The size of these regions tracks the size of the letters scaled to account for the drop in acuity with eccentricity (see Figure 2, right; Anstis, 1974). Spontaneous, uninstructed eye movements are biased toward the central grate region (Figure 3, bottom left) because of the statistics of that part of the image. Matisse tucked a sketch of a figure in the broad band of blue on the left side of the canvas—a figure that most viewers do not see immediately, but that they cannot avoid seeing once it is discovered. The consequence is a dynamic play of eye movements, jostling between the central grate of high spatial frequency (a bottom-up driver of eye movements) and the figure (a top-down driver of eye movements) (Lafer-Sousa & Conway, 2009; Massaro et al., 2012).

Open in new tab

Figure 3. (Top left) Entrance to Kasbah (1912) by Henri Matisse, oil on canvas. (Top right) Undredal (2002) by David Hockney, watercolor on paper 138.5 × 61 cm, three sheets. (Bottom) Gaze pattern of a macaque monkey viewing images of the paintings on a computer monitor (30-s viewing time; image size ~17 × 25 degrees of visual angle; eye-tracker sampling at 1000 Hz; data collected by Reza Azadi).

Hockney plays a similar game as Matisse in taming the viewer’s gaze, but he breaks with the traditional compositional placement of the focal point near the center of the canvas, placing it instead at the very bottom of the work (Figure 3, right). Hockney’s painting is a watercolor made of three sheets of paper that are stitched together. Regardless of where the viewer’s gaze is launched, it immediately sinks, tracking the cascading marks of the mountains of the middle sheet to the sea, finding rest in the field of colorful, high spatial frequency marks growing in front of the houses in the bottom of the bottom sheet. As with the Matisse painting, part of what makes Hockney’s picture successful is the dramatic variety in the scale of the marks and the consequent dynamic looking behavior that is elicited. In the case of Matisse, the looking behavior is radiating from the central focal point, inviting the viewer into the picture; in the case of Hockney, the pattern of fixations is rich in vertical motion, embodying the dramatic vertical rise of the mountains.

Where someone looks is influenced by many factors, not only image statistics but also task (Yarbus, 1967) and subjective preferences (Mitrovic et al., 2020). Presumably, the content of the image also shapes the temporal dynamics over which an image is experienced, although the extent to which these temporal dynamics are consistent for a given person, and among different people, is unclear (Mannan et al., 1997). These considerations highlight the possibility that pictures, although static objects, are not experienced in a single instant in time. Static objects such as paintings may share something in common with movies in terms of the dynamics of retinal stimulation: A movie can be experienced by holding the eyes fixed while the image changes over time; a painting is experienced by holding the image fixed while the eyes move. The artist may use various strategies, such as the range of scale of marks, to engage a viewer’s visual interest, and it may be fruitful to unpack these strategies in quantitative detail (Kesner et al., 2018; Leonards et al., 2007; Massaro et al., 2012). The impact of the temporal dynamics of looking behavior on how viewers appreciate art, and the strategies that artists use to command the temporal dynamics of eye movements, remain almost entirely unexplored areas of study.

Eye Movements and Image Making

In an approach made famous by Yarbus (Figure 4, top), vision scientists often consider the impact of image structure on where eyes look, as exemplified by the analysis of the Matisse and Hockney pictures in the section on “Eye Movements and Looking Behavior.” But what impact does the biology of eye movements have on how artists make images? Put yourself in the shoes of Hockney making his picture. The three sheets of paper are blank. Where do you start? How does the pattern of your own eye movements—and what you want to look at—influence what kind of marks you make, and how you proceed across the surface to create the work? Keep in mind that the artist is not copying a picture already made, but creating one. To answer these questions, consider another act of creation: people writing on whiteboards. The creators in this act will typically look at what they are writing because doing so provides visual feedback to the motor commands of the arm and hand; they will typically use similar scale marks with similar spacing throughout, maintaining consistent visual interest. These behaviors of creation echo the behavior desired in the viewer: sequential, consistent, uniform attention across the board.

Open in new tab

Figure 4. (Top) Eye-tracking pattern of a human subject looking at an image of the bust of Nefertiti. (Bottom left) Portrait of Gerti Schiele at the Age of Twelve (1907) by Egon Schiele, charcoal on paper. (Bottom right) Two Sisters (1918)(Maria and Eva Steiner) by Egon Schiele.

Source: Top figure from Yarbus (1967). © John Baldessari 1969. Courtesy Estate of John Baldessari © 2022. Courtesy Sprüth Magers. John Baldessari. Commissioned Painting: A Painting by Jane Moore, 1969. Oil and acrylic on canvas. 150.5 × 115.6 cm. 59 1/4 × 45 1/2 in. John Baldessari. Commissioned Painting: A Painting by Pat Nelson, 1969. Oil and acrylic on canvas. 150.5 × 115.6 cm. 59 1/4 × 45 1/2 in.

How does the artist avoid inevitably making a picture that looks like a uniform field of high spatial frequency marks demanding consistent visual attention? The artist must somehow break the normal pattern of behavior if they seek to make a work with the varied scale of marks needed to create a dynamically interesting pattern of eye movements in viewers of the finished work. For the Hockney painting, some areas of the painting will require different levels of technical facility and be arguably less interesting to create. Consider the top sheet of Hockney’s three-panel work, which is almost entirely occupied by one monochrome area of gray sky. Painting that sheet must have been a very different kind of experience compared to painting the bottom sheet—challenging in its own way but perhaps more boring. Yet without the top panel, the painting is much less interesting to look at. In speculating about the role of eye movements in creating a work—as opposed to simply considering the role of eye movements of a viewer looking at a finished work—it can be appreciated that the task for which looking behavior evolved is different from the one that the artist engages in when they make art. I once asked Hockney about the strategy he uses to circumvent the normal patterns of behavior in creating work, and he said, “I suppose I just like painting. I would enjoy painting a door.”

The development of the work of the Austrian expressionist Egon Schiele provides clues that may support the notion that the artist must learn to undo patterns of normal behavior. An early drawing by Schiele of his sister, created when the artist was 17 years old (Figure 4, bottom left), is realistic but not photographic. It is composed of a few overworked lines that correspond to those that the brain uses to define the face—the eyes, nose, mouth, and face contour. The marks that Schiele uses to create the work track the eye movements one might expect from a viewer looking at the sitter herself: Compare Schiele’s drawing with the eye trace by Yarbus. I speculate that Schiele’s hand in creating the work is tracking his eye, natural behavior akin to writing on a whiteboard. The result is that the drawing is a distorted copy of reality, distorted by the mechanisms used by the brain during vision that make the visual features salient. The problem is that this distortion is doubled when one then looks at the drawing: In the viewer’s mind, the image becomes a distortion of a distortion through a positive feedback loop. The original retinal image is passed through the visual system of the artist to extract the edges, and then the rendering of the image in the drawing, which shows exaggerated edges, is processed by the visual system of the viewer of the artwork whose own visual system further emphasizes the edges and contours. The drawing is instructive about the process of vision, but it is not the work that made Schiele a great artist.

Schiele’s creative genius derived instead from drawings such as the one shown in the bottom right of Figure 4. The drawing is considered a masterpiece: elegant and visually arresting. It is distinguished by the extraordinary restraint Schiele shows in crafting the image. The line around the face is not overworked. Instead, at some locations where one would expect a line, as by the chin, Schiele omits it. By leaving out sections of the line, Schiele creates a puzzle that requires a viewer’s visual system to complete. And in so doing, Schiele successfully achieves what a critic might call “visual interest.” The evolution of Schiele’s drawing practice suggests how the creative process depends on undoing normal visual computations.

Eye movements must impact all stages of image making, and an acknowledgment of this might prompt a re-evaluation of the causes underlying certain observations. For example, Christopher Tyler (1998) observed that painters often center one eye in portraits. Eye-centering may have its roots in “hidden principles [that] are operating in our aesthetic judgments,” as argued by Tyler (p. 877), or it may simply reflect the fact that the artists themselves are fixating on one eye (because you can only look at one eye at a time) and placing it in the center of the canvas because, absent any other visual cues (as the blank canvas appears to the artist), the center is where viewers look (Conway & Livingstone, 2007).

Looking Behavior, Artistic Choices, and Conceptual Art

The choice about where to look represents a fundamental aspect of art making. John Baldessari, who is considered the godfather of conceptual art (Shaw, 2020), underscored this point in his series of Commissioned Paintings (Figure 5). The set of paintings have been interpreted as satire (Johnson, 1998), but that assessment misses an important part of the story. The work is a serious attempt to uncover “some bedrock ideas about art” (Stiles & Selz, 1996, p. 891). Baldessari says that “one of the things that . . . I arrived at was the ‘choice’—that seems to be a fundamental issue of art. We say this color over that color, this subject over that or this material over that . . . so much of my work . . . is about the moment of decision” (cited in Stiles & Selz, 1996, p. 891). To create the Commissioned Paintings, Baldessari hired a dozen county fair artists to paint enlarged, faithful copies of photographs that he took of his friend George Nicolaidi, pointing out “the things in a visual field that jumped out at him.” Baldessari “was exploring the impetus to do art—first you have to choose and select.” The photographs “weren’t about beautiful composition . . . [they were] just recording the fact of something being selected” (Davies & Hales, 1996, p. 97). The work is substantially about where people look. And it was an attempt to undo the normal patterns of visual selection engaged by Sunday painters. Baldessari described his “fondness for amateur painters,” saying that “some of these painters are pretty good, they’re just showing in the wrong venues, and they’re painting the wrong subject matter (Davies & Hales, 1996, p. 97).” Across art styles, artists are confronted with a challenge to understand the behavior of visual selection, and in some sense to undo it in the process of re-creation. The steps in vision that happen after the brain has decided where to direct the eyes are discussed in the section on “First Steps of Post-Receptoral Processing.”

Open in new tab

Figure 5. (Left) Commissioned Painting, a Painting by Pat Nelson (1969) by John Baldessari, oil and acrylic on canvas, 150.5 × 114.3 cm. (Right) Commissioned Painting, a Painting by Jane Moore (1969) by John Baldessari, oil and acrylic on canvas, 150.5 × 114.3 cm.

Source: © John Baldessari 1969. Courtesy Estate of John Baldessari © 2022. Courtesy Sprüth Magers. John Baldessari. Commissioned Painting: A Painting by Jane Moore, 1969. Oil and acrylic on canvas: 150.5 × 115.6 cm; 59 1/4 × 45 1/2 in. John Baldessari. Commissioned Painting: A Painting by Pat Nelson, 1969. Oil and acrylic on canvas: 150.5 × 115.6 cm; 59 1/4 × 45 1/2 in.

First Steps of Post-Receptoral Processing: Local Contrast and Edges

The only part of the visual system that operates like a camera is the front end of the eye: The pupil, cornea, and lens focus an image on the retinal sheet lining the back of the eye. At each point in the retina, the miniscule light detectors—photoreceptors—are jam packed next to each other like eggs in an enormous egg carton, all facing the incoming light. Each photoreceptor responds to light corresponding to a restricted patch of the visual field, the part of the retinal image that happens to coincide with the location of the photoreceptor. The spatial extent of the visual field to which a visually responsive neuron responds is called the cell’s receptive field. Photoreceptors have very tiny receptive fields, and they are arranged topographically across the visual field because of the optics of the eye: The receptive fields of adjacent photoreceptors are next to each other. Photoreceptors contact bipolar cells, each of which integrates the pattern of responses across a small set of photoreceptors, resulting in a slightly larger receptive field compared to that of a photoreceptor.

Bipolar cells, and almost all neurons in the visual system, are sensitive to the pattern of light cast within their receptive fields. Some bipolar cells respond to small light increments centered in their receptive fields (ON cells), and other cells respond to small light decrements centered on their receptive fields (OFF responses) (Figure 6, top) (Ammermuller & Kolb, 1995). These response properties give rise to the spatial structure of bipolar receptive fields, and they derive from the pattern of excitatory and inhibitory connections between photoreceptors and bipolar cells.

Open in new tab

Figure 6. (Top) Schematic showing the responses of ON- and OFF-center bipolar cells to spots of different sizes and annuli (the drawing is based on data reported by Ammermuller and Kolb, 1995). (Bottom) Woman Reading (1882) by George Seurat, Conté crayon and white chalk, 31.5 × 23.7 cm.

Bipolar cells connect to retinal ganglion cells, which carry visual signals as discrete electrical impulses, called action potentials, into the brain. The area of the cerebral cortex that is the first to receive visual signals is called primary visual cortex or V1. Each single neuron in V1 receives input from a precise sample of retinal ganglion cells, giving rise to V1 receptive fields that respond selectively to the orientation in visual space of a bar or edge (Figure 7, top) (Hubel & Wiesel, 1962). Some V1 cells respond to vertically oriented bars, others to obliquely oriented bars, and so on. The discovery of orientation-selective neurons won Hubel and Wiesel the Nobel Prize in 1982 (Hubel, 1982). V1 cells are arranged across the cortical surface in a systematic pattern organized by orientation preference that is one of the most beautiful discoveries of the 20th century (Figure 7, bottom) (Blasdel, 1992).

Open in new tab

Figure 7. (Top) Schematic showing how axons from four center-surround receptive fields from the lateral geniculate nucleus converge on a cell in primary visual cortex (dashed line) to create an orientation-selective receptive field. (Bottom) The organization of orientation selectivity in primary visual cortex across the surface of the cortical sheet.

Sources: Top figure courtesy of David Hubel and Wiesel (1962). Bottom figure courtesy of Gary Blasdel (1992).

What We See, and What We Do Not See

The history of visual neurophysiology is instructive about the prejudices people have regarding how vision works. The location in the brain of primary visual cortex was known for a long time, from stroke patients. But how the cells in V1 work was a stubborn nut to crack. Early studies tested cell responses to large fluctuations in light level achieved essentially by turning the room lights on and off. But the cells did not respond very well, which was surprising to the investigators: After all, visual cortex should respond to light, and the light levels were being dramatically changed. It was a puzzle. The discovery that V1 cells do not respond to uniform changes in light but instead to very specific patterns of light within their receptive fields was profound: Vision is not about detecting absolute light levels but instead about detecting changes in light level on a very local spatial scale.

At a fundamental level, the patterns of connections between photoreceptors and bipolar cells, and how these retinal signals get wired up to neurons in primary visual cortex, governs what we see. My first art teacher told me that, in a picture, the darkest dark is almost always next to the lightest light. The reason this is so is provided by the way bipolar cells are hooked up to the photoreceptors. The resulting center-surround receptive fields respond to local spatial changes in luminance contrast. An ON-center cell will respond well to a very tiny spot of light, but it will respond worse as the spot grows in diameter; it will respond very poorly (indeed it will be suppressed) by an donut of light centered on the cell’s receptive field (Figure 6, top). As a result, the cell will respond poorly to light that falls on both the receptive-field center and the receptive-field surround. The strongest response will be to a stimulus of high local contrast: the darkest dark next to the lightest light. OFF cells work like ON cells but with the opposite tuning: They are excited by small spots of dark surrounded by light. ON and OFF cells are not the only kind of receptive fields found in the retina, but they are very common.

At the risk of entertaining teleology, one can ask why vision works the way it does, using center-surround receptive fields to encode the retinal image. One idea is that this strategy confers sensitivity to transitions between light and dark, which is useful because such transitions signal boundaries of objects. Another idea, not mutually exclusive, is that it prevents people from discriminating absolute levels of uniform illumination such as changes in ambient light levels, which are not meaningful for behavior. The average absolute light level encountered in normal vision ranges from a meager 10 flux on an overcast day to more than 100,000 flux in bright sunshine. People are grossly insensitive to discriminating among these levels, which is adaptive: The precise level of light is simply not useful for the tasks people use vision to accomplish, such as finding food, navigating through the environment, and detecting friends. The gradual fluctuations in light level across most walls are also virtually invisible to people, and similarly meaningless (it is known that light levels vary across most walls because they can be measured with light meters). People are, however, well-equipped to distinguish abrupt lightness boundaries at almost any average light level, and at any location on the wall, which is adaptive because lightness boundaries define objects, places, and faces. Regardless of the answer to the question, it can be safely concluded that the way the visual system is wired up determines what people see and what they do not see. And knowledge of how vision works provides insight into how artists make images.

Discovering Lines

Considerations of the field of vision and art often point to the use of line drawings such as those by Egon Schiele as examples of how the mechanisms of vision are exploited by artists (Sayim & Cavanagh, 2011). There are few lines in real scenes, yet artists can effectively depict scenes simply by using lines. The discovery that lines can be exploited in this way dates to the origins of art, evident in cave paintings that are tens of thousands of years old. The effectiveness of lines cannot be chalked up to cultural convention: Line drawings of animals are not used as symbols, like text, where graphic marks stand in for something else. Rather, the lines coalesce to form an image that is seen to be the thing represented. In the line drawing of a cow, one sees a cow. Linking the line drawing and the actual object requires no obvious training. Indeed, monkeys and apes can effortlessly recognize objects depicted in line drawings, and neurons that respond only when one looks at complex objects such as faces also respond to line drawings of the objects (Desimone et al., 1984). Why are lines effective? In part, they are effective because they activate the visual system in a way that is similar to how the real world activates it. Each location of the cortical surface comprising V1 corresponds to a location in the visual field; and for each location, there is a panel of orientation-tuned neurons, at least one neuron for every orientation. A given image projected on the retina will activate just those neurons whose receptive fields align with the location and orientation of the abrupt transitions in the image. From the pattern of activated neurons the brain can therefore encode the contours of objects in the retinal image. The retinal image cast by a real object and a line drawing of the same object will activate a similar set of orientation tuned neurons in V1.

Making Things Look Lighter and Darker Than They Are

The extent to which light is absorbed or reflected from the surface of a drawing obviously determines the structure of the image and constrains what can be represented. It is not physically possible for reflective surfaces, such as paper or canvas, to accurately capture the range of light levels evident at any given instant in most scenes. If you are reading this text on a computer monitor in your living room, the range of light levels you experience in the room will extend from a meager 0.1 lux for a shadow under the couch to 1,000 flux of the monitor—four orders of magnitude. By comparison, the light reflected from a charcoal drawing on a wall in the same room might only vary from 10 lux for a black mark to 100 lux for the white paper—just two orders of magnitude. The compression inherent to working with a reflective surface means that an instinctive goal, “to just draw what you see,” is impossible. In almost all cases, one cannot, with charcoal and paper, render a representation of a scene that perfectly reconstitutes the absolute light levels across the scene.

Artists exploit the way the visual system encodes visual information to deal with this challenge, to create the impression of a greater range of light levels in the work than is physically possible. Seurat illustrates some of the tricks in his drawing Woman Reading from 1882 during the rise of modernism (Figure 6, bottom). The figure is defined by abrupt transitions in light level: On the left side, the woman’s shoulder is dark against the light background; on the right side, the woman’s shoulder is light against a dark background. Neurons with receptive fields aligned with these abrupt transitions will respond well. An OFF-center receptive field will respond well to the left shoulder: The dark marks would activate the receptive field center, and the light background would not suppress the receptive field surround. Neurons with receptive fields at other locations in the image will respond less well. The upshot is that there will be a sequence of neurons whose receptive fields trace out the contours of the woman’s body that will be more active than the rest, and it is this pattern of activity that gets stitched together by the brain to convey the form.

Drawing What We Do Not See

In Seurat’s Woman Reading drawing, how does he manage to have the body of the woman be both dark (on the left side of her body) and light (on the right side of her body)? He does so by creating a gradual transition in gray value across the body of the woman—a transition that is not clearly visible because it is not an effective stimulus for center-surround receptive fields.

How, and why, does Seurat draw these gradual transitions in gray value? These gradual transitions are not a very effective visual stimulus. How does Seurat see them? And why does he choose to create them if they do not activate visual neurons very well? In doing so, Seurat effectively undoes the edge calculation that is an automatic feature of how V1 works. Rather than showing the edges of the object as other artists might do using lines, Seurat re-creates the conditions for a viewer’s visual system to extract the edge from the image, using areas built up by gradual transitions of gray value. In effect, Seurat serves up an input to the visual system of the viewer, relying on the operation of the viewer’s brain to decipher where the edges are, instead of reporting to the viewer the output of the normal visual operation that reports object boundaries. Seurat’s technique raises the question, How was he able to see the gradual transitions, to know that they could be created? It is unlikely that there is anything different about the essential operation of the visual system of artists versus non-artists (Perdreau & Cavanagh, 2012).

Instead, Seurat probably worked through a process of trial and error, using his visual system as a feedback system to report on the efficacy of his art making. Conventions developed by previous generations of artists were also likely helpful. For example, renaissance artists such as Leonardo Da Vinci developed practices like sfumato that produce almost undetectable gradations of color and shading. As da Vinci wrote, developing an image through gradients “requires much more observation and study . . . than in merely drawing the lines of it” (Leonardo, 1877, p. 91). Da Vinci argued that the painting student could be assisted by knowledge of how light plays across three-dimensional surfaces,

that between the shadows there are other shadows, almost imperceptible, both for darkness and shape; and this is proved by the third proposition, which says, that the surfaces of globular or convex bodies have as great a variety of lights and shadows as the bodies that surround them have. (p. 71)

The case of Seurat is analogous to the example of Egon Schiele: The techniques of both artists depend on an undoing of the normal operation of vision to create a visual puzzle that requires the visual system of a viewer to complete.

Seurat’s deft use of gradual transitions in gray value allows him to deploy maximum local lightness contrast to overcome the essential limitation of a reflective surface. A result is that one region of the image, the book held by the woman, looks like it is glowing, which is obviously physically impossible (a reflective surface does not emit light). The glowing appearance derives from the juxtaposition of the unmarked white paper (the book) with the occluding arm. The arm is the darkest area in the work, and it resides next to the lightest light. The juxtaposition is not a coincidence; rather, it is the darkest dark that creates the lightest light. Moreover, the black apex of the arm covering the book is the climax of an imperceptibly gradual darkening that begins on the right side of the image. These tricks by Seurat work because they exploit the way in which visual information is encoded, and they reflect an implicit undoing of these mechanisms. But they are not sufficient to make the work successful. They are the tools, the medium, that Seurat uses to create the work. The status of the work in the cannon of Western art derives from a list of other factors, such as its intimacy and self-referential nature—a piece of paper (Seurat’s drawing itself), depicting a sheet of paper (the woman’s book)—that advanced the modernist agenda of self-awareness. One might add to this list Seurat’s implicit awareness of how the visual system works and his exploitation of this knowledge in making the drawing.

The Different Uses of Color

There is a deeply engrained prejudice in art school that working in color should commence only following substantial preparation, after years of drawing with charcoal and painting in shades of gray. This prejudice is ascribed either to a privilege afforded color—that color is so important as to require advanced training of the eye—or to the idea that color is inessential, secondary, superfluous (Batchelor, 2000). Da Vinci holds off until Chapter 222 of his Treatise on Painting to touch upon color, lecturing first on the importance of lines, edges, shading, anatomy, physiognomy, and the physics of light and shadow. Da Vinci makes clear his view that color is secondary: “Painting is divided into two principal parts. The first is the figure, that is, the lines which distinguish the forms of bodies, and their component parts. The second is the color contained within those limits (p. 2).” Even in discussing color, da Vinci exposes his prejudice that lightness trumps hue: “According to this order of things, White will be the first, Yellow the second, Green the third, Blue the fourth, Red the fifth, and Black the sixth (p. 89).” In this ranking, colors are listed by their typical relative luminance, not by the importance of their hue. If ranked according to hue, red would be first, after white and black, followed by green and yellow (in either order), then blue (Berlin & Kay, 1969). In his advice to the trainee Henri Matisse, he suggests that “it is only after years of preparation that the young artist should touch color, not color used descriptively, that is, but as a means of personal expression” (as cited in Flam, 1995, p. 121). So both da Vinci and Matisse agree that the beginner should hold off on using color. But for da Vinci, the reason is that color is not a priority, whereas for Matisse, the reason is that color is of utmost priority and consequently poses a great risk.

The discrepant opinions of color implicit in the advice of these two great artists can be understood by tracing how visual signals are processed through the brain. On a coarse scale, the visual cortex cleaves into two main streams: the dorsal visual pathway, which carries visual signals from V1 up over the crest of the brain into parietal cortex, and the ventral visual pathway, which carries visual information from primary visual cortex through a series of brain structures that run on the ventral belly of the brain (Figure 8) (Ungerleider & Mishkin, 1982). Chromatic signals make their way from the retina through visual cortex into both streams, but the way in which the neurons within these two pathways use the chromatic signals is different. Broadly speaking, the dorsal stream responds to color when chromatic information is useful to determine the movement of objects or to make visually guided movements to interact with objects (sometimes the dorsal stream is called the “where” pathway). Under most circumstances, color is not necessary for these tasks: People can determine where objects are, track their movement, navigate a scene, and successfully grasp objects perfectly well without color. For the most part, stimuli must have luminance contrast to effectively elicit responses from neurons in the dorsal stream.

Open in new tab

Figure 8. Two cortical visual systems: dorsal (“where”) and ventral (“what”).

Source: Adapted from Ungerleider and Mishkin (1982).

The ventral stream, on the other hand, uses color to provide information about the identity of objects and about their state (loosely referred to as the “what” pathway). These two aspects of object vision—identity and state—are separable, they use color signals in different ways, and they are computed by largely independent pathways through the ventral visual pathway (Figure 9) (Lafer-Sousa & Conway, 2013; Lafer-Sousa et al., 2016). The object-recognition pathway, of which the face-processing network is a key component, can use color to support parsing object boundaries, to identify where an object boundary is located. This pathway encodes object shape, and it can exploit information about color boundaries irrespective of the colors forming the boundaries, so it is useful in defeating camouflage. Another pathway, the color-biased network, responds to color itself and can provide information about the state of objects—for example, whether a piece of fruit is ripe or a friend is angry. This pathway yields the information commonly thought of as color, that people identify with a color label, and that people associate with subjective meaning.

Open in new tab

Figure 9. The ventral view of the human brain showing the functional organization for faces, color, and places. Portion of the painting Woman With a Hat (1905) by Henri Matisse (left) and the Mona Lisa (1503) by Leonardo da Vinci (right).

Source: Adapted from Lafer-Sousa et al. (2016).

Da Vinci prioritizes visual computations that give rise to object recognition and scene structure. For da Vinci, color is of secondary importance: Color is useful to the extent that it relays information about the shapes of objects and their locations in a scene. Matisse, however, prioritizes the visual computations that yield subjective meaning. For Matisse, color holds primary meaning. Pitting da Vinci against Matisse misses the point: The brain is equipped to use color information in many ways, and the work of these artists highlights the point.

Color Contrast and Color Constancy

Painting in color is difficult. Following standard art-school practice, I would first draw the outlines of the fruit in the still life, and then proceed to paint the image. My impulse, like most students, was to begin by painting the colors of the fruits themselves. But his art teacher would admonish him, “Don’t forget to paint the background.” The problem anticipated by the teacher is that as soon as paint is applied to the background, it changes the color of the painted fruit. The change is wholly unanticipated and unpredictable. The experience is demoralizing: A satisfactory drawing of the scene is quickly ruined. The spatial interaction that determines color is obvious to anyone who has tried to match items of clothing, and it has been documented by many artists, famously by Josef Albers (1963) (Figure 10). The literature is captured by the advice from the author’s spouse: You must hold the tie next to the shirt to see if it matches. You cannot make the tie selection in theory. Instead, the success of the matching task depends on using feedback from your own visual system. As my mentor, Margaret Livingstone, would say, “It’s empirical.”

Open in new tab

Figure 10. (Top) Spatial structure of the cone inputs to a double-opponent cell in macaque monkey primary visual cortex, with a summary schematic. (Middle) Color judgments for the stripes in #thedress reflect assumptions about the chromaticity of the light. (Bottom) Morning Snow Effect (1891) by Claude Monet, oil on canvas, 65.4 × 92.4 cm, Museum of Fine Arts Boston.

Source: Top figure from Conway (2014).

That the color of a mark depends on the colors of the marks around it shows that the visual system encodes chromatic information through spatial opponency. In this regard, mechanisms of color are identical to mechanisms of lightness: Both depend on local contrast. The blackest black is created through juxtaposition with the whitest white, and the reddest red is created through juxtaposition with the most saturated bluish green. But whereas neurons that encode luminance contrast are found in the retina, neurons that encode color contrast are only first encountered in primary visual cortex (see Figure 10). These cells are called double opponent because they show both color opponency and spatial opponency; the example shown in Figure 10 responds best to a small red spot with a green surround (Conway, 2001). Double-opponent cells have relatively large receptive fields compared to center-surround neurons in the retina, reflecting the fact that their computations depend on integrating information from many retinal ganglion cells.

Double-opponent neurons are instructive about how color is encoded. Importantly, these cells do not respond well when the same color signal is applied to both the receptive field center and the receptive field surround. The cell in Figure 10, for example, would be excited when a red spot is presented in the receptive field center, but its response would decrease if red were also presented to the surround. The cell is essentially blind to uniform color fields such as those that accompany changes in the illuminant. Indeed, the spectrum of different light sources can vary dramatically. Daylight has higher power in shorter frequency light, whereas tungsten light has very high power in longer frequencies. Yet under most circumstances, these spectral biases are entirely invisible. Thus, the mechanism of encoding color is adaptive in the same way that the mechanism of encoding lightness is adaptive. Just as there is not much meaning in the overall ambient lightness level, there is little meaning in the overall spectral bias of a scene.

One consequence of the wiring of the visual system is that it achieves color constancy: Colors of objects remain relatively stable despite changes in the illuminant (Hurlbert, 2007). This is adaptive because biological relevance is derived from the stuff in the world. The visual system ensures that colors remain properties of objects, not conditioned upon the viewing conditions. The importance of color constancy is evident when it fails, as shown in the exuberant response to #thedress (Rogers, 2015). The colors of the dress depicted in the viral image are seen as either blue and black or white and gold. The striking difference in how the same image can be seen by two people is accounted for by differences in the assumptions people make about the spectrum of the light (see Figure 10). If you assume the light illuminating the dress has a warm bias, your visual system will discount an orange component from the colors, and you will see them as blue and black. If, however, you assume the light is cool indirect blue sky, you will discount a cool component and will see the colors as white and gold (Lafer-Sousa & Conway, 2017; Lafer-Sousa et al., 2015; Rogers, 2015).

Working in Color

Henri Matisse

Many of Matisse’s finished paintings leave unpainted portions of the underlying white canvas (Figure 11). These white marks pull back the curtain on Matisse’s strategy of working with color and reveal something profound about how color is built up by the brain.

Open in new tab

Figure 11. Interaction of color inspired by a work byJosef Albers (top). White lines surrounding the central “X” regions destroy the interaction (Conway, 2012; bottom).

Although double-opponent cells have larger receptive fields compared to retinal neurons, their receptive fields nonetheless encompass a fairly small patch of the visual field, about a degree of visual angle. The neurophysiological data therefore suggest that color-contrast calculations are made over a relatively fine spatial scale. We can test this prediction using the Josef Albers example shown in Figure 12 (Albers, 1963). Albers’ illustration is appealing because of the clever interplay of foreground color and background color: The X on the left looks to be the same gray as the background on the right and vice versa. But the two X’s are actually painted using the same paint, which you can see because they are connected at the top. The prediction from the neurophysiology data is that the color induction requires that the line forming the X immediately abuts the background color. If this prediction is true, a stable percept of the X should be recovered by obliterating the local interactions with a white buffer, which the bottom panel of Figure 12 shows is the case (Conway, 2012).

Open in new tab

Figure 12. (Left) Interior at Collioure/The Siesta (1905) by Henri Matisse, oil on canvas, 23¼ × 283/8 in. (Right) Plum Blossoms, Green Background (1948) by Henri Matisse, oil on canvas, 45 5/8 × 35 in.

These context relationships are what make painting in color so traumatic for the art student, and they are what make paint-by-number kits so fun. These kits are a visual joke—the color one is instructed to put in each region initially seems wrong, but then as the color of the painting is filled in, as if by magic, the color is induced to be correct.

The importance of chromatic interactions might be new to many readers because most people do not have to think about how color comes about. But challenges posed by working in color must be confronted by painters. Vollard was Paul Cezanne’s dealer and was unhappy with one of Cezanne’s paintings because he thought it was unfinished. There were two small spots of unfinished canvas. Cezanne responded to the critique somewhat sarcastically, “Maybe tomorrow I will be able to find the exact tone to cover up those spots. Don’t you see, Monsieur Vollard, that if I put something there by guesswork, I might have to paint the whole canvas over starting from that point?” The exchange goes on to say that the dealer “shuddered at the thought,” and quickly reclaimed the portrait (Bois, 1993, p. 48).

Matisse was a wealthy man and could afford as much paint as he needed, so it is unlikely the white naked canvas in Matisse’s paintings remained unpainted for lack of paint. Why did he leave these regions unfinished? One possibility is that the white regions act as buffers protecting against color induction. By doing this, Matisse disabled a critical component of the brain’s color-processing mechanism, ensuring that the painted marks do not change color as he completes the rest of the painting. Like Egon Schiele, Matisse discovered something fundamental about how the visual system works by breaking a small piece of it. Beyond a useful trick for painting in color, the strategy deployed by Matisse had a lasting impact on Western art. By revealing the canvas on which the painting was made, Matisse underscored the materiality of the work. Contrast Matisse’s approach with earlier Western traditions, in which a painting was supposed to be of something without reference to its own making—no brush marks, no naked unpainted canvas. Matisse breaks this tradition and makes viewers aware of the fact that the picture is not just an illusion of something, instead underscoring the fact that the picture is paint and canvas. In the cannon of Western art, this is partly what makes Matisse so radical. And at the time, his work, like that of Monet, was ridiculed by critics. That works by these artists are now celebrated further underscores the ability of the visual system to adapt, not simply over timescales of short duration, over a day, but longer durations, perhaps across generations.

Chuck Close

The strategies for working in color exemplified by Monet and Matisse showcase the importance of spatial context in determining what color a given painted mark appears. The painting process adopted by Chuck Close illustrates a radical alternative, “aperture viewing,” which is not unlike paint-by-number (Figure 13). The process obviates entirely the normal work of the visual system to integrate across a visual scene to generate color. Rather, it depends on color matching isolated regions in a reference image. The reference image is masked with white except for the small region to be reproduced; the masking abolishes color-contrast interactions, allowing the artist to reproduce in paint the chromaticity of the isolated region. The creative work in the process arises in how the isolated regions are painted. Close often uses swirls, Gabor-like elements, or concentric rings, which are unrelated to the original image except for their average chromaticity. Close’s work is successful because he decouples the information normally obtained from local spatial cues from the global information derived from variations in color and luminance. This success arguably depends on the segregated pathways through the visual-processing hierarchy in the brain, whereby retinal information is interpreted by different parallel pathways to yield different kinds of information about the scene (see Figures 8 and 9).

Open in new tab

Figure 13. Chuck Close working in studio.

Exploiting the Neural Hardware

In 1874, a group of artists who felt shunned by the academic painters of their day convened an exhibition in the studio of the photographer Paul Nadar. The show of work by Monet, Degas, Renoir, Cezanne, Guillaumin, Pissarro, Sisley, and Morisot was reviewed in a satirical essay by Louis Leroy. The essay was given the derogatory title “The Exhibition of the Impressionists,” borrowing from the title of Monet’s painting Impression Sunrise (Figure 14). With his essay, Leroy unwittingly gave name to one of the most successful artistic movements in western history. Monet’s painting is peculiar: The sun is about the same luminance as the surrounding gray sky. This is, of course, a physical impossibility because the sun will always be the brightest thing in a scene. Margaret Livingstone (2002) has argued that Monet intentionally used red paint that is equiluminant with the gray because doing so produces an unstable visual quality. The sun appears to shimmer or vibrate. Chromatic signals are handled differently by dorsal versus ventral visual pathways, and these pathways tell us different things about the world. Many dorsal-stream neurons may respond to color boundaries, but the cells are not usually tuned for color. Instead, the neurons respond most vigorously to differences in luminance contrast. One consequence is that the cells are relatively blind to stimuli defined entirely by color contrast, such as Monet’s sun. Livingstone argues that by rendering the sun with equiluminant paint, Monet makes the sun invisible to the dorsal stream. The idea is that Monet essentially impairs the normal function of the dorsal stream—to indicate where things are in visual space and to guide interactions with objects in the world—leading to an unstable percept of the spatial structure in the scene. The result is an “impression” that is more meaningful than could be captured by rendering veridically the luminance profile of the scene.

Open in new tab

Figure 14. Impression Sunrise (1874) by Claude Monet, 19 × 25 in., in color (left) and grayscale (right).

For Monet, the color of the paint in the sun is secondary to its luminance: Any color that is equiluminant with the gray sky would produce the same shimmering effect. Matisse, however, leverages the same feature of visual information processing but toward a different end. Matisse seeks to ensure the spatial structure of the scene is intact by using paint with the appropriate luminance-contrast relationships. But he does so by using colors that are divorced from those people would normally see (Figure 15). Through his work, Matisse discovered that the information content associated with luminance contrast can be liberated from the information content associated with hue—and this discovery reveals fundamental aspects of how the brain processes visual information that can be related to the dorsal and ventral streams: The dorsal stream is sensitive to luminance-contrast relationships and uses that information to construct the spatial relationships in a scene; the ventral stream is sensitive to hue and uses color to confer subjective meaning on what is seen.

Open in new tab

Figure 15. Woman With a Hat (1905) by Henri Matisse, 31¾ × 23½ in., in color (left) and grayscale (right).

Spatial Structure: Artist’s Physics

The composition of a picture is a central feature of many artworks. An analysis of how artists organize the spatial layout of their images can provide clues to how the brain constructs a sense of scene structure. As with many aspects of art production, composition is influenced by conventions that have roots in culture and may not be entirely determined by constraints imposed by the brain. One convention is linear perspective, which is a set of rules devised by Filippo Brunelleschi (1377–1446) and Battista Alberti (1404–1472) that codifies how to create an illusion of depth by having parallel lines converge on a single vanishing point. Curiously, there is no natural circumstance under which humans experience linear perspective. Each eye will have its own vanishing point, and the eyes and head are in almost constant motion, so the vanishing point of each eye is almost constantly changing. Yet linear perspective is successful, suggesting that it reflects something about the way the brain constructs a representation of scene structure invariant to head/body motion within the scene. Is a single vanishing point necessary in images that exploit perspective? No, as illustrated by the Cezanne painting shown in Figure 16. The aperture of the blue jug is rendered as if viewed from above, whereas the base of the same jug is depicted as if viewed from the side. Other objects in the scene are similarly painted as if amalgamated from multiple perspectives. The shifting vanishing points give the work a dynamic quality, putting the viewer in the scene actively surveying the objects, and capturing the aspects of the objects that are relevant for each viewing angle—the opening of the jug is round, reflecting people’s experience interacting with the jug to pour fluid out of it, whereas the base is a flat horizontal line, reflecting the fact that the jug sits, stably, on a table. Artworks such as this provide hypotheses for how spatial representations of scenes are represented within the parts of the brain implicated in an analysis of places, such as the place-processing stream of the ventral visual pathway (which includes the parahippocampal place area), and areas of the dorsal stream.

Open in new tab

Figure 16. Still Life With Apples (1893–1894) by Paul Cézanne, oil on canvas, 65.4 × 81.6 cm.

Stephen’s drawing, shown in Figure 16, underscores the conclusion that the brain does not encode the physical pattern of light cast on the retina but, rather, the way in which that pattern of light is behaviorally relevant. Patrick Cavanagh (2005) takes up this idea—the artist’s physics—in an analysis of shadows, noting that the pattern of shadows represented in a scene need not conform to any plausible physical circumstance. The shadows in Fra Carvenale’s painting from 1467 do not immediately strike the viewer as bizarre (Figure 17). But where is the light source casting the shadows on the ceiling of the interior? And how could the same light source cast the shadows of the figures in the foreground? The painting depicts a physical impossibility. And it reveals that the representation of a physical reality is not the objective of the visual-cognitive apparatus. One hypothesis is that there are not strong priors about the direction of light sources, because the sun is constantly shifting, so the brain does not exploit information about the direction of shadows to encode scene structure. Rather, shadows are defined by their local relationship to the objects casting them, a feature that is relatively consistent across lighting conditions and therefore a useful cue to scene structure.

Open in new tab

Figure 17. The Birth of the Virgin (1467) by Fra Carnevale, tempera and oil on wood.

Faces

Burrowing deep into extrastriate cortex, one finds extensive cortical resources allocated to the computation of faces, presumably reflecting the importance of social interactions to primates. But once again, the way that artists represent faces indicates that the brain does not encode facial identity through a veridical representation of the pattern of light cast on the retina when one looks at a face. This is perhaps most clear in the intimate portraits by Picasso of his close friends and lovers (Figure 18). His pictures would not readily be characterized as photorealistic, and yet each one is immediately recognizable. Great portraits such as these provide a treasure trove of data about what aspects of a face make it distinct, and they could be used to generate hypotheses about how the brain learns familiar faces (Conway & Livingstone, 2007).

Open in new tab

Figure 18. Portraits by Pablo Picasso (right) and photographs of each subject (left): From top: Jacqueline Picasso (1957), Jacqueline Picasso (1955), Marie-Thérèse Walter (1936), Marie Françoise Gilot (1946), Emilie Marguerite Walter (1939), Ambroise Vollard (1910), and Wilhlm Uhde (1910).

Fakes

This article is motivated by the conviction that an analysis of art could be productive for understanding how the brain works and, vice versa, that knowledge of visual cognition could be insightful in appreciating art. But headway in this enterprise will require that neuroscientists and art historians work together to establish a meaningful basis for points of contact. Such collaborations demand not only an openness on the part of all participants but also close attention to conceptual clarity—the challenge of cross-disciplinary accounts of “beauty” bring these issues to the fore (Skov & Nadal, 2021). Nonetheless, there are many opportunities for such interactions. As I’ve argued, most neuroscience of art has dealt with the perception of finished works of art, yet a consideration of art practice, rather than the products of that practice, has the potential to reveal an enormous amount about creativity and about what is and becomes valued in a society.

The final figure in this article (Figure 19) shows a pen-and-ink drawing from the 1940s signed “Henri Matisse.” The work is a forgery made by the Hungarian artist Elmyr de Hory. That this is a fake is obvious to most experts today. The drawing was bought by the curator of the Fogg Art Museum at Harvard University, Agnes Mongan, not long after the drawing was made. Mongan and all her advisors, including many acclaimed art historians, thought at the time that the drawing was genuine. De Hory’s drawing was of a beautiful young woman. Notions of beauty morph over time, evidence against the idea of universal concepts of beauty. In the 1940s, when the drawing was made, the accepted notion of beauty must have been implicitly informed by the actresses of the day and photographs of them. The forger, like the viewers he was attempting to dupe, was steeped in this implicit notion of beauty and did a lovely job capturing this zeitgeist. Compare the drawing to the photographs of three contemporary actresses. From the objectivity granted through time, it is clear that these photographs, and the beauty they capture, are characteristic of the 1940s. In the 1940s, they would not have been qualified by this context—they would have been deemed “beautiful.” Real Matisse drawings from approximately the same time, although superficially similar to the fake, look nothing like the young actresses, and they show a more nuanced and astute rendering of space.

Open in new tab

Figure 19. (Top) Imitation of Henri Matisse, Woman With Flowers and Pomegranates (1944) by Elmyr de Hory, ink on paper, 39.7 × 51.9 cm. (Bottom) Photographs of actors from the same time as the drawing by de Hory. Left to right: Ava Gardner (no date), Lauren Bacall (1938), Hedy Lamarr (1944).

Forgeries are almost always only successful in deceiving contemporary viewers. Twenty years after the drawing was made, Agnes Mongan and her colleagues easily concluded that the drawing looked nothing like a Matisse drawing, but rather like the idea of a Matisse in the minds of a 1940s viewer. An analysis of the limits of perception and cognition revealed by neuroscience could help us understand the shifts in our evaluation of the work. It would be useful to learn from art historians what attributes of the work made it so readily acceptable in its day and what features call it out so clearly as a fake today. What do these aspects of the work reveal about the visual features that drive probabilistic models of vision, cognition, and visual preference? Could such an analysis shed light on neural mechanisms that subserve a collective aesthetic sense? The example shows how both neuroscience and art history have much to gain from taking up a common problem. For neuroscience, this example shows how an analysis can extend beyond treatments of artworks as fixed, static, time-detached objects and may begin to touch upon some of the central concerns of art history—why certain ideals are promulgated at certain times (Cavanagh et al., 2013). Ultimately, the shifting conceptions of beauty, how these are represented and reappraised, undoes the reflexive idea of a universal conception of beauty and redirect attention to the phenomenal power of the visual brain to adapt to changing demands while consistently leaving the impression that the eyes are reporting something universal and veridical about the physical world.