Introduction to Spatial Vision for Action

Spatial vision refers to extracting information about the layout of surroundings from the light entering the eyes. This information consists of a wide variety of spatial attributes, such as where surfaces are; their sizes, shapes, and orientations; and any changes in their positions, either relative to the observer or to other surfaces. It is concerned with spatial attributes as opposed to attributes such as the surfaces’ reflective properties, which are primarily relevant for identifying objects or judging the material of which they are made.

Most attributes cannot be extracted directly from the image on the retina. For instance, an object’s size cannot be determined from the retinal size of its image alone because the retinal image size also depends on the egocentric distance to the object or its relative distance with respect to other objects of known size. Judging egocentric distance in different ways could lead to different judgments of object size. Another attribute that can be judged in various ways is surface slant. Slant can be judged from binocular disparities, but if the surface is textured (either because there is some structure in the reflective properties of its reflectance or because it is not completely smooth) gradients in the pattern of its retinal image can help to extract information about the surface’s slant relative to the line of sight. In doing so, the implicit assumption is made that the true surface texture is uniform in some respect (see Figure 1A; Todd & Oomes, 2002). Similarly, a surface’s orientation at a given point can be extracted from the luminance at the corresponding point on the retina, but doing so involves making implicit assumptions about the surface’s reflectance and the illumination (Khang et al., 2007; Todd et al., 2014). Such assumptions are made automatically, so people rely on information based on such assumptions in their actions. For instance, they rely on texture isotropy when they interact with a slanted, textured surface (Knill, 2005).

Open in new tab

Figure 1. How implicit assumptions influence spatial judgments. (A) The items are assumed to be identical and symmetrical, so the textured surface created by these gray disks is perceived as slanted. (B) The gray items that share a border with the black square are perceived as partially occluded disks. (C) Even if one of the disks is completely occluded one is likely to infer its presence, probably because it would be quite coincidental for precisely the disk that is completely hidden by the square to be missing. (D) This is even so when the square itself is inferred from occlusion (Kanizsa, 1976; van Lier, 1999). (E) The presence of partially occluded disks may not even be essential. (F) However, there must be some reason to believe that there might be an occluded disk.

The most direct spatial information about visually perceived structures is their egocentric direction. Each retinal image provides an ordered representation of directions with respect to the eye, which provides direct information about structures’ relative directions with respect to the observer’s eye. There is generally only one visible item in any direction (ignoring areas that are visible to one eye but not the other) because most surfaces occlude everything that is behind them. Therefore, people often have to infer properties of items that are partially or even completely hidden, and they do so quite readily (see Figure 1B–E). The capacity to infer properties of hidden surfaces is essential in the control of actions. When grasping objects, people do not hesitate to place their digits at positions on surfaces that are hidden from view (Voudouris et al., 2012a). They use memory of where things were to guide the movements of their eyes (Aivar et al., 2005) and hand (Brouwer & Knill, 2007, 2009) if doing so is opportune.

Irrespective of whether a target position is visible, inferred, or remembered, in order to guide one’s finger to that position the orientation of the eyes in the head and all the angles of the joints connecting the finger to the head must be considered. In some cases, it might seem advantageous to circumvent some of this complexity by not using egocentric positions, relying directly on the position of the finger relative to the target instead. This alternative attribute (relative position) is directly available from the retinal image. It might be especially advantageous to use the relative position when moving a cursor across a screen with a mouse, because there is no direct way to relate the position of the cursor on the screen to that of the hand holding the mouse through proprioception. However, studies in which a cursor had to quickly be moved to a target show that the cursor and target are localized independently rather than relative to each other (Brenner & Smeets, 2003; Crowe et al., 2021; Franklin et al., 2016).

Judging the distance to a surface is more complicated than judging directions. The most direct source of information about egocentric distance is the difference in direction with respect to the two eyes. However, the modest distance between the eyes limits how precisely distance can be judged in this manner, because for a modest distance between the eyes the difference between the orientations of the two eyes when fixating an object and the differences between the images in the two eyes are small. This is presumably why many other sources of information about egocentric distance are also considered (for reviews of depth cues and their resolutions at different distances, see Brenner & Smeets, 2018b; Cutting & Vishton, 1995). When one moves, the change in a static structure’s direction relative to oneself can be combined with knowledge about how one has moved to judge the structure’s distance (de la Malla et al., 2016). When an object is on a horizontal surface, the height in the visual field of the position at which the object touches the surface can be combined with knowledge about how far the surface is below one’s eyes to judge the object’s distance (Dixon et al., 2000). When one has some idea of an object’s size, its retinal image size can be combined with such knowledge to judge its distance (McIntosh & Lashley, 2008; Sousa et al., 2012). In line with the findings on direction in the previous paragraph, studies in which a cursor had to quickly be moved to a target in depth also suggest that the cursor and target are localized independently rather than relative to each other (Brenner & Smeets, 2006).

If egocentric directions and distances guide human actions, is the position of a target relative to other items in the environment completely irrelevant for guiding actions? There is some evidence that changing the positions of relevant items in a scene can influence actions to a remembered target position, but only if the other items are relevant and one does not notice that they have been moved (Fiehler et al., 2014; Lu & Fiehler, 2020). One way to interpret this is that the movement is still directed toward a remembered egocentric position and that moving relevant reference items influences where one remembers this position to have been.

The distinction between positions relative to oneself and positions relative to other objects is critical for the interpretation of the anatomical segregation of the primate cortex into two visual streams. The idea that ongoing actions are normally not guided by relative positions is fundamental to the interpretation of the classical distinction between a what and a where pathway (Haxby et al., 1991; Mishkin et al., 1983) as a distinction between pathways dedicated to extracting information that is constant (i.e., object identity) and information that is constantly changing (such as positions relative to oneself). In addition to nonspatial properties, such as surface reflectance, object identity includes spatial properties, such as the object’s size and shape (including the relative positions of the object’s components). The idea of segregating different kinds of spatial information is consistent with evidence that the processing of the instantaneous egocentric position and motion of relevant objects relative to the observer takes place in the dorsal visual cortical pathway, while most processing of attributes that are relevant for judging the objects’ identities takes place in the ventral visual pathway (Goodale et al., 2004; Milner & Goodale, 2008; Schenk, 2006).

Some have been tempted to take this distinction one step further, proposing that there is a fundamental distinction between judgments that are used to guide actions and those that are used to recognize objects (Milner & Goodale, 2008; Pisella et al., 2000). The latter distinction is often referred to as a distinction between action and perception (Goodale, 2011; Goodale & Milner, 1992). This step is not just a change in terminology. Attributes such as size and shape do not constantly change. Neither do the materials of which an object is made (and therefore its likely weight distribution). Nevertheless, an object’s size and shape, the materials of which it is made, and even knowing how the object is used if one recognizes it can all influence how an object is grasped (Feix et al., 2014; Klein et al., 2020; Paulun et al., 2016). One might argue that deciding where to place one’s fingers on an object in order to pick it up is not part of the action (e.g., because such positions are likely to have been chosen before the movement started), but there is evidence that people can adjust their choice of grasping points during an ongoing movement (van de Kamp et al., 2009; Voudouris et al., 2013) just as they can adjust the choice of a target while already moving (Brenner & Smeets, 2015; Resulaj et al., 2009). Thus, although the dorsal visual pathway presumably processes the information that is most extensively used to guide movements, a fundamental segregation between visual processing for perception and for action, where at least some spatial attributes are processed differently for perception than for action, is not very likely. Moreover, contrary to the idea that the dorsal visual pathway is immune to contextual information, experimental evidence has shown that illusions do influence the dorsal stream (Walter & Dassonville, 2008; Weidner & Fink, 2007) and thereby influence actions (de Brouwer et al., 2015; de la Malla et al., 2019; Somers et al., 2000). Thus, the anatomical segregation between the two visual pathways probably reflects a distinction between attributes that are constantly changing because they rely on spatial positions relative to oneself, and must presumably be combined with extra-retinal information, and attributes that rely on more persistent properties and relationships.

Ongoing movements may be guided by egocentric positions, orientations, and motion, but how about the role of spatial vision in the initiation of actions? How do people decide whether to try to jump across a stream or to walk through a narrow gap? An individual probably knows from experience how far they can easily jump, and therefore whether the stream is easily jumpable, possibly jumpable, or impossible to jump across (such understanding of one’s ability with respect to items in the environment is often referred to as perceiving an affordance; Gibson, 1979). Similarly, if a doorway is clearly wide enough to pass through, a person will just walk through it. If it is just passable they will rotate their shoulders to walk through. If it is clearly too narrow to pass through, they will usually not try (Fath & Fajen, 2011; Warren & Whang, 1987).

Judgments about action capabilities have also been studied by asking participants to indicate whether they could step onto (Warren, 1984), jump or step over (Day et al., 2015), pass through (Warren & Whang, 1987), or extend their arm to reach (Carello et al., 1989) something. It is easy to imagine how the ability to make such judgments could arise from experience, and such judgments are flexible in the sense that they change if the situation is changed, for instance by giving people a tool with which to reach for the object (Bourgeois et al., 2014). Since it is a bit trivial that one can reach further positions with a stick than without, attempts have been made to show that judgments about reachability actually influence judged distance and not just judgments about whether something is reachable. Some studies have found that it does (Bourgeois et al., 2014; Witt et al., 2005), while others have not (de Grave et al., 2011).

Returning to visually guiding ongoing movements, people cannot avoid adjusting their movement to a target perturbation when they know they should stop moving if the target is perturbed (Pisella et al., 2000). Similarly, they cannot avoid adjusting their grip to changes in grasping points’ positions before switching to new grasping points when an object that they are reaching out for is rotated (Voudouris et al., 2013). This is because it takes less time to adjust the ongoing movement to visual information about the original endpoint than to reconsider the endpoint or the movement altogether (Smeets et al., 2016). This is consistent with visual information about the egocentric positions of selected points on surfaces being processed in brain areas that are close to and well connected with motor areas (e.g., the dorsal cortical pathway, and possibly subcortical pathways involving areas such as the superior colliculus; Reynolds & Day, 2012).

Vision Is Very Selective

As mentioned previously, it is important to be aware of two common, compelling, but incorrect assumptions about spatial vision. The first assumption is that people constantly register everything that is within their field of view. Being aware that this is not true is an important step toward treating vision as an active process rather than as a passive accumulation of information. There is obviously some accumulation of registered information about some aspects of what we see (often referred to as memory), but this is only a tiny portion of what was visible at the time. One cannot simply recall a scene an hour later to recover details that one had initially missed. Contrary to our subjective experience, we do not register everything that is within our field of view; we only process what is directly relevant to the task at hand (Triesch et al., 2003), as will be explained in more detail in the rest of this section. This does not mean that we can always completely ignore everything else. It is evidently important not to ignore things that could warn us of dangerous situations. Consequently, it is almost impossible to ignore sudden changes of potential relevance (Mulckhuyse et al., 2008; Schreij et al., 2008). However, the visual processing that we perform is usually quite well tuned to the task.

Our ability to register precisely what we need is probably why so many things go by unnoticed. A very simple example is that we do not notice that our spatial resolution is quite poor except near where we are looking. The poor spatial resolution is simple to verify by trying to read this text while intentionally looking to the side. People normally do not notice how steeply their spatial resolution decreases with eccentricity; if they want to identify a structure they quickly direct their gaze toward it. A phenomenon known as change blindness, whereby people fail to detect obvious changes in a scene, also illustrates the selectivity of visual processing. If a single object suddenly changes, people immediately see the change (Movie 1); but if the change occurs while the image is moving across the retina (Schofield et al., 2006; Movie 2), as might happen during a saccade (Henderson & Hollingworth, 2003); is accompanied by other global (Blackmore et al., 1995; Movie 3) or even local (O’Regan et al., 1999; Movie 4) changes; or occurs very gradually rather than abruptly (Simons et al., 2000), it can take quite a long time to find the change. This is the case even though the task is to detect the change. Thus, the only reason that an isolated change is detected so easily is because the change itself reveals where the relevant information is to be found. Masking detection of the change by also having changes occur elsewhere removes this and means one has to actually search for the changing item.

Even extremely conspicuous events, such as a gorilla walking through a scene, can go by unnoticed if the observer is performing an unrelated task at the time, such as counting the number of times a ball is thrown (Simons & Chabris, 1999). Moreover, even changing a task-relevant property of an object that one is manipulating can go by unnoticed if the change does not take place at a moment at which that property is critical. This was shown in an experiment in which virtual objects had to be sorted by size by placing them on one of two virtual conveyer belts (Triesch et al., 2003). Participants were instructed to first move the large objects to one belt and then the small objects to another. The experiment was designed in such a way that participants looked at the object while reaching out to pick it up and then shifted their gaze to the conveyor belt to guide the placing movement. Near the time of the shift in gaze the object sometimes changed size. Participants often did not notice that the size had changed because the change took place during the saccade from the object to the conveyor belt. Interestingly, they placed the object on the wrong belt for its final size despite the fact that they were looking at the object again when placing it on the conveyor belt. Presumably they did not notice the new size because size determined which object was to be picked up, so the size had already been judged when reaching for the object. As size is a persistent property of objects, there was no reason to judge it again when executing the placing movement a fraction of a second later.

This interpretation is supported by a similar study in which the information that was relevant for placing the object could differ from the information that was relevant for picking it up; each could depend on the object’s height, width, color or texture (Droll & Hayhoe, 2007). Again, the most important trials were occasional ones in which the object’s appearance changed. On more than half of such trials the participant did not notice the change. When participants knew that the same information was relevant for picking up and placing the object, as in the former study, they mostly placed the object on the conveyor belt that corresponded with the object’s original appearance. When information about where to place the object was only provided once they had picked up the object, they mostly placed the object on the conveyor belt that corresponded with the object’s final appearance. This combination of findings nicely illustrates how details of the task determine not only what attributes are processed but also when they are processed.

The reason people believe that they see everything within their field of view despite not assessing many attributes is presumably that at any moment they can gain access to information about anything they are interested in, so they never notice missing any information (O’Regan & Noë, 2001). Actually, the whole idea that one could notice missing information is strange, because no information is really missing; the presence of the information is just not being processed.

Many of the limitations described here are about our conscious experience of what we see. Processing information to guide ongoing movements may not be limited in quite the same way. It has long been known that people adjust their goal-directed movements in response to changes in target locations that they do not notice (Goodale et al., 1986). In fact, people are generally not aware of continuously adjusting their ongoing movements on the basis of ever-changing judgments of egocentric positions (Brenner & Smeets, 2018a). It is therefore worthwhile to study the role of spatial vision in guiding ongoing movements and the conscious perception of space separately. Even if all the available information is always processed, and the reason that information is missed in change-blindness tasks is that only a selection reaches conscious awareness, it is likely that a similar selection from the available information guides each aspect of people’s actions.

Vision Is Not Always Consistent

The second common, compelling, but incorrect assumption about spatial vision is that related spatial attributes are judged in a manner that is consistent with the laws of physics. One generally has the impression of seeing a coherent world, in which an approaching car changes its position over time in accordance with its speed. However, the consistency between physically equivalent attributes is not always found for judgments of those attributes. This is because related attributes are judged separately, each considering components of the available information in relation to how relevant they are for correctly judging that attribute. For instance, the rate at which an approaching car’s retinal image is expanding provides useful information for assessing the car’s speed, but it is not very informative about its instantaneous position. And indeed, variations in the rate of retinal image expansion can influence judgments of a target’s velocity more than they do judgments of its position at corresponding moments (Brenner et al., 1996).

Systematic errors in judging the value of a specific attribute could arise from considering information that is specific to that attribute and is based on assumptions that are violated. For instance, using perspective cues to judge a surface’s slant is based on assuming that shapes are symmetrical or texture is isotropic, so one will misjudge the slant if these assumptions are violated. The extent to which the slant is misjudged will depend on how such incorrect information from perspective is combined with correct information from other cues. With no additional knowledge about the true value of the attribute in question, minimizing the uncertainty of the combined estimate is the best one can do. In general, estimates from conflicting cues are indeed combined in ways that maximize the overall precision (Ernst & Banks, 2002; Fetsch et al., 2010; Hillis et al., 2004; Knill & Saunders, 2003; Muller et al., 2009; Scarfe & Hibbard, 2011). However, if feedback reveals that combining the cues in this manner gives rise to judgments that are incorrect, despite being precise, the accuracy is also considered (i.e., cues providing information that is inconsistent with the feedback are subsequently given less weight; Cesanek et al., 2020; van Beers et al., 2011). With some cues only contributing to certain judgments or being given different weights for some judgments than for others, one should not be surprised to occasionally encounter situations in which judgments about related attributes are not consistent with each other (Brenner & van Damme, 1999).

Another reason errors might not always be consistent across attributes is that different attributes may make different use of information within different reference frames. Two identical parallel lines placed side by side (Figure 2A) appear to have about the same length and orientation, and the tops and bottoms of the two lines appear to be aligned vertically. This is a consistent set of judgments. When these same lines are superimposed on an image of a scene (Figure 2B), they no longer appear to have the same orientation or length, possibly because it is impossible to completely ignore the depth portrayed in the scene when judging their lengths and orientations (Gregory, 1963). However, if one decides to check whether the line on the left is really longer than the one on the right by checking the vertical alignment of the two lines’ endpoints, one will discover that the top and bottom of the two lines appear to be aligned vertically, which is inconsistent with the line on the left appearing to be both less slanted and longer. In accordance with a lack of consistency across judged attributes, attempts to define a single coherent visual space (e.g., Cuijpers et al., 2000) have often failed. If there is no guarantee of consistency across attributes, such a space cannot exist (Smeets et al., 2009).

Open in new tab

Figure 2. How reference frames can influence spatial judgments. (A) Two identical slanted lines placed next to each other appear to have about the same length and orientation. That is not surprising because they are equally long and parallel. (B) Superimposing the same two lines on the image of a scene can change this. The line on the right now looks shorter and slanted further from vertical, probably because one cannot ignore the reference frame provided by the scene. When considered as part of the scene, the line on the left would really be longer and less slanted because it appears to be on the road so that it recedes in depth, whereas the line on the right appears to be parallel to the wall so that it does not recede in depth and is altogether nearer the observer.

Illusions provide a useful tool to study how spatial vision guides action because illusions sometimes influence certain attributes while leaving others unaffected, or at least not influenced to the same extent. For instance, variants of the Müller-Lyer illusion influence judged line length without having a similar influence on the judged positions of the lines’ endpoints (Gillam & Chambers, 1985; Smeets et al., 2002b). Another example arises after adapting to motion for some time: a static scene appears to move but does not appear to change position accordingly (Anstis et al., 1998). Specially designed stimuli can also give the impression of continuous motion without a corresponding change in position (Mather, 2006; Tse & Hsieh, 2006).

Despite the abundance of examples of inconsistencies between attributes in spatial vision, and the evidence that visual attributes are processed separately in distinct brain areas (e.g., Fattori et al., 2009; Grill-Spector & Malach, 2004), the myth of consistency perseveres. It remains surprising for the judged distance between two positions to not correspond with the distance between their judged positions (Smeets & Brenner, 2019) or for an object to appear to move faster than another but not to move farther in the same amount of time (Brenner et al., 1996; Smeets & Brenner, 1995; see Movie 5). A likely reason for the surprise is that we normally do not compare attributes in this manner (or try to combine them to construct a coherent global representation of the scene), but rather judge all the attributes that are relevant to the demands of the current task at the moment that we need them. Judging each attribute as precisely as possible will normally result in judgments that are consistent enough for whatever one is doing simply because the physical basis is consistent.

If one embraces the lack of consistency between judgments about related attributes, one can manipulate images to primarily influence a specific judgment and use the resulting lack of consistency to evaluate what visual information is used to guide our actions, and maybe even how it is used. Unfortunately, despite the abundant evidence that judgments need not be consistent, many researchers still implicitly assume consistency when interpreting their data, leading to incorrect conclusions and, more importantly, to missed opportunities. Examples involving assumptions of consistency between judged positions and the judged distance between them and between judged motion and judged change in position are provided in the ‘Size and Grasping’ and ‘Interacting with moving objects’ sections, after discussing several other issues.

Differences Between Seen and Felt Space

One might worry that errors, biases, and inconsistencies in spatial vision not only make it difficult to characterize the geometrical properties of spatial vision (Cuijpers et al., 2000, 2002; Todd et al., 2001) but also make it impossible for spatial vision to reliably guide actions. However, to guide actions there is no need for geometrical consistency, as long as the brain can convert visual judgments about spatial attributes into appropriate motor commands (Braddick & Atkinson, 2013; Georgopoulos, 1994; Körding & Wolpert, 2006). Errors in judging a relevant attribute will influence the way a movement is made, but it need not prevent one from reaching one’s goal because actions are normally continuously controlled using the latest visual information about both the target and the moving limb (Brenner & Smeets, 2018a). Thus, for instance, systematically misjudging the direction to a target by 5° will make one start to move in the wrong direction, but even if the direction continues to be misjudged throughout the movement, adjusting the trajectory will ensure that one reaches the target, although the trajectory will be curved (Smeets & Brenner, 2004).

Having decided in which direction to move, one has to activate the correct muscles. Which motor commands will do the job depends on the position of the limbs (Sober & Sabes, 2003). Proprioception is therefore essential for this decision (Gordon et al., 1995). Since the movements of the limbs are also continuously controlled on the basis of proprioceptive feedback (Manning et al., 2012; Sittig et al., 1987), as is evident from fast adjustments to sudden perturbations of the unseen hand (Kurtzer et al., 2014; Marsden et al., 1977), felt position must be important for the relationship between spatial vision and motor control.

One might expect that one’s extensive experience with one’s own limbs would keep spatial vision aligned with one’s notion about the locations of parts of one’s body, and in particular of one’s hand (Henriques & Cressman, 2012; Shadmehr et al., 2010). However, it has been shown that there are systematic idiosyncratic differences between seen and felt space (Kuling et al., 2016; Rincon-Gonzalez et al., 2011; Smeets et al., 2006; van Beers et al., 1998). The direction and extent of this visuo–proprioceptive mismatch depend on the specifics of the task (Kuling et al., 2017), which might give the impression that the mismatch is a random artefact. However, the idiosyncratic mismatch is systematic: It does not change when the arm is held still for many minutes (Rana et al., 2020) and is even reproduced when the same task is repeated a month later (Kuling et al., 2016).

Normally, the large and persistent idiosyncratic mismatches between vision and proprioception are not a problem because we see our hand approach objects that we want to manipulate. Under such circumstances, seen and felt space appear to be attuned (see Box 1 for an explanation of the way we use the term attuned). This is presumably not because we use the relative positions of the hand and the target to control our movements; even when guiding a cursor to a target on a screen, relative positions appear to make a very small contribution (see “Introduction to Spatial Vision for Action”). It is probably because people use visual feedback to attune their movements to compensate for idiosyncratic errors. This attuning is similar to the way in which people adapt to experimentally imposed errors (van der Kooij et al., 2013), which has been studied extensively using the prism adaptation paradigm (Redding & Wallace, 2002). Such attuning is discussed in more detail in the next section. Importantly, when the hand is hidden from view and no feedback is provided about the relationship between seen and felt space, the attuning gradually disappears and the idiosyncratic matching errors reappear (Smeets et al., 2006).

Maybe the ability to quickly and temporarily attune vision with proprioception is what makes people so proficient at using tools; it makes it possible to consider the tool as an extension of the body (Maravita & Iriki, 2004). One might have ways of feeling the position of the tip of a tool when moving a physical object like a fork (Turvey & Carello, 2011), but people can also proficiently control the movement of a cursor on a computer screen as if it were part of their body (Brenner & Smeets, 2003) as long as certain relationships between directions are maintained (Brenner et al., 2020). Being able to feel the positions of relevant parts of tools would explain why people generally look at the target (or at obstacles; Johansson et al., 2001) rather than at their hand or even a cursor that they are moving to the target by moving their hand. However, although they seldom direct their gaze at their hand or the cursor, they do use visual information to guide the hand or cursor to the target, relying on information from peripheral vision to do so (Cámara et al., 2018). The lower resolution of peripheral vision is apparently not a problem (Cámara et al., 2020), which is in line with other observations that goal-directed movements do not necessarily require a high spatial resolution to be controlled precisely (Mann et al., 2007).

Attuning Behavior to Sensory Inconsistencies

It has long been known that people can attune their behavior to experimentally imposed changes in the relationship between the images on their retinas and the felt position of their body, such as occurs when people are forced to look at the world through prisms (Harris, 1965; Hay & Pick, 1966). Attuning to some unusual relationships is easier than attuning to others (Lillicrap et al., 2013). This is probably because some have logical anatomical interpretations (van den Dobbelsteen et al., 2003), but many details are not yet fully understood. The previously mentioned lack of permanent recalibration (Smeets et al., 2006) does not mean that exposure to a new relationship never has a long-lasting effect because relearning an unusual relationship sometimes progresses faster than the initial learning (Criscimagna-Hemminger & Shadmehr, 2008; Kitago et al., 2013; van der Kooij et al., 2016; Vaswani et al., 2015). However, even people who regularly switch between glasses and contact lenses (that give rise to different degrees of image magnification) do not immediately attune their behavior when they switch between the two (Schot et al., 2012).

Attuning one’s behavior to the circumstances occurs in many different ways. It can be intentional or automatic (Maresch et al., 2021). It can be a response to perceived spatial errors with respect to the target (van Beers, 2009) or with respect to the planned movement (Tseng et al., 2007). It can even be based on success rather than on perceived spatial errors (Kuling et al., 2019; Leow et al., 2018; Mehler et al., 2017; van der Kooij & Smeets, 2019). Changes across subsequent movements are often smaller if one can see the whole movement than if one can only see the outcome (Bernier et al., 2005), presumably because such changes are based on errors in the outcome of the movement and continuous vision of the hand reduces such errors. The many ways to attune behavior suggest that there are multiple mechanisms involved, probably each with its own learning rate and rate of decay, so as time passes performance may even sometimes shift to be governed by different mechanisms without an evident change in performance (McDougle et al., 2015; Smith et al., 2006).

Visual localization is unreliable near the time of saccades, partly because people cannot reliably synchronize retinal information with eye orientations (Dassonville et al., 1992; Maij et al., 2011). During saccades, the image shifts so quickly across the retina that details are blurred and masked by what is visible after the saccade (Duyck et al., 2018; Maij et al., 2012), and even the motion itself is too fast to see (Castet & Masson, 2000). There is a lot of interest in spatial vision near the moment of saccades, mostly driven by surprise that the world looks static despite its image shifting across the retina (Bisley et al., 2020; Burr & Morrone, 2012; Colby et al., 1995). However, if vision consists of actively selecting relevant information to process at each instant (see “Vision Is Very Selective”) and not of attempting to build a consistent internal representation of the outside world, changing gaze should simply be seen as one of the tools for selecting relevant information. It seems pointless to constantly shift representations of earlier retinal images to combine them into a single representation of the scene when information can be obtained from any position in the scene by simply looking there (O’Regan & Noë, 2001).

Saccades themselves, our most common movements, are often overlooked as actions because their goal is usually to help us to obtain information for other actions. Due to the poor localization around the moment of a saccade, a slight shift of a target during a saccade is not noticed. Despite not being noticed, such a shift obviously gives rise to an error in fixating the target. Repeatedly shifting the target to a position further (or closer) than its initial position during the saccade results in subsequent saccades being lengthened (or shortened) so that they end closer to the target (Pélisson et al., 2010). Such repeated shifts of target position during a saccade sometimes influence judgments of that position before the saccade, while the target is still in the periphery (Collins et al., 2007; Gremmler et al., 2014; Kröller et al., 1999), although they do not always do so (Hernandez et al., 2008; Schnier & Lappe, 2012). That precise information gathered after a saccade can calibrate judgments made before the saccade, when relying on peripheral vision, has also been found for judgments of shape (Herwig et al., 2015) and size. Increasing a target’s size during saccades toward the target, such that it was larger when viewed in central vision than when viewed in the periphery, led to its size being overestimated when viewed in the periphery (Bosco et al., 2015; Valsecchi & Gegenfurtner, 2016). In accordance with the notion that consistency between judgments is not imposed (see “Vision Is Not Always Consistent”), the relationship between changes in judged size and judged position were not consistent across adapting procedures (Bosco et al., 2020; Pressigout et al., 2020; Valsecchi et al., 2020). Thus, spatial vision can be attuned through experience (Zimmermann & Lappe, 2016).

Movements can be attuned in various ways, and doing so can even influence perceptual judgments. There appear to be many mechanisms involved, and at least some mechanisms return to specific nonveridical values when all feedback is removed, so presumably the mapping between the retina (spatial vision) and the limbs (action) has some intrinsic structure and cannot be arbitrarily reassigned.

Selecting Information to Rely On

Which spatial attributes can one best rely on to guide an action? When two related attributes provide redundant information, it is not necessary to use them both. Which should one use? Can one also totally neglect an attribute? Which attributes one relies on might depend on how quickly and reliably various attributes can be used to guide the action. It might also depend on how and where information about the attribute can be obtained. People direct their gaze at relevant items (Hayhoe & Ballard, 2005; Land, 2006) and move their eyes in a way that optimizes the processing of the retinal image (Intoy & Rucci, 2020; Rucci & Victor, 2015), but gaze can only be directed at one place at a time so one may need to choose what information to prioritize (whereby one presumably considers the extent to which judgments deteriorate with retinal eccentricity). Consequently, some attributes that might appear to be very useful may not be used to guide movements, or they may be judged so unreliably that the visual judgment has to be combined with expectations.

An example of a visual attribute that one might expect to be considered, but for which there is ample evidence that it is not, is acceleration (Lee et al., 1983). In the case of acceleration by gravity, people rely on experience rather than judging acceleration visually (Jörges & López-Moliner, 2017; McIntyre et al., 2001), presumably because prior experience provides a more reliable estimate than the instantaneous visual information. However, when faced with less familiar accelerations, the visual information is still neglected. It may simply not be worth considering visual judgments of acceleration because it takes so much time to obtain a reliable estimate that it is better to only consider the position and velocity (Brenner et al., 2016).

An example that illustrates combining visual information that is essential for guiding an action with expectations based on the actor’s recent experience is in intercepting moving targets, where performance relies not only on the target’s speed in the current trial but also on its speed in the previous trial (de Lussanet et al., 2001). Similar reliance on recent experience has also been found in grasping, where performance is sometimes influenced by the previously felt distance (Volcic & Domini, 2018) and shape (Cuijpers et al., 2008).

Knowing which attributes are used to guide various aspects of one’s actions may seem to be relevant for only a small selection of specialists in motor control. However, ignoring such selection can have far-reaching consequences for how studies are interpreted, especially if there are substantial inconsistencies between related attributes in those studies. Such substantial inconsistencies are likely to occur in studies involving visual illusions (see “Vision Is Not Always Consistent”). The next two sections provide examples of the importance of considering which attributes guide certain actions. The first combination of attribute and action is the use of judged size to guide reaching out to grasp a static object. The second is the use of judged motion when intercepting a moving target.

Size and Grasping

The reach-to-grasp movement provides a nice example of how misjudging the attributes that are used can lead to wrong conclusions (for a more extensive analysis of the literature on this issue, see Smeets et al., 2019). In an influential study, participants were presented with two objects, one of which was surrounded by small items and the other by large items (the Ebbinghaus illusion; Aglioti et al., 1995). When the two objects were matched in apparent size (rather than physical size), the maximal grip aperture when reaching out to grasp them was different. When the objects were matched in physical size, the effect of the surrounding items on the maximal grip aperture was about half the effect on the judged sizes (Aglioti et al., 1995, Figure 5). The authors considered these results to show that the attribute size is judged separately for perception and action. But is this what they show?

Various authors have discussed how experimental details such as the continuous visibility of hand and target may have influenced the result. Since the target object and the hand were continuously visible during the action, and maximum grip aperture occurs when the hand is close to the target, the effect of the illusion on the perceived size of the target might have decreased as the hand approached the target. This has been interpreted in terms of using different information for planning and control (Glover & Dixon, 2002; but see Smeets et al., 2003). Various authors have also discussed that one can expect an approximately 20% smaller effect of any illusion on maximum grip aperture than on perceptual judgments, because increasing object size generally only results in an increase in grip aperture of about 80% of the increase in size (Smeets & Brenner, 1999). Scaling the illusion effects to compensate for this can change the conclusion (Hesse et al., 2016).

The discussion about these issues has resulted in experiments being performed in which online control was prevented by blocking vision of both the hand and the target during the action. Comparing the influence of the Ebbinghaus illusion on size judgments with its influence on maximum grip aperture in such experiments and considering the change in grip aperture with target size in the analysis shows that the illusion can influence grip aperture in accordance with its effect on judged size (Kopiske et al., 2016). Thus, the idea that size is judged fundamentally differently for perceptual judgments than for planning grasping actions is wrong (also see e.g., Uccelli et al., 2019). This conclusion is of course only valid if the attribute that is being considered—size—is used to guide the opening of the hand to grasp the object.

A hint as to why it took so long to establish that illusions influence actions in the same way they influence perceptual judgments is the manner in which early studies using the Ponzo illusion, which was shown not to influence peak grip aperture (Brenner & Smeets, 1996; Jackson & Shaw, 2000), were received. These studies are frequently cited in support of the idea that action is not susceptible to illusions, although the same studies clearly showed that the build-up of the grip and lifting forces are affected by the illusion, even if grip aperture is not. An alternative interpretation is that judged object size does not determine how wide the hand is opened in grasping, so an illusion that influences judged size influences lifting forces but not the grip aperture when reaching to grasp the object. It is generally assumed that grip opening is guided by judged size (Jeannerod, 1988), but Smeets and Brenner (1999) formulated an alternative view on grasping in which the relevant attribute is the position of the contact points. That grip aperture is guided by judged positions rather than judged size is consistent with the variability in grip aperture being independent of the size (Ganel et al., 2008; Smeets & Brenner, 2008), because one would not expect the variability to scale with size (following Weber’s law) if people do not rely on judgments of size (but see Bruno et al., 2016, or Utz et al., 2015, for an alternative explanation).

That grip aperture is guided by judged positions rather than judged size is also the only way to explain why grip aperture when grasping an object changes in accordance with the anticipated aftereffects for the individual digits after adapting the digits independently to spatial offsets in opposite directions (prism adaptation) while making pointing movements (Schot et al., 2017). That reaching to grasp an object is determined by the movements of the individual digits, rather than by the movement of the hand and the closing of the grip, can also explain why the digits’ movements can depend on the orientation of an object’s surfaces even if the final grip is the same (see Figure 3A, B). The finding that individual digits move quite similarly when grasping and pushing if the constraints on their movements are similar (see Figure 3C, D) also suggests that grasping is controlled based on positions rather than size.

There is no clear neural evidence indicating whether size or positions are used in the control of grip aperture. Differences in brain activation have been found between grasping and similar actions that do not involve closing one’s grip (Di Bono et al., 2015; Fattori et al., 2010). Such differences are sometimes interpreted as evidence for separate neuronal processing of the grip, but those studies did not match the tasks that are compared in terms of the constraints on the movements of the digits.

Open in new tab

Figure 3. Schematic representation of the paths of the index finger and thumb when grasping simple objects. When grasping trapezoidally shaped objects (A, B), the digits’ paths to the same positions depend on the object’s orientation, despite the fact that the final grip is identical (Kleinholdermann et al., 2007). This is presumably the result of it being advantageous to approach surfaces orthogonally (Smeets & Brenner, 1999). When grasping an object, each digit ultimately has to push onto the surface with which it makes contact (C). When asked to push the object away with one digit (D) that digit moves in much the same manner as it does during grasping (Smeets et al., 2010).

Guiding grip aperture by judged positions is not always possible. For instance, when one cannot see the object that is grasped, one might have to switch to a strategy in which grip aperture is based on judged size. Evidence that this happens has been found using the Müller-Lyer illusion: the illusion does not influence peak grip aperture when the hand and target are visible throughout the movement, but it does influence the aperture in open-loop grasping (Bruno & Franz, 2009; Foster et al., 2012). Presumably, real-time information about egocentric positions is preferable, but if such information is lacking because vision is blocked, one switches to remembered allocentric information. Another example of a task not being performed in the usual manner without direct visual feedback is forgetting to restore the original driving direction when asked to turn a steering wheel to change lanes when driving (Wallis et al., 2002). Similarly, the movements that are made in pantomimed grasping are very different from those of real grasping (Goodale et al., 1994a).

The fact that the Ebbinghaus illusion normally influences grasping to some extent (Aglioti et al., 1995; Pavani et al., 1999) might appear to suggest that judgments of size must be relevant for grasping. However, besides influencing judgments of size, the Ebbinghaus illusion also influences judged positions. It does so in accordance with the changes in judged size (Smeets & Brenner, 2019). For some reason, the Ebbinghaus illusion lacks the inconsistency between size and position that characterizes other illusions (Gillam & Chambers, 1985; Smeets et al., 2020). Importantly, the latter illusions do not influence the grip aperture when grasping normally (Smeets et al., 2020).

Interacting With Moving Objects

A second example of an inconsistency between attributes that has led to a lot of confusion is between judgments of motion and judgments of (changes in) position. An effective way to intentionally create such an inconsistency is with a moving Gabor stimulus. This stimulus influences judged motion to a much greater extent than it does judged displacement (de la Malla et al., 2018; Movie 6). It can therefore be used to study how judgments of motion guide people’s actions. This is obviously an artificial stimulus, but the idea of texture on an object moving differently than the object as a whole, and thus influencing its apparent speed, is less contrived than it may seem. It can happen when a textured ball rolls across a surface (de la Malla et al., 2017) or spins as it flies through the air (Casanova et al., 2015; Shapiro et al., 2010). Moreover, although the judgments about what is happening in the scene are inconsistent with a physical scene, the scene does not look “wrong” in any way.

Moving Gabor stimuli have been used to study how visual information guides the human eye or hand toward moving objects. One way in which this has been done is by taking a Gabor with a laterally moving vertical grating and moving it up and down along a path that is tilted so that the Gabor appears to move vertically when viewed while fixating off to the side (Figure 4; Lisi & Cavanagh, 2015; Shapiro et al., 2010). Despite appearing to move vertically, when the participants made saccades to the Gabor, the landing points followed the true position (Lisi & Cavanagh, 2015). This was interpreted as showing that actions are guided by instantaneous information while perceptual judgments integrate information across time. An alternative interpretation is that the direction of motion was misjudged, and that this judgment of direction was inconsistent with the judgments of position. The judged positions were systematically misjudged, but by a modest constant amount (see De Valois & De Valois, 1991). The authors were comparing reporting on a judged direction of motion with making saccades to an anticipated position.

Open in new tab

Figure 4. Schematic representation of the study by Lisi and Cavanagh (2015), with two possible interpretations of the data. (A) A Gabor stimulus is a grating of which the contrast is modulated by a two-dimensional Gaussian profile. If one is fixating to the side (+), and the grating is moving to the right within the Gaussian, the Gaussian has to move leftward as it moved downward for the Gabor as a whole to appear to move straight down. (B) Combining a bias in the perceived position of the Gabor in the direction of the motion of the grating (red dot) with the Gabor’s apparent downward motion (blue arrow) to anticipate where it will be a short duration in the future (when the saccade ends) results in a saccade endpoint that shifts consistently with the true future position of the Gabor (black arrow). (C) Thus, when describing what they see, participants describe a Gabor moving up and down, presumably at the average horizontal position, but when making saccades to the Gabor the saccade endpoints follow the instantaneous horizontal position.

The alternative interpretation is consistent with several other studies that have been conducted with similar stimuli. In one case (Lisi & Cavanagh, 2017), the same stimulus was used except that a brief flash was superimposed on the Gabor when it reached one of the reversal points. Participants were asked to saccade or move their finger to where the flash had been. Saccades that had a short latency were hardly influenced by the misjudged direction of motion; presumably they were directed to the instantaneous position of the flash immediately after the flash. Misjudging the direction of motion did influence the finger, presumably because the finger had to be guided to the remembered position of the flash. Misjudging the motion probably influenced the remembered position because the latter was inconsistent with a combination of the judged motion and the instantaneous position at later moments. In accordance with this interpretation, misjudging the motion also influenced saccades that had a long latency to some extent (as it did saccades that were intentionally delayed; Massendari et al., 2018).

That the distinction between judgments of position and of motion is critical, rather than that between perception and action, is evident from yet another study using the same stimulus. In this study, the Gabor disappeared once a saccade was made and participants had to make perceptual judgments about the place from which it had disappeared (Nakayama & Holcombe, 2020). In accordance with the Gabor not being present to influence the remembered position, not only did saccades land near the actual position at which the target had been, but that position was indicated correctly. In accordance with positions also being influenced to some extent in these Gabor stimuli, both judged positions and saccadic endpoints were slightly shifted in the direction in which the grating was moving (Ueda et al., 2018).

In another experiment involving a moving Gabor, participants were asked to tap on the Gabor when it reached a specified position that participants were fixating with their eyes (de la Malla et al., 2018). In this case, the motion of the grating was either in the same direction as or in the opposite direction from the motion of the Gabor as a whole, so the Gabor appeared to move faster or more slowly toward the fixation point rather than appearing to move in a different direction. Separate experiments were used to determine how the moving grating influenced the judged velocity of the Gabor, and its judged position 100 ms before it reached the fixation point (movements are adjusted to the latest visual information until about 100 ms before they end; Brenner & Smeets, 1997, 2018b). In these experiments the Gabor disappeared 100 ms before it reached the fixation point, and judgments were made about the speed and position at that time. Combining these judgments gives a quantitative prediction of the anticipated tapping error. The actual tapping errors were very similar to these predictions. This illustrates the importance of considering not only which attributes of spatial vision are used to guide actions but also when and how they are used.

Beyond Visual Illusions

Visual illusions provide a useful tool for trying to identify the visual attributes that guide our actions, but they obviously do not constitute the only tool that one has for identifying the information that is used, and other tools are certainly more suitable for exploring how such information is used. Manipulating the available visual information at different times can help determine when information is used (Dessing et al., 2009; Droll & Hayhoe, 2007; López-Moliner et al., 2010; Whiting & Sharp, 1974) and how long it takes to use it (Veerman et al., 2008). Modeling how visual information guides movements, considering strategic choices and the relevant biomechanics, can prevent inadvertently attributing inevitable aspects of the way movements are executed to errors in spatial vision (Brenner & Smeets, 2018a; Dessing et al., 2002; de Lussanet et al., 2002; Smeets & Brenner, 1999). This is important because even systematic errors in performance do not necessarily reflect errors in either visual information or in the commands to the muscles that move the limbs. They may be tactical compromises (Harris, 1995; Liu & Todorov, 2007), such as considering issues like maintaining balance when walking (Barton et al., 2019). All these considerations imply that when relying on models one must keep in mind that if the model is wrong the interpretation in terms of spatial vision is likely to be wrong as well, because the model determines how the attribute contributes to the action (de Lussanet et al., 2004; Pinter et al., 2012; Smeets & Brenner, 2002; Smeets et al., 2002a, 2003). This may make relying on models less attractive, but it is probably impossible to even think of how spatial vision guides actions without having some model in mind.

When the question is not what attributes are used but how those attributes are judged, a popular method is to examine how people respond to cue conflicts. This technique has been used extensively in experiments involving depth. Contrary to suggestions that binocular depth cues have a special role in guiding movements (e.g., Bruggeman et al., 2007; Servos et al., 1992), studies using specially made stimuli containing cue conflicts have shown that the extent to which cues are used depends on their precision (Camponogara & Volcic, 2019; Knill, 2005; Louw et al., 2007), on whether the information that they provide is consistent with the haptic feedback (Cesanek & Domini, 2019; van Beers et al., 2011), and on how quickly they provide information (van Mierlo et al., 2009). Precision (Knill, 2005) and accuracy (Cesanek et al., 2020) also influence perceptual judgments. The influence of how quickly cues provide information is probably only important in guiding actions, where a small temporal advantage is likely to result in movements primarily being adjusted on the basis of the faster cue when there is no cue conflict.

Epilogue

Ironically, the very studies that led to the realization that it is worthwhile to consider what visual information is used to guide human actions also held back research on this topic to some extent. By identifying systematic differences between influences of damage to dorsal and ventral regions of the brain and how illusions influence tasks that were chosen to represent perception or action, it has become widely accepted that perception and action rely on separate processing of visual information (reviewed in Goodale, 2011; Milner & Goodale, 2008; but see Schenk & McIntosh, 2010). This work has introduced important distinctions, such as the fact that action is primarily guided by egocentric, instantaneous information about positions and motion. However, the overarching notion of separate processing of visual information for perception and for action has led many researchers to explain all discrepancies that they observe as a result of this dichotomy, rather than considering what one could learn from them. Abandoning both this dichotomy and the notion of consistency between spatial attributes provides the opportunity to use situations in which different attributes are affected in different ways to determine what attribute is used for a given task and how, rather than just considering differences as showing that not all tasks use the same information. This article described how doing so helped develop the idea that grip aperture is guided by positions rather than object size and explained how motion information is used to guide the hand when intercepting a moving target. Irrespective of whether this interpretation of these particular examples is correct, this approach of trying to understand how specific actions are guided by various kinds of (visual) information could be very fruitful.

People are reluctant to accept the compelling evidence that humans do not perceive the world around them correctly and in fine detail. The reason that people are so confident about the consistency of what they see is probably the constant reinforcement by the experience that they can rely on spatial vision to guide their actions. This ability is indeed amazing. Unless you have some disease, or the circumstances are very unusual (for instance you are in the dark or intoxicated, or for some reason need to look elsewhere), you probably have no difficulty placing your fingers at suitable positions on an object that you want to manipulate or placing your foot at suitable places when walking or running. This gives the impression that you see the world as it is and act on it. However, it is clearly not that simple. Spatial vision is used in very clever ways to make this possible, and there is still much to discover about this exciting topic.