Visuality, Visibility, and Big Data

The meaning of the word “visuality” has evolved over the years. As Sand (2012, p. 89) points out, the Oxford English Dictionary’s four definitions of visuality—(a) “The state or quality of being visual or visible to the mind; mental visibility”; (b) “A mental picture or vision”; (c) “Vision, sight”; (d) “Visual aspect or representation; physical appearance”—have overlooked a “newly complex and technical” use of visuality in art history since 1988. In this context, visuality has come to mean “the element of visual experience that is contingent on culture and therefore far more unstable and resistant to description than even the most complex of biological functions” (2012, p. 89). This fifth meaning of the term is explained by Foster (1988, p. ix), when he discusses two aspects of the visual and their connections: “Although vision suggests sight as a physical operation, and visuality sight as a social fact, the two are not opposed as nature to culture: vision is social and historical too, and visuality involves the body and the psyche.” Thus the term visuality relates to both the quality of (physical and mental) visibility and the (social, historical, cultural) construction of visual experience. More recently, Mirzoeff (2011, p. xv) has traced the genealogy of visuality and tied it more directly to authority: “I consider visuality to be both a medium for the transmission and dissemination of authority, and a means for the mediation of those subject to that authority.” For Mirzoeff, visuality is a discursive practice that classifies, separates, and aestheticizes history in such a way that makes the domination of authority seem natural and self-evident. Countervisuality, on the other hand, is “the right to look,” which “claims autonomy from this authority, refuses to be segregated, and spontaneously invents new forms” (2011, p. 4). As Halasz (2013, p. 90) points out, “[a]t stake in these battles between visuality and countervisuality is cultural and political power.”

In his analysis of the work of CCTV operators, Smith (2014, pp. 15–16) conceptualizes operators’ work as a form of “supervisory practice” partly defined by visibility and visuality as “co-relational, iterative, and conjoined entities.” For Smith, CCTV’s supervision of the streets depends on a “visibility-visuality alternation, where visibility exertions (a projected gaze) interweave with visuality spectacles (a reflected outlook)” (2014, p. 13). Smith’s analysis suggests that it is through the “co-productive” processes of “visibility making (revealing things)” and “visuality processing (construing things)” that abnormalities are defined, decisions made, and interventions directed within supervisory systems (2014, p. 21). Empowered to both reveal (e.g., select and focus) and construe (e.g., decide on appropriate action), CCTV camera operators in Smith’s research often act as risk assessors and gatekeepers of police deployment.

The discussion of visibility and visuality so far has assumed the centrality of the human body, especially its capacity for vision. As Paglen (2016) has observed, over the last decade or so, with the massive digitation of images, visual culture “has become detached from human eyes and has largely become invisible”:

What’s truly revolutionary about the advent of digital images is the fact that they are fundamentally machine-readable: they can only been seen by humans in special circumstances and for short periods of time . . . However, the image doesn’t need to be turned into human-readable form in order for a machine to do something with it . . . The fact that digital images are fundamentally machine-readable regardless of a human subject has enormous implications. It allows for the automation of vision on an enormous scale and, along with it, the exercise of power on dramatically larger and smaller scales than have ever been possible.

(Paglen, 2016)

Thus, even where human-to-human sharing (such as sharing photographs on social media) appears to be the primary purpose of some digital platforms, any analogy with previous practices of sharing physical photographs with friends is “deeply misleading”: uploading an image on social media amounts to “feeding an array of immensely powerful artificial intelligences systems information about how to identify people and how to recognize places and objects, habits and preferences, race, class, and gender identifications, economic statuses, and much more.” More fundamentally,

In aggregate, AI systems have appropriated human visual culture and transformed it into a massive, flexible training set. The more images Facebook and Google’s AI systems ingest, the more accurate they become, and the more influence they have on everyday life. The trillions of images we’ve been trained to treat as human-to-human culture are the foundation for increasingly autonomous ways of seeing that bear little resemblance to the visual culture of the past.

(Paglen, 2016)

In spite of this, the performance of Big Data often engages with visuality. The data “deluge” is not necessarily seen as a “river of gold” bearing business opportunities and tools for innovation; more likely it promotes anxieties among “surveillers” relating to missing clues as well as overload: “that no matter how much data they have, it is always incomplete, and the sheer volume can overwhelm the critical signals in a fog of possible correlations” (Crawford, 2014). To the rescue are myriad analytical tools that promise to translate the rapidly building mountains of information into simple, accessible visual cues to overcome the limitation of the human brain to digest large volumes of complex information.

As Leese (2016) points out, the “strength of vision,” based on a sense of ocularcentrism, has remained an unshakable assumption among the “exact” sciences, where information visualization (infoviz) has emerged as a dominant way of presenting insights from data. Seen as a rigorous and neutral technique, infoviz or dataviz is nevertheless a matter of design as well as science:

The toolbox of infovis is in fact stacked with elaborate measures of visual representation, including the likes of scatterplots, box plots, heat maps, 3D coordinates, link graphs, treemaps, node-link diagrams, and many more . . . Moreover, taking advantage of design principles that facilitate cognitive access, visual elements such as colours, positions, motions, orientations, size, shape, saturation or hue . . . have been incorporated to enhance the presentation, and thus the understandability of data. By asking questions such as: “What is the best way of visualizing data? What choice of colour best supports the communication of properties we are interested in? Does shape and placement help improve perception?” (as cited in Marty, 2009, p. 9), it becomes quite clear that the infovis community conceives itself as a field that seeks to provide neutral techniques that are completely devoid of politics.

(Leese, 2016, p. 145)

The production of visuality is a practice that is political, as images operate in an economy where visuality is invested with an amount of “attention capital” that is enmeshed with various forms of political capital.

The Attention Economy

Georg Franck (1999, 2002) has written about the “scientific economy of attention,” where the attention received by scientists is “capitalised into the asset called reputation” (2002, p. 3). The economy of attention operates slightly differently in risk culture. The routine accumulation of massive volumes of information through communications and sensing devices does not demand attention in itself. Rather, it is the capacity of multinational commercial enterprises to turn these volumes of information into profit that shifts attention to these data collections, giving weight to the label “Big Data.” It is the potential of global intelligence apparatuses to make use of these data for surveillance and risk identification that alerts community groups and citizens to pay attention to the dangers of dataveillance and profiling of suspects. It is the uncomfortable realization that social media information can be manipulated (or automatically generated) to spread false news, incite emotions and actions, and influence thinking and opinions that bring to the attention of users and non-users the social harms that such communications can bring. Hence data has little intrinsic attention capital independent of its usage and societal reactions to its usage.

Nevertheless, data visualizations are not all equal in commanding attention: some are more memorable than others. In an online experiment researchers exposed 261 participants to 410 images of data visualization to measure the extent to which an image was readily detected (Borkin, Vo, Isola, Sunkavalli, Oliva, & Pfister, 2013). They found that “higher memorability scores were correlated with visualizations containing pictograms, more color, low data-to-ink ratios, and high visual densities” (2013, p. 2312). Perhaps not surprisingly, diagrams were found to be more memorable than bar charts, line graphs, and tables.

The performance of visuality—both in the sense of making data issues visible and reconfiguring data through visualization—clearly adds value to Big Data in the economy of attention. How attention capital is intertwined with political capital is examined below.

The Performance of Visuality and Its Consequences

Leese (2016) has argued that, in specific cases such as risk flags on PayPal and security alerts in airports, “translations of algorithmic calculations into visualisations” create an “affective startle” that can trigger reactions that over-ride other considerations:

complexity becomes transformed into reductionist binaries that unfold a particularly “dark” set of future narratives, as they turn fine-grained risk assessment into simple threat scenarios that must be acted upon . . . Anticipation then becomes dominated by an affective startle that intentionally seeks to override the full spectrum of contingency and prioritises a particularly bleak set of threatening futures that become folded back into the present.

(Leese, 2016, p. 143)

For Leese, the “politicality of visuality relies on opacity instead of transparency, and on reduction of complexity and contextuality—until all that remains is a plain visual artifact that produces no clear statement but the garnering of attention” (2016, p. 149). Ultimately, the affective triggers “create an immediate atmosphere of suspicion . . . [which] prioritises a distinctive set of negative narratives over possible more positive others” (2016, pp. 153–154). More generally, algorithms “function as a means of directing and disciplining attention, focusing on specific points and cancelling out all other data, appearing to make it possible to translate probable associations between people or objects into actionable security decisions” (Amoore, 2009, p. 22).

The performance of data visualization can have consequences for social groups, as demonstrated by a study of Sentiment in New York City, conducted by the New England Complex Systems Institute (NECSI) in 2013. The results, based on an algorithm to “automatically classify tweets as positive or negative, and use their geotags and time information to paint a map” (Bertrand, Bialik, Virdee, Gros, & Bar-Yam, 2013) mistakenly linked a high school with the lowest sentiment in Manhattan. The researcher admitted later that “the negative tweets were not primarily from the high school, but rather from a region just south of the school. In particular, a single twitter account was the source of many of the tweets classified as having negative sentiment in this area” (Kaplan, 2016; Kilgannon, 2013a, 2013b). This prompted a commentator to urge caution in data visualizations:

Data visualizations might seem inert, but there are many ways they can cause harm. A visualization might bring unwanted attention to a person or a group (or a high school). A map can trivialize human experience, by reducing life to a dot or a vector. Representations of violent or tragic events can be traumatic to people that were directly or indirectly involved. We’re well trained to be aghast at a truncated Y-axis or an un-squared circle; we need to expand our criticality to include the possible social impacts of “well-made” visualizations.

(Thorp, 2016, p. 15)

In general, as Kennedy, Hill, Aiello, and Allen (2016) have argued, data visualization is not a neutral or automated process but can be a tool for political ends. Apparently “technical” conventions of visualization do ideological work “to imbue visualisations with the quality of objectivity (which brings together other qualities such as transparency, scientific-ness and facticity)” while masking the power implications of these conventions (2016, p. 716). This entanglement of power and practice is complex, so that perceived technical qualities of visualization are not always independent of gender, class, or age related cultural assumptions (Hill, Kennedy, & Gerrard, 2016).

Data Visualization for Justice

While Leese (2016) has focused on the performance of Big Data visualization in creating suspicion and making suspects, this kind of visualization can also help to concentrate attention on criminal harms and injustices. Three examples are discussed.

One example is the use of standard statistical representations (line graphs, bar charts, and data maps) to visualize the unprecedented growth of imprisonment in the United States (Shannon & Uggen, 2013). The authors want to address the “relative lack of attention to the punishment boom” by communicating complex trends through visual narratives. Their visuals document the steep climb in incarceration rates since the early 1970s, the massive growth in correctional populations (including probation and parole), the increase in imprisonment and correctional control of African Americans in every state, and the extent of “felon disenfranchisement” in some states. Told in colorful graphics with accessible text, the narrative calls for alternative visions of penal policy choices.

A second example (see Figure 1) is an interactive visualization of gun deaths in the United States in 2010 and 2013 by Persicopic. It is a powerful representation of the tragedy of gun violence in America. Each death is animated as an arc across an axis of age. By clicking on each arc, the viewer is shown a brief summary of the name, sex, age, location, and date of death by shooting, and the normal life expectancy of the victim cut short by the death. Viewers can filter the information by sex, age group, region, or time period, as well as read summaries of “What This Data Reveals.” The visualization in 2013 was based on different sources of data from 2010. While the 2010 gun deaths data came from the FBI’s Uniform Crime Reports, the 2013 data was based on real-time crowd-sourced information from online news sources, through the anonymous Twitter user @GunDeaths. The website acknowledges that the 2013 data was less reliable and included suicides as well as homicides.

Open in new tab

Figure 1. US gun killings in 2010.

Source: US Gun Deaths, The Stolen Years^© (2013). Periscopic[www.periscopic.com], reproduced with permission.

The third example involves an ambitious project by ProPublica exposing the bias of Big Data analytics (Angwin, Larson, Mattu, & Kirchner, 2016—see Figure 2). It contains mostly textual content and visualized case studies and only two bar charts. By comparing the risk assessment scores generated by a computer algorithm on sets of two offenders convicted of the same crime, the ProPublica study not only demonstrates the “remarkably unreliable” nature of the scores, it points to the “significant racial disparities” generated by the software: “The formula was particularly likely to falsely flag black defendants as future criminals, wrongly labelling them this way at almost twice the rate as white defendants” and “White defendants were mislabelled as low risk more often than black defendants.” This effect remains even after criminal history and demographics (age and gender) are controlled for. The website provides various documents (sentencing reports with risk assessments), full data used in their analysis and a detailed account of how they analyzed the algorithm. It also includes a response by the company Northpointe, which created the software: the company “criticized ProPublica’s methodology and defended the accuracy of its test”; the proprietary nature of the specific calculations, however, prevented a more transparent analysis of the risk scores.

Open in new tab

Figure 2. Comparison of risk scores between two arrestees.

Source: Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016).Machine Bias. ProPublica. Permission being sought.

These examples reflect the “belief that visualisations can promote data transparency and awareness . . . a belief in the benefits of information as a means of empowerment” (Kennedy et al., 2016, p. 721). Though not direct contestations of the powerful’s right to conceal (see Brown’s “Visual Criminology”), these visualizations are useful for revealing harms, injustices, and practices that are not normally visible. They are “counter-visuals” that attempt to present aggregate trends and, in the third example, also individualized portraits (Brown, 2014). Such attempts, of course, require a great deal of resources and expertise not usually available to community groups, so that differential access to resources and visualization strategies is likely to favor well-resourced agencies and sophisticated users.

Anesthetic Effects of Visual Representation

Our analysis of visuality of Big Data so far has focused on one possible way in which affective responses are evoked—through the affect that visualization could trigger. But it is possible to argue the opposite, that data-based visuality could dampen emotionality or at least insulate readers from the deeper level of emotionality that case-based visuality is capable of evoking. The evidence that visuality can dampen emotionality even in cases of extreme violence has been a subject of discussion among visual criminologists. Carrabine (2012) cites Susan Sontag (1977) in relation to photographic images of atrocity, which can both “transfix” and “anesthetize,” so that photography can “deaden conscience” just as easily as “arouse it.” In discussing the ethics of representation, Carrabine emphasizes that the central issue “is a concern with our relationships to each other, and that these relationships must be premised on an understanding of difference” (2012, p. 466). In fact, his analysis of documentary photography cites critics who saw the camera as aestheticizing tragedies and deadening emotions:

Leading critics like Sontag (1977), Berger (1980) and Solomon-Godeau (1991), who have been influenced by Benjamin’s thinking, have each accused the documentary photograph of favouring a detached sentimentalism that mystifies political and historical reality. The complaint is that much photojournalism and social documentary exploit the other and reinforce the differences between the superior and inferior. The charge is that to aestheticize tragedy will ultimately deaden the feelings of those who witness suffering . . . (Carrabine, 2012, p. 478).

The advent of digital photography, the Internet, and social media may have democratized the production of images and brought about increased opportunities for amateurs to “bear witness” to crimes and injustices, but even disturbing scenes from Abu Ghraib did not “provoke so much shock as recognition amongst many commentators . . . these violations of humanity scarcely trouble consciences—a view borne out by the banal response to the cruelty in so many sections of American public opinion” (Carrabine, 2012, pp. 485–486).

It can be argued that data visualization, by removing the visibility of real people or events and aestheticizing the representations in the race for attention, can achieve further distancing and deadening of conscience in situations where graphic photographic images might at least garner initial emotional impact. In this sense the ProPublica presentation of text with photographs of real people, their predictive scores and subsequent offending record may be read as more emotionally charged than the Periscopic visuals of gun deaths, even though details of individual victims can be read; and both presentations are arguably more powerful than the standard bar charts and heat maps, such as those used by Shannon and Uggen (2013). As Brown (2014, pp. 181–182) points out, the unpredictability of interpretation of visual images is part of process: “Regardless of its uncertain outcomes, including voyeuristic spectacle, egregious appropriations, and silent apathy, the act of representation remains a vital form of social engagement.”