1-15 of 15 Results

  • Keywords: multimodality x
Clear all

Article

Cynthia M. Harley and Mark K. Asplen

Annelid worms are simultaneously an interesting and difficult model system for understanding the evolution of animal vision. On the one hand, a wide variety of photoreceptor cells and eye morphologies are exhibited within a single phylum; on the other, annelid phylogenetics has been substantially re-envisioned within the last decade, suggesting the possibility of considerable convergent evolution. This article reviews the comparative anatomy of annelid visual systems within the context of the specific behaviors exhibited by these animals. Each of the major classes of annelid visual systems is examined, including both simple photoreceptor cells (including leech body eyes) and photoreceptive cells with pigment (trochophore larval eyes, ocellar tubes, complex eyes); meanwhile, behaviors examined include differential mobility and feeding strategies, similarities (or differences) in larval versus adult visual behaviors within a species, visual signaling, and depth sensing. Based on our review, several major trends in the comparative morphology and ethology of annelid vision are highlighted: (1) eye complexity tends to increase with mobility and higher-order predatory behavior; (2) although they have simple sensors these can relay complex information through large numbers or multimodality; (3) polychaete larval and adult eye morphology can differ strongly in many mobile species, but not in many sedentary species; and (4) annelids exhibiting visual signaling possess even more complex visual systems than expected, suggesting the possibility that complex eyes can be simultaneously well adapted to multiple visual tasks.

Article

Kathy A. Mills and Len Unsworth

Multimodal literacy is a term that originates in social semiotics, and refers to the study of language that combines two or more modes of meaning. The related term, multimodality, refers to the constitution of multiple modes in semiosis or meaning making. Modes are defined differently across schools of thought, and the classification of modes is somewhat contested. However, from a social semiotic approach, modes are the socially and culturally shaped resources or semiotic structure for making meaning. Specific examples of modes from a social semiotic perspective include speech, gesture, written language, music, mathematical notation, drawings, photographic images, or moving digital images. Language and literacy practices have always been multimodal, because communication requires attending to diverse kinds of meanings, whether of spoken or written words, visual images, gestures, posture, movement, sound, or silence. Yet, undeniably, the affordances of people-driven digital media and textual production have given rise to an exponential increase in the circulation of multimodal texts in networked digital environments. Multimodal text production has become a central part of everyday life for many people throughout the life course, and across cultures and societies. This has been enabled by the ease of producing and sharing digital images, music, video games, apps, and other digital media via the Internet and mobile technologies. The increasing significance of multimodal literacy for communication has led to a growing body of research and theory to address the differing potentials of modes and their intermodality for making meaning. The study of multimodal literacy learning in schools and society is an emergent field of research, which begins with the important recognition that reading and writing are rarely practiced as discrete skills, but are intimately connected to the use of multimodal texts, often in digital contexts of use. The implications of multimodal literacy for pedagogy, curriculum, and assessment in education is an expanding field of multimodal research. In addition, there is a growing attention to multimodal literacy practices that are practiced in informal social contexts, from early childhood to adolescence and adulthood, such as in homes, recreational sites, communities, and workplaces.

Article

Frank Serafini

Visual literacy was originally defined as a set of visual competencies or cognitive skills and strategies one needs to make sense of visual images. These visual competencies were seen as universal cognitive abilities that were used for understanding visual images regardless of the contexts of production, reception, and dissemination. More contemporary definitions suggest visual literacy is a contextualized, social practice as much as an individualized, cognitively based set of competencies. Visual literacy is more aptly defined as a process of generating meanings in transaction with multimodal ensembles that include written text, visual images, and design elements from a variety of perspectives to meet the requirements of particular social contexts. Theories of visual literacy and associated research and pedagogy draw from a wide range of disciplines including art history, semiotics, media and cultural studies, communication studies, visual ethnography and anthropology, social semiotics, new literacies studies, cognitive psychology, and critical theory. Understanding the various theories, research methodologies, and pedagogical approaches to visual literacy requires an investigation into how the various paradigm shifts that have occurred in the social sciences have affected this field of study. Cognitive, linguistic, sociocultural, multimodal, and postmodern “turns” in the social sciences each bring different theories, perspectives, and approaches to the field of visual literacy. Visual literacy now incorporates sociocultural, semiotic, critical, and multimodal perspectives to understand the meaning potential of the visual and verbal ensembles encountered in social environments.

Article

Healthy aging is accompanied by decrements in episodic memory and working memory. Significant efforts have therefore been made to augment episodic and working memory in healthy older adults. Two principal approaches toward memory rehabilitation adults are restorative approaches and compensatory approaches. Restorative approaches aim to repair the affected memory processes by repeated, adaptive practice (i.e., the trained task becomes more difficult as participants improve), and have focused on recollection training, associative memory training, object-location memory training, and working memory training. The majority of these restorative approaches have been proved to be efficacious, that is, participants improve on the trained task, and there is considerable evidence for maintenance of training effects weeks or months after the intervention is discontinued. Transfer of restorative training approaches has been more elusive and appears limited to other tasks relying on the same domains or processes. Compensatory approaches to memory strive to bypass the impairment by teaching people mnemonic and lifestyle strategies to bolster memory performance. Specific mnemonic strategy training approaches as well as multimodal compensatory approaches that combine strategy training with counseling about other factors that affect memory (e.g., memory self-efficacy, relaxation, exercise, and cognitive and social engagement) have demonstrated that older adults can learn new mnemonics and implement them to the benefit of memory performance, and can adjust their views and expectations about their memory to better cope with the changes that occur during healthy aging. Future work should focus on identifying the personal characteristics that predict who will benefit from training and on developing objective measures of the impact of memory rehabilitation on older adults’ everyday functioning.

Article

Michael Karlsson and Kristoffer Holt

In the early 21st century, almost everyone takes journalism on the web for granted. However, it was not many years ago that journalism moved online and a distinct form of journalism began to develop. Ranging from online doubles of the paper editions to publications exclusively produced for the web, the evolvement of web journalism has entailed both dramatic and not-so-dramatic changes in the way that journalistic products are produced, disseminated, and received. Online journalism has usually been demarcated from traditional journalism by four traits: interactivity, immediacy, hypertextuality, and multimodality. These characteristics are generally identified by scholars as points where journalism on the web brings added value in comparison to the old print newspapers. Interactivity involves various aspects of user activity and participation in the processes of consuming, contributing to, and disseminating news afforded by the web. Immediacy refers to the nature and consequences of the faster pace of publication in web news. Hypertextuality has to do with the possibilities of linking journalistic texts to other texts, which makes the text more transparent and open. Multimodality denotes the telling of news with the use of many different modes at the same time. When studying research about these aspects of web journalism, three general observations can be made. First, researchers have approached these characteristics unevenly in terms of scope and interest. The interactive aspects of web journalism are by far the most investigated. Second, the four characteristics have been studied through the lenses of different theoretical frameworks. Third, empirical research shows that change in journalism is slow and not always as radical as many predicted when journalism on the web was in its infancy.

Article

From a digital culture perspective, this article has as main objective to assess two contemporary qualitative research methods in the field of education with distinct theoretical orientations: the cartographic method as a way of tracing trajectories in research-intervention with a theoretical basis in the biology of knowledge, enactive cognition and inventive cognition; and the cartographic method as a means of identifying and mapping the controversies linked to the different associations between human and non-human actors with a theoretical basis in actor-network theory (ANT). With their own specificities, both methods have been fruitful in the development of qualitative research in the field of education, in the context of digital culture, and more recently, in the hybrid culture of atopic habitation, mainly because they also relate to equally consistent theories and aspects of human cognition, making it possible to detect traces and clues in the fluid associations between actors enhanced by different digital technologies (DT), including data mining and learning analytics. From the Brazilian perspective on the topic, this article approaches the experience of the cartographic method of research intervention as well as the cartography of controversies as tools for developing qualitative research in education. These different forms of the cartographic method have inspired the construction of didactic-pedagogical experiences based on theoretical approaches linked to cognition, producing inventive methodologies and interventionist pedagogical practices. These methodologies and practices, which will be discussed at length in this article, have been developed and validated by the Research Group in Digital Education at Unisinos University at different levels and in varied educational settings.

Article

Valentin Werner

Pop cultural linguistics represents an emerging research subfield. It can be conceptualized as a specific type of media linguistics concerned with the study of performed language as represented in various pop culture manifestations, such as music, TV series, movies, comics, cartoons, and video games, among others. Pop culture is thus viewed as a broad category that includes artifacts with a commercial, entertainment-related purpose that are (mass-)mediated, fall within the mainstream, and represent largely fictional and scripted content. Linguists working in pop cultural linguistics explicitly take account of the current social and practical relevance of pop culture and the fact that it is largely a multimodal phenomenon with a strong linguistic component and the potential for affective engagement. Pop cultural linguistics possesses inherent relevance for the broader area of cultural studies, which may benefit from quantitative and qualitative approaches used in linguistics to increase the overall validity of findings and to develop a comprehensive picture of pop culture artifacts. The main object of study in pop cultural linguistics is performed language. While performed language was traditionally sidelined in linguistics due to its alleged “inauthentic” nature, it has gradually been acknowledged as a regular part of everyday language use and thus has been normalized in linguistic study. The increasing availability of resources relevant for pop cultural linguistics, such as language corpora and thematic bibliographies, illustrates the vitality of the field, as does the growing body of research. Research in pop cultural linguistics is methodologically eclectic and commonly adapts approaches and frameworks used in established linguistic subfields, such as sociolinguistics, stylistics, or corpus linguistics. It serves to explore salient topics, such as the linguistic construction of authenticity and identity from a sociolinguistic angle or the representation of politeness from a pragma-stylistic point of view, occasionally also applying a contrastive perspective in terms of performed language vs. natural conversation. Pop cultural linguistics is further characterized by increasing methodological reflection and a growing recognition of the affordances of multimodal analysis, even though these aspects will have to be addressed more explicitly in the future.

Article

Visual representation has been important in communicating and constructing the environment as a focus for public and political concern since the rise of environmentalism in the 1960s. As communications media have themselves become increasingly visual with the rise of digital media, so too has visual communication become key to public debate about environmental issues, no more so than in public debate and the politics of climate change. This chapter surveys the methods, approaches, and frameworks deployed in emerging research on public-mediated visual communication about climate change. Research on the visual mediation of climate change is itself part of the emerging field of visual environmental communication research, defined as research concerned with theorizing and empirically examining how visual imagery contributes to the increasingly multimodal public communication of the environment. Focused on a sociological understanding of the contribution that visuals make to the social, political, and cultural construction of “the environment,” visual environmental communication research analytically requires a multimodal approach, which situates analysis of the semiotic, discursive, rhetorical, and narrative characteristics of visuals in relation to the communicative, cultural, and historical contexts and in relation to the three main sites—production, content, and audiences/consumption—of communication in the public sphere.

Article

Donna E. Alvermann and William Terrell Wright

Naming is a curious practice. It entails rudiments, now mostly taken for granted, that serve to categorize everyday literacy practices across fields as diverse as cultural anthropology and the management of multiple Git profiles. As a term unto itself, adolescent literacies is not immune to the vagaries of naming. In fact, it serves as an excellent example of how commonly named concepts in education embed the field’s histories, debates, pedagogies, and policies writ large. Conceptualizing literacy in its plural form raised eyebrows among academics, researchers, practitioners, publishers, and indexers concerned with the noun–verb agreement in phrases such as “adolescent literacies is a subfield” of adolescence. For some, the very notion of literacy extending beyond reading and writing is still debatable. With each passing day, however, it becomes noticeably more evident that multimodal forms of communication—images, sounds, bodily performances, to name but a few ways of expressing oneself—are competing quite well in the marketplace of ideas that flow globally with or without a linguistic component attached to them. Aside from the naming process and its attendant political overtones, the practice of treating youth between roughly the ages of 12 and 17 as a monolithic group has been common in the United States. Largely traceable to a time in which developmental psychology dominated the field of literacy instruction (in the early to late 20th century), designating youth as adolescents equated to viewing them as some a normative group devoid of racial, class, gender, and any number of other identity markers. Even with the sociocultural turn in early 21st century and its abundance of studies reifying the socially constructed nature of adolescents, the term persists. Its adhesive-like attraction to literacies, however, may be weakening in light of research that points to youth who are agentic and dynamic game changers when it comes to participating in a world grown more attuned to the need for collaboration based not on hierarchical standing but instead on working through commonplace tensions too complex for any one solution.

Article

In the digital era, written communication for children and youth is changing. As texts and media include complex intersections of print, image, sound, and other modalities, the ways in which writing is conceived is shifting. The evolution and impact of digital technologies follow a long history of invention, innovation, and change in written communication, with critiques of writing and communication technologies present in both historical and contemporary contexts. A new development in contemporary digital culture is the significant and widespread participation of children and youth in digital media and communication due to the ubiquity, affordances, and appeal of mobile digital devices. In the history of writing instruction, pedagogical approaches and perspectives have continued to evolve, with the teaching of writing at times positioned as subordinate to the teaching of reading, a pattern that has repeated into the digital era in which an emphasis on digital writing production and text creation has been similarly less of a focus than receptive consumption of media. Shifts in digital practice and the emergence of new devices for writing present both challenges and opportunities for the teaching of writing and the creation of texts in schools, with issues of digital resource provision and access to technology presenting hurdles for some teachers. Teacher awareness of the digital worlds, practices, and “funds of knowledge” that students are bringing to the writing classroom is vital to reimagining the writing classroom within contemporary digital culture. In the 21st century, writing instruction needs to be inclusive of the operational demands of writing as well as sociocultural and critical requirements, in addition to responding to fluid technoliteracy contexts and consideration of how “writing” itself is changing.

Article

Leelo Keevallik

Communicative action can be carried out by not only verbal but also embodied means. People regularly use multimodal resources to make sense for each other. Consider a mundane activity, such as a greeting. In addition to the choice of lexical items to fit the relationship, such as “hi,” “yo,” or “good morning, Mister Smith,” extreme prosody featuring high pitch, increased loudness, and extensive lengthening on a “hi” may be necessary for your friend to feel recognized and truly appreciated. In some contexts and relationships, a handshake, a bow, or a hug may be mandatory, while the appropriate duration of those behaviors, the adequate spatial distance, and the exact positions of touch are culturally significant. Across activity settings, bodily behavior is regularly treated as meaningful by coparticipants, as it plays a role in action formation alongside the use of lexicon and grammar. Qualitatively different semiotic resources are juxtaposed so that they mutually elaborate each other and constitute actions within the locally emerging interactional sequences, as understood by the current participants. Aspects such as gaze, gesture, posture, objects, and movement are all potentially recruited to achieve social action, depending on the praxeological context. It is, for example, crucial to pay attention to a specific area in the surrounding space when someone does a pointing gesture or to adjust one’s pace when interacting on the go. We are held socially accountable for our embodied behavior, be it designed for others or not, and the body is constantly interpreted in regard to its action import. Social actions can furthermore be exclusively carried out by the bodies within their spatial, material, and praxeological settings.

Article

Linguistic and cultural shift are some of the most pressing issues facing minoritized speakers around the world. Language revitalization initiatives seek to increase the number of speakers through various pedagogical and social interventions. Language, however, is not simply a code transmitted between individuals, but comprised of a wealth of associated practices, norms, and forms of interaction in which that code has meaning. Multimodality is both an approach to the various communicative modes or semiotic fields of language, as well as a form of ethnographic practice related to media. Multimodality matters for the pedagogical methods, communicative modes, and media technologies involved in language revitalization. A multimodal approach to language revitalization includes modalities beyond a single communicative channel or form of media in recognition of the multifunctional and multidimensional nature of language.

Article

Carol A. Fowler

The theory of speech perception as direct derives from a general direct-realist account of perception. A realist stance on perception is that perceiving enables occupants of an ecological niche to know its component layouts, objects, animals, and events. “Direct” perception means that perceivers are in unmediated contact with their niche (mediated neither by internally generated representations of the environment nor by inferences made on the basis of fragmentary input to the perceptual systems). Direct perception is possible because energy arrays that have been causally structured by niche components and that are available to perceivers specify (i.e., stand in 1:1 relation to) components of the niche. Typically, perception is multi-modal; that is, perception of the environment depends on specifying information present in, or even spanning, multiple energy arrays. Applied to speech perception, the theory begins with the observation that speech perception involves the same perceptual systems that, in a direct-realist theory, enable direct perception of the environment. Most notably, the auditory system supports speech perception, but also the visual system, and sometimes other perceptual systems. Perception of language forms (consonants, vowels, word forms) can be direct if the forms lawfully cause specifying patterning in the energy arrays available to perceivers. In Articulatory Phonology, the primitive language forms (constituting consonants and vowels) are linguistically significant gestures of the vocal tract, which cause patterning in air and on the face. Descriptions are provided of informational patterning in acoustic and other energy arrays. Evidence is next reviewed that speech perceivers make use of acoustic and cross modal information about the phonetic gestures constituting consonants and vowels to perceive the gestures. Significant problems arise for the viability of a theory of direct perception of speech. One is the “inverse problem,” the difficulty of recovering vocal tract shapes or actions from acoustic input. Two other problems arise because speakers coarticulate when they speak. That is, they temporally overlap production of serially nearby consonants and vowels so that there are no discrete segments in the acoustic signal corresponding to the discrete consonants and vowels that talkers intend to convey (the “segmentation problem”), and there is massive context-sensitivity in acoustic (and optical and other modalities) patterning (the “invariance problem”). The present article suggests solutions to these problems. The article also reviews signatures of a direct mode of speech perception, including that perceivers use cross-modal speech information when it is available and exhibit various indications of perception-production linkages, such as rapid imitation and a disposition to converge in dialect with interlocutors. An underdeveloped domain within the theory concerns the very important role of longer- and shorter-term learning in speech perception. Infants develop language-specific modes of attention to acoustic speech signals (and optical information for speech), and adult listeners attune to novel dialects or foreign accents. Moreover, listeners make use of lexical knowledge and statistical properties of the language in speech perception. Some progress has been made in incorporating infant learning into a theory of direct perception of speech, but much less progress has been made in the other areas.

Article

Rhonda K. Reger and Paula A. Kincaid

Content analysis is to words (and other unstructured data) as statistics is to numbers (also called structured data)—an umbrella term encompassing a range of analytic techniques. Content analyses range from purely qualitative analyses, often used in grounded theorizing and case-based research to reduce interview data into theoretically meaningful categories, to highly quantitative analyses that use concept dictionaries to convert words and phrases into numerical tables for further quantitative analysis. Common specialized types of qualitative content analysis include methods associated with grounded theorizing, narrative analysis, discourse analysis, rhetorical analysis, semiotic analysis, interpretative phenomenological analysis, and conversation analysis. Major quantitative content analyses include dictionary-based approaches, topic modeling, and natural language processing. Though specific steps for specific types of content analysis vary, a prototypical content analysis requires eight steps beginning with defining coding units and ending with assessing the trustworthiness, reliability, and validity of the overall coding. Furthermore, while most content analysis evaluates textual data, some studies also analyze visual data such as gestures, videos and pictures, and verbal data such as tone. Content analysis has several advantages over other data collection and analysis methods. Content analysis provides a flexible set of tools that are suitable for many research questions where quantitative data are unavailable. Many forms of content analysis provide a replicable methodology to access individual and collective structures and processes. Moreover, content analysis of documents and videos that organizational actors produce in the normal course of their work provides unobtrusive ways to study sociocognitive concepts and processes in context, and thus avoids some of the most serious concerns associated with other commonly used methods. Content analysis requires significant researcher judgment such that inadvertent biasing of results is a common concern. On balance, content analysis is a promising activity for the rigorous exploration of many important but difficult-to-study issues that are not easily studied via other methods. For these reasons, content analysis is burgeoning in business and management research as researchers seek to study complex and subtle phenomena.

Article

Fiona Scott and Jackie Marsh

The study of digital literacies in early childhood (0–8 years) is an emergent and fast-growing area of scholarship. Young children’s communicative practices are today more complex and diverse in scope than ever before, encompassing both “traditional” reading and writing and a growing range of “new” communicative competencies across multiple digital media contexts. Scholars are increasingly interested in children’s literacy practices outside traditional print-based texts, and the theory of multimodality helps them to understand children’s communicative practices in relation to a range of modes, including those present in digital technology. At the same time, the boundaries between what constitutes “digital” and “traditional” literacies are themselves blurred. Multiple academic disciplines have contributed to our understanding of children’s digital literacy practices. Numerous definitions for digital literacy or literacies exist, and scholars have proposed a range of theoretical approaches to the topic. Bill Green’s “3D model” of literacy provides a useful starting point for understanding the different dimensions of children’s digital literacy: operational, cultural, and critical. It is acknowledged that children’s digital literacy practices are specific to particular social and cultural contexts. In particular, scholars have identified important differences between accepted literacy practices in schools and early years’ settings (“school literacies”) and children’s literacy practices in a socioculturally diverse range of home settings (“home literacies”). A growing field of research is explicitly concerned with the unique skills developed at home, as children learn to produce and interpret a range of “new” digital and multimodal texts. At the same time, numerous scholars have suggested that there is still a general lack of progress with regard to early years’ practitioners’ use of technology in the curriculum. Gaps and absences in knowledge still exist, and it will be important for scholars over the coming years to continue research into young children’s digital literacy practices, both in homes and communities and across early years’ settings.