Definition

Originally developed for use with written texts, quantitative content analysis (QCA) aims to distill the many words presented in a text into meaningful categories. These categories can then be treated as variables, allowing for a descriptive interpretation of the texts, or functioning as variables in statistical analyses. QCA has expanded beyond the written word to many other types of media, but the basic principal of classifying larger content into smaller categories remains at the core of the method. Through analysis of how these categories inter-relate with each other and intersect with the broader cultural context, the goal is to discover how materials communicate meaning and what meaning is communicated.

Basic Methodology of QCA

A coding schema is central to the method. As with sampling, the development of a coding schema is driven by precedents in the research or theoretical literature, and by the teams’ research questions. Weber (1990) outlines several basic steps in the coding process. The first step is to define recording units, that is, whether coders should attend to certain words, phrases, images, or overarching themes of a passage or piece. Depending on research questions, recording units often are some combination of these or other units. For example, in a television show, researchers may want to know what words were in the dialogue and also the overarching theme of the conversation between characters.

The second step in developing a coding schema is to define categories. One may think of these as boxes to be marked on a rubric, even if computer software is aiding in the coding process. The categories may be defined narrowly or broadly. For example, the presence of “violence” between two characters may be narrowly defined by physical contact or broadly defined to include abusive language. The categories may be mutually exclusive, or an incident may be coded into multiple categories. This decision may be influenced by the analytic intent; basic statistical assumptions are violated when categories are not exclusive.

Once a preliminary rubric is established, the third step is a pre-test, in which a subsample is coded. The pre-test process should reveal where categories may need further refinement or where coding rules require additional clarity. The pre-test also produces information regarding the accuracy or correctness with which human coders or computerized coding programs are classifying the text. If accuracy is low, the coding rules should be revised. Step four, then, is revising the coding schema. The fifth step is to pre-test again. This process should continue until the coding process, whether human- or computer-coded, yields an adequate level of accuracy.

The sixth step is to code the full sample using the established schema. Following data collection, the final step of the coding process is checking the achieved accuracy of the human coders or the computer program. Individual human coders may fatigue over time, thus making more mistakes, or their interpretation of categories may shift slightly over time, resulting in misclassification. Computerized output should be reviewed to confirm whether code rules were applied correctly. During the process, for example, the program may encounter text combinations not anticipated by the programmers or not present in the pre-test, resulting in misclassification.

As with other forms of measurement, issues of reliability and validity may emerge in QCA (Neuendorf, 2002). Particularly with several human coders, a primary issue is inter-rater reliability. In brief, inter-rater reliability is the extent to which different people code the same text in the same way. Differences, for example, may occur when coding rules or categories are not clear, or when there are cognitive differences across coders. The pre-test process and adequate coder training may reduce these differences, but inter-rater reliability should also be assessed at the end of coding. Various statistical tests exist to assess inter-rater reliability.

Validity can broadly be divided into internal and external. Internal validity refers to the match between concepts and their operationalized definitions in variables. There is no parallel statistical metric to assess internal validity, but there are several dimensions that researchers may consider. External validity, in contrast, refers to the generalizability and replicability of the results generated by a measure. Breadth and representativeness of the sample improve generalizability, while a full accounting of the procedures of the coding and variable creation improve replicability.

Analysis and Interpretation in QCA

Analyzing data generated by the coding process can take many forms. Again, the analysis of the data is driven by existing theory and the established research questions. Once data are collected, however, researchers fully quantify the data by creating variables from the coded data that are most meaningful for the hypotheses they want to explore and the analyses they want to conduct. These analyses may range from completely descriptive in nature to mean-difference or correlational tests to multivariate regression models.

In interpreting the analytical results, researchers bring the accumulated evidence to bear on the research questions, determining what story their results tell about the texts and their content. Regardless of the analytical technique used in QCA, any interpretation of quantified content must be corroborated by reference to the original texts. That is, researchers should compare their interpretation of the data to a subsample of their source documents. If the story of the data analysis reasonably represents the story within the documents, the interpretations of analytical results are not just products of classification schemes or statistical techniques. If the stories do not match, then reconsideration of the analysis or interpretation is necessary. In short, although QCA aims to quantify what could be considered qualitative information, it nonetheless retains a portion of qualitative art in the final interpretation of the data.

Example of QCA

One example of QCA is Britto and Dabney’s (2010) analysis of justice issues on political talk shows, which illustrates the coding and analysis process. Britto and Dabney were interested in crime content on these programs and, in particular, how the content was politicized. They selected the central primetime talk show across each of the three major cable news networks. Over the course of six months, they randomly selected one day per week to record the shows. They chose this approach for two reasons: to avoid one particular news story dominating conversation (e.g., a high-profile crime) and to ensure they were watching a “typical” example of the show. Coding was performed at two levels of analysis. At the program level, the schema included numeric coding categories for the number of segments in an episode, speaking time given to guests, and racial/ethnic and gender characteristics of offenders and victims in crime stories. At the individual level of analysis, the schema included codes for guest characteristics and guests’ interactions with hosts. These categories became variables in the analyses. Coders received four hours of training, which included a discussion of how concepts were operationalized, a practice coding session, and the follow-up discussion. During data collection, coders were instructed to watch their assigned episodes at least twice in order to code at each analytic level separately.

Britto and Dabney’s statistical analysis of their enumerated data began with a presentation of guest profiles, comparing population numbers to the demographic characteristics of guests, and then comparing general show guests to guests in justice-related segments. The analysis continued with a description of the amount of justice-related content on each show, then compared shows to each other. To address their major research questions, the analysis examined the interactions of hosts with their guests, using chi-square tests to reveal differences across guests’ political persuasions. The analysis also presented ratios of offender and victim characteristics, comparing these ratios across shows and to official United States crime data. Britto and Dabney used these statistical presentations as evidence in answering their research questions; however, they also corroborated their results by presenting qualitative descriptions of each of the source programs.

Other examples of QCA include Welch, Weber, and Edwards’ (2000) analysis of corrections debates in the New York Times; Cavender, Bond-Maupin, and Jurik’s (1999) analysis of gender in reality crime television; and Chiricos and Eschholz’s (2002) analysis of crime and race/ethnicity in local television news.

Definition

The quantitative approach to content analysis, while useful, may result in removing the coded content from the context surrounding it (Muzzatti, 2006). For example, a “violent action” in a television program may be counted, but the scene or setting in which it occurred, the offender’s motive for the violent act, the victim’s reaction, and other visual or auditory detail are lost. To address the shortcomings of QCA, Altheide (1987) proposed for ethnographic content analysis (ECA).

Like QCA, the goal of ECA is to discover how materials communicate meaning; however, in ECA, meaning is assumed to be present in various modalities, such as text, format, visual and auditory style, and in the positioning of one piece of information among others. Although this approach may involve some degree of enumeration, the emphasis is on descriptive and conceptual data, similar to what one might record in an observational study. The procedures for data collection, analysis, and interpretation are designed to be reflexive, with the researcher constantly engaged in the material, comparing observations as the process unfolds, and attentive to conceptual and theoretical nuances as they arise.

Basic Methodology in ECA

The basic methodology of ECA is outlined by Altheide and Schneider (2013). A review of the literature should guide the selection of the specific problem or issue to examine, as well as the medium to examine. Once researchers decide on a medium, they should learn about its production process and context. For example, if researchers are interested in studying comic books, they should become familiar with how comics are created, developed, and disseminated. Next, researchers should become familiar with several key examples to understand elements of formatting or general patterns in how the text and imagery are presented. Together, an understanding of the production process and the patterning within the source material provide essential background for creating a data collection protocol.

Constructing an ECA protocol is similar to developing a coding schema in QCA in the sense that the protocol is developed, tested, and re-tested. The format of an ECA protocol, however, is oriented conceptually rather than categorically. A protocol is designed as a means by which to query the information source—to ask questions, capture definitions, understand meanings, and reveal processes. Theoretically derived concepts, drawn from the research literature and from evaluation of key examples, guide the creation of a preliminary protocol. The protocol should have appropriate preset coding categories, (e.g., citation, date, length of text), but most of the conceptual categories should be open-ended for the researcher to input pertinent information, whether it be a count, a quotation, or a narrative description. During protocol development, new conceptual classifications are expected to emerge; as a result, original categories may be modified, and new categories may be incorporated into the protocol.

Data collection using the established protocol entails populating the categories with a wealth of descriptive examples. Throughout data collection, the researcher remains attuned to the process, such as including additional notes about emergent themes or observations about how pieces in the sample connect to each other. Indeed, Altheide and Schneider (2013) recommend a midpoint analysis of the gathered data in order to detect emergent themes or interaction among concepts, which may lead to refinement of the data collection process. Previously collected data may need recoded in light of these refinements; newly collected data can proceed under a revised protocol.

Issues of reliability in ECA are approached differently than QCA. The open-ended format of an ECA protocol precludes the use of standard metrics to assess inter-rater reliability, but efforts can be taken to ensure a level of agreement among coders. For example, as part of the training process, coders should also become familiar with the production process of the medium under study, as well as with the patterns in key examples. Familiarity with the sources facilitates more detailed coding. Research teams employing ECA can promote consistency in coding their observations by coding the same sources independently, then meeting to discuss meanings of concepts and how they are interpreted, as well as recurrent and emergent themes. Coder agreement may be achieved by working together to recode a source. Persistent disagreement may be indicative of another dimension that needs to be in a revised protocol. In general, in ECA, individual consistency in coding is more critical than inter-rater reliability, given the priority placed on reflexivity during data collection.

Analysis and Interpretation in ECA

Data analysis in ECA, like data collection, is a reflexive process. Notes are carefully read, re-read, compared to other notes, sorted, and read again. The early stages of analysis allow the researcher to explore, describe, explain, and perhaps theoretically link elements of the data. In addition, researchers attend to differences within the various conceptual categories. As analysis continues, researchers take notes on their notes, writing small summaries of overarching themes, concepts, and divergent ideas. Summaries include supporting details, such as descriptions of or quotes from the data sources. These smaller summaries are then combined, identifying typical cases and exceptions to those cases, and documenting unexpected elements that push the data in intriguing directions.

Analytical interpretation extends from this process. The small summaries are reviewed in light of the data collection protocol, by which the researcher can return to the original questions of interest. Referring to the protocol allows the researcher to sort the summaries into more distinctive categories to answer those questions, as well as to determine what does not fit and why it does not fit. There may be variation in the nature of the original documents, the concept may have been an unanticipated but important part of the story, or it may suggest a direction for future research. Like QCA, the results of evidence accumulated in ECA are brought to bear on the research questions. ECA, however, is designed not only to tell the story presented in the data but also to reveal, discuss, and contextualize the manifest and latent meaning of that story by grounding it in the social world and the broader social processes by which meaning is produced.

Example of ECA

Welsh, Fleming, and Dowler (2011) conducted an ECA of crime movies to investigate how crime and victimization were constructed in film, and to discover the messages presented about justice modalities. Given their intention to uncover these themes and messages, they outlined four questions, suggested by the research literature. To select a sample from the population of American films made from 1930–2009, they performed crime and justice-related keyword searches using the movie reference site imdb.com. From this first set of results, they then employed a theoretical sampling approach, selecting films based on two criteria, derived from their research questions, regarding plot narratives of the films. To collect data, each of the 30 films in the final sample was watched twice. The research questions served as the basis for an open-ended protocol. Coding entailed taking detailed notes about characters’ interactions and about the films’ narratives by observing both dialogue and visual imagery.

Welsh and his coauthors analyzed their data using a constructivist grounded theory approach, which exemplifies the reflexive nature of the ECA process: details from the protocol were read repeatedly and comparatively, with analysis drawing on existing theory. Their analysis revealed three primary themes across the films. In interpreting these three themes, the authors identified several sub-themes, as well as points of contrast. Supporting evidence for each theme is presented as a mix of narrative summary and discussion by the authors and as strategically placed quotations from the films. Where appropriate, connections were made to existing theory, either as points of comparison or as extensions to earlier work. Thus, Welch and his coauthors offered answers to their research questions, supporting their interpretations with descriptive evidence from their sources and with reference to prior literature.

Other examples of ECA include: Altheide and Michalowski’s (1999) analysis of fear discourse in newspapers; Kort-Butler’s (2013) study of crime and justice representations in superhero cartoons; and Myers’ (2012) analysis of televised reports on juvenile detention using program transcripts.