Research and Assessment Methods

Survey has been a widely used data collection method for a variety of purposes in educational research. Although response styles have the potential to contaminate survey results, educational researchers often do little to control for such negative effects. Under discussion are five common response issues, their impact on survey data, and the methods that may be used to minimize the negative impact of these response issues on survey data. The five response issues in question are acquiescence (including disacquiescence), careless responding, extreme response, social desirability, and item-keying effect. Acquiescence (disacquiescence) refers to a respondent’s general tendency to agree (or disagree) with an item regardless of its content. This response style can distort item and construct correlations, compromising the results of factor analytic and correlational findings. Careless responding refers to a respondent’s tendency to pay insufficient attention to item content before responding, which can also lead to a biased estimation of relationships. Extreme response refers to the tendency of selecting extreme response options (e.g., strongly agree or strongly disagree) over middle options (e.g., neutral). Social desirability refers to a respondent’s tendency to rate him- or herself in an overly positive light. Finally, item-keying effect refers to a respondent’s differential responses to regular-keyed and reverse-keyed items. This effect often creates the illusion that items with opposite keying directions measure distinct constructs even when they may not.

A growing amount of research has been done on how to control for the negative impact of these response styles, although the research may be limited and uneven for different response issues. A variety of approaches and methods exist for handling these response issues in research practice. Different response issues may require considerations at different stages of research. For example, effective handling of acquiescence response may require steps in both survey construction (e.g., including a hidden measure of acquiescence) and survey data analytic treatment (partial correlation technique), while controlling for item-keying effect may require more sophisticated modeling techniques (e.g., multitrait-multimethod confirmatory factor analysis).

Arts-based research encompasses a range of research approaches and strategies that utilize one or more of the arts in investigation. Such approaches have evolved from understandings that life and experiences of the world are multifaceted, and that art offers ways of knowing the world that involve sensory perceptions and emotion as well as intellectual responses. Researchers have used arts for various stages of research. It may be to collect or create data, to interpret or analyze it, to present their findings, or some combination of these. Sometimes arts-based research is used to investigate art making or teaching in or through the arts. Sometimes it is used to explore issues in the wider social sciences. The field is a constantly evolving one, and researchers have evolved diverse ways of using the communicative and interpretative tools that processes with the arts allow. These include ways to initially bypass the need for verbal expression, to explore problems in physically embodied as well as discursive ways, to capture and express ambiguities, liminalities, and complexities, to collaborate in the refining of ideas, to transform audience perceptions, and to create surprise and engage audiences emotionally as well as critically. A common feature within the wide range of approaches is that they involve aesthetic responses.

The richness of the opportunities created by the use of arts in conducting and/or reporting research brings accompanying challenges. Among these are the political as well as the epistemological expectations placed on research, the need for audiences of research, and perhaps participants in research, to evolve ways of critically assessing the affect of as well as the information in presentations, the need to develop relevant and useful strategies for peer review of the research as well as the art, and the need to evolve ethical awareness that is consistent with the intentions and power of the arts.

Autoethnography is an increasingly popular form of postpositivist narrative inquiry that has recently begun to appear in educational contexts. The multiple lineages of autoethnography include the insider accounts of early anthropologists, literary approaches to life history and autobiography, responses to the ontological/epistemological challenges of postmodern philosophies, feminist and postcolonial insistence on including narratives of the marginalized, performance and communication scholarship, and the interest in personal stories of contemporary therapeutic and trauma cultures. Approaches vary widely from fragmented, experimental, performative, and multimodal texts through to realist tales. Advocates claim that autoethnography enables us to live more reflective, more meaningful, and more just lives.

The European Commission launched a renewed agenda for adult learning with the objective of ensuring access to high-quality educational opportunities to adult learners for the promotion of their personal and professional development. Thus, European researchers in this field are paying attention to lifelong learning actions in order to address this challenge. Studies in this area are exploring how adult education can strengthen adults’ skills, in particular those required in the current knowledge society (information and communication technologies, problem solving, foreign languages, etc.). Simultaneously, some investigations focus in depth on the role that adult education can play in overcoming social exclusion for the most underserved groups. This paper describes the contributions of these investigations as well as the steps carried out by programs and theories that have contributed the most to adult learning. Lastly, future developments and challenges on this field are explained.

Transnational flows of educational knowledge and research are fundamentally guided by the global geopolitics of knowledge—the historically constituted relations of power born out of the continuing legacy of modernity/coloniality. In the early nation-building stage of the 19th century, state-funded education was at the core of states’ pursuit for economic and social progress. Newly formed nation states actively sought new educational knowledge from countries considered more advanced in the global race toward modernity and industrialization. The transnational lesson drawing in education at the time was guided by the view of modernity as originating in and diffusing from the West. This created the unidirectional flow of educational influence from advanced economies of the West to the rest of the world. Central to the rise of modernity in Western state formation is the use of education as a technology of social regulations. Through the expansion of state-funded education, people were turned into the people, self-governing citizens, and then the population that was amenable to a state’s social and economic calculation and military deployment. But this development was embedded in the geopolitical context of the time, in which Western modernity was deeply entangled with its underside, coloniality in the rest of the world. Various uses of education as a social control were tested out first in colonial peripheries and then brought back to the imperial centers.

Today, the use of education for the modernist pursuit of perfecting society has been intensified through the constitution of the globalized education policy space. International organizations such as the World Bank and the Organisation for Economic Co-operation and Development (OECD) act as the nodes through which transnational networks of education policy actors are formed, where the power of statistics for social and educational progress is widely shared. Both developed and developing countries are increasingly incorporated into this shared epistemological space, albeit through different channels and due to different factors. The rise of international academic testing such as OECD’s Programme for International Student Assessment (PISA) has certainly changed the traditional pattern of education research and knowledge flows, and more lesson drawing from countries and regions outside the Anglo-European context is pursued. And yet, the challenges that PISA poses to the Eurocentric pattern of educational knowledge and research flows are curtailed by the persistence of the colonial legacy. This most clearly crystalizes in the dismissive and derogatory characterization of East Asian PISA high achievers in the recent PISA debate. Hence, the current globalization of education knowledge and research remains entangled with the active legacy of coloniality, the uneven global knowledge structure.

Interviews are frequently used in ethnographic research, but it is argued that they pose particular difficulties in interpretation. While ethnographers are interested in understanding how people construct and interpret cultures in their natural settings, interviews are based on rules that counteract most normal interactions. Thus interviews in ethnography can only be interpreted within the context of that wider ethnography and the data generated has to be tested against other data generated by different means and data generated in other interviews.

Although some ethnographers avoid the use of interviews, others use a range of different forms of interviews. It is argued that Basil Bernstein’s concepts of classification and framing can be used to clarify the range of forms and to highlight the potential relationships between the form of interview and class, gender, and ethnicity.

In the context of offenders who have learning difficulties, autism, and/or social, emotional, and mental health problems, their families, and professionals who work with them, caring and ethical research processes can be explored via fieldnotes. Conducting life story interviews and recording fieldnotes within qualitative criminological, education, and sociological research have long since been used to document and analyze communities and institutions and the private and public spheres. They richly tell us about specific research contexts, or everyday lives and relationships, that interview transcripts alone perhaps overlook. It is in the process of recording and reflecting upon research relationships that we can see and understand care-full research. But caring and ethical research works in an interdependent and relational way. Therefore, the participant and the researcher are at times vulnerable, and recognition of this is critical in considering meaningful and healthy research practices. However, the acknowledgment of the fact that particular types of research can be messy, chaotic, and emotional is necessary in understanding caring research.

The longitudinal study of teachers gives a time perspective on the life and work of teachers, instead of just a snapshot at a particular point. The time period in question may be just a few intense months, as in some ethnographic research, or several decades, as in some life-history research. Longitudinal research is useful in exploring such topics as how teachers change and grow over their careers, changes in teachers’ professional satisfaction over the years, patterns of teacher retention and drop-out, the impact of teachers on their students over time, and the influence of preservice and/or in-service teacher education on teachers.

Continuous study of the same teachers over many years is challenging and accordingly not common. It is typically expensive and time-consuming, and extends beyond the time span of most research funding; moreover, many participants either leave the profession or move to other locations, making it difficult to keep in touch with them. Accordingly, additional ways to do longitudinal research need to be found: for example, studying teachers intensively for a shorter period; asking teachers to recall earlier phases in their life and/or career; or studying different cohorts of teachers at various career points (as in the classic Huberman study and parts of the U.K. VITAE research). Each of these methods has limitations but maintains the valuable outcome of providing a time perspective.

Where it can be arranged, however, interviewing the same teachers at intervals over several years has the advantage of enabling researchers to get to know the participants well. As a result, the researchers are in a better position to understand what the participants are saying in the interviews, and assess the veracity of their self-reporting about their views and practices, past and present. Also, a degree of trust is established such that the teachers are more likely to be frank about their feelings, challenges, and concerns. But one danger of the emerging relationship is that the support the relationship it provides may positively impact the teachers’ experience (e.g., helping them fine-tune their practice and maintain their morale to an unusually high level). This limitation has to be weighed against the advantages in deciding whether or not to use this approach to the longitudinal study of teachers.

At core, moral and character education aims to develop the moral person. How this end state develops has been hindered by interest from different theoretical positions, differences between practitioners and theoreticians, different assumptions about how far character is educable, and associated measurement problems. Traditionally, moral education is concerned with the interpretation and strategies one uses to understand moral phenomenon and defines the moral person as a predominantly thinking entity, whereas character education emphasizes the development of habits and dispositions as a precondition for the moral person. Current interest is in finding commonalities across these traditions towards the achievement of human flourishing. These points of intersection have often been overlooked, but current work is demonstrating the importance of interdisciplinary and multidisciplinary approaches for practitioners, researchers and policymakers.

Marginalized populations are by definition composed of people who have fewer possibilities and options in their lives than those studying them. This fact has to be reflected before, during, and after the research itself. There are many facets of this basic assumption. One of them is, how are marginalized perceived by the researcher? Are they helpless victims, or people who are able to tell their own stories? Another relevant detail is the personality of the researcher. When the researcher comes from outside the marginalized group, the key question is, which methodology can be best applied to give a voice to those who are marginalized? On the other hand, when the researcher is a member of the group being studied, the key question is how to achieve the distance necessary for analysis. There could be many more such relevant facets, but the quality of the final research product is partially determined by any number of decisions that are made during the planning of the research and the conducting of the research.

All of these decisions have methodological consequences. There are a wide range of qualitative research approaches, such as participatory research, autoethnographic research, narrative and biographical research, or traditional qualitative research based on interviews with representatives of marginalized groups. In the early 21st century, there has been a shift away from a top-down, outsider perspective that sees the marginalized as helpless victims and toward more participatory research designs that promote and give a space to the marginalized voice.

The common denominator of all these decisions is whose voice is being heard—does it belong to the marginalized group or to the outside world? Is it possible to overcome the boundaries between these two worlds? And what role does methodology play in this story?

Emerging in the learning sciences field in the early 1990s, qualitative design-based research (DBR) is a relatively new methodological approach to social science and education research. As its name implies, DBR is focused on the design of educational innovations, and the testing of these innovations in the complex and interconnected venue of naturalistic settings. As such, DBR is an explicitly interventionist approach to conducting research, situating the researcher as a part of the complex ecology in which learning and educational innovation takes place.

With this in mind, DBR is distinct from more traditional methodologies, including laboratory experiments, ethnographic research, and large-scale implementation. Rather, the goal of DBR is not to prove the merits of any particular intervention, or to reflect passively on a context in which learning occurs, but to examine the practical application of theories of learning themselves in specific, situated contexts. By designing purposeful, naturalistic, and sustainable educational ecologies, researchers can test, extend, or modify their theories and innovations based on their pragmatic viability. This process offers the prospect of generating theory-developing, contextualized knowledge claims that can complement the claims produced by other forms of research.

Because of this interventionist, naturalistic stance, DBR has also been the subject of ongoing debate concerning the rigor of its methodology. In many ways, these debates obscure the varied ways DBR has been practiced, the varied types of questions being asked, and the theoretical breadth of researchers who practice DBR. With this in mind, DBR research may involve a diverse range of methods as researchers from a variety of intellectual traditions within the learning sciences and education research design pragmatic innovations based on their theories of learning, and document these complex ecologies using the methodologies and tools most applicable to their questions, focuses, and academic communities.

DBR has gained increasing interest in recent years. While it remains a popular methodology for developmental and cognitive learning scientists seeking to explore theory in naturalistic settings, it has also grown in importance to cultural psychology and cultural studies researchers as a methodological approach that aligns in important ways with the participatory commitments of liberatory research. As such, internal tension within the DBR field has also emerged. Yet, though approaches vary, and have distinct genealogies and commitments, DBR might be seen as the broad methodological genre in which Change Laboratory, design-based implementation research (DBIR), social design-based experiments (SDBE), participatory design research (PDR), and research-practice partnerships might be categorized. These critically oriented iterations of DBR have important implications for educational research and educational innovation in historically marginalized settings and the Global South.