Methods of Foreign Policy Analysis
- Philip B.K. PotterPhilip B.K. PotterWoodrow Wilson Department of Politics, University of Virginia
Foreign policy analysis (FPA) is the study of how states, or the individuals that lead them, make foreign policy, execute foreign policy, and react to the foreign policies of other states. This topical breadth results in a subfield that encompasses a variety of questions and levels of analysis, and a correspondingly diverse set of methodological approaches. There are four methods which have become central in foreign policy analysis: archival research, content analysis, interviews, and focus groups. The first major phase of FPA research is termed “comparative foreign policy.” Proponents of comparative foreign policy sought to achieve comprehensive theories of foreign policy behavior through quantitative analysis of “events” data. An important strand of this behavioral work addressed the relationship between trade dependence and foreign policy compliance. On the other hand, second-generation FPA methodology largely abandoned universalized theory-building in favor of historical methods and qualitative analysis. Second-generation FPA researchers place particular emphasis on developing case study methodologies driven by social science principles. Meanwhile, the third-generation of FPA scholarship combines innovative quantitative and qualitative methods. Several methods of foreign policy analysis used by third-generation FPA researchers include computer assisted coding, experiments, simulation, surveys, network analysis, and prediction markets. Ultimately, additional attention should be given to determining the degree to which current methods of foreign policy analysis allow predictive or prescriptive conclusions. FPA scholars should also focus more in reengaging foreign policy analysis with the core of international relations research.
The periodic reassessment of research methods is important to the vitality of any academic discipline, but it has particular salience for a relatively young field such as foreign policy analysis (FPA). Hudson and Vore (1995:221) acknowledge as much in their review of the FPA literature, noting that, “in the study of foreign policy decision-making, the issues are not theoretical but methodological.” I define foreign policy analysis as the study of how states, or the individuals that lead them, make foreign policy, execute foreign policy, and react to the foreign policies of other states. This topical breadth results in a subfield that encompasses a variety of questions and levels of analysis, and a correspondingly diverse set of methodological approaches. This essay surveys FPA’s methodological development from its inception to the present and, in the process, outlines the body of existing methodological practice and identifies opportunities for future progress. The objective is to provide both an indication of the role that various quantitative and qualitative methods play in the FPA literature and an entryway for contemporary researchers seeking to apply these approaches to future work. Where appropriate, the reader is directed to more specific guides to the intricacies and execution of each method.
For the sake of organizational clarity, this review follows a stylized format roughly based on Neack, Hey, and Haney’s (1995) concept of “generational change” in foreign policy analysis. The section that immediately follows is partially archeological, that is, it surveys methods of events data analysis that were important to the early development of FPA, but in some cases have fallen out of widespread usage. The second section, which surveys qualitative methods, most closely reflects the current state of the art in the discipline. The third and final section addresses both cutting-edge and underutilized approaches.
The Methodological Origins of Foreign Policy Analysis
The unique historical context and intellectual environment of the early 1950s – specifically, the Cold War and the behavioral revolution – crucially shaped the early methodological development of foreign policy analysis. These origins have proven central to the methodological arc of the sub-discipline.
FPA was born of the opportunities presented by the largely atheoretical nature of historically oriented diplomatic analysis and the exclusion of political leadership and decision-making from the prevailing theories espoused by mainline international relations. Prior to the advent of FPA as a distinct subfield, the study of foreign policy relied on traditional methods and had long been the domain of political historians and diplomatic strategists in the tradition of thinkers such as Thucydides and Machiavelli. Early FPA researchers saw this longstanding tradition as part of their heritage, but, inspired by the methodological imperatives of the behavioral revolution, believed that systematizing the study of foreign policy would lead to progress in the form of generalizable and cumulative findings. Thus, from its inception, FPA was an explicitly theoretical exercise aimed at uncovering the systematic elements of foreign policy interactions, and the methods deployed reflected this.
Simultaneously, in response to the near monopoly of system-level theory in international relations, the pioneers of FPA argued that individual leaders or groups of decision makers are often the primary drivers of outcomes in international interactions (Snyder et al. 1954). Thus, at the very core of FPA’s intellectual identity lies a revisionist methodology (vis-à-vis diplomatic history) applied to a revisionist conception of the basic unit of analysis (vis-à-vis mainline international relations).
The strategic environment, specifically the position of the US in the early Cold War, also figured prominently in the early development of FPA methods. In the face of this protracted geopolitical conflict, American political leaders became unusually involved in the FPA academic endeavor. The promise of concrete conclusions and general enthusiasm for “scientific” approaches to political problems that stemmed from the success of the Manhattan Project led the US government to invest large sums in early FPA efforts. With funding came the expansion of major research centers such as the Rand Corporation and the Brookings Institution that were instrumental to the maturation of FPA as a subfield and methodological approach in international relations. However, the money and attention from the policy community came with strings attached – most notably, an expectation for immediately relevant research. Over time this requirement became increasingly difficult to reconcile with the relatively high uncertainty surrounding quantitative estimates of foreign policy phenomenon.
The first major phase of FPA research that emerged from this crucible is termed “comparative foreign policy.” Proponents of comparative foreign policy argued that controlled comparison of the domestic sources of external conduct across different countries could produce comprehensive theories of foreign policy behavior. Methodologically speaking, these scholars sought to achieve these ends primarily through quantitative analysis of “events” data, which I describe in detail in the section that follows. However, this transition to quantitative analysis was, at least in part, a refinement of even earlier attempts to develop a more robust understanding of the foreign-policy decision making process. Snyder, Bruck, and Sapin’s (1954) classic essay was arguably the first to encourage international relations scholars to reopen the “black box” of the state in order to study the actions of individual leaders. A significant body of early qualitative case study research flowed from this call to arms. To take just two examples, Paige (1968) took a decision making approach to understanding the origins of the Korean War, while Allison (1971) followed along similar lines with his well-known study of the Cuban Missile Crisis.
The decision making school provided a useful groundwork, particularly by identifying the leader as a crucial unit of analysis, a tradition that has persisted in FPA ever since. However, the developments of the behavioral revolution eventually overtook the primarily qualitative methods of these early FPA scholars. An increasing premium was placed on the generalizability garnered by operationalizing foreign policy interactions numerically and analyzing them quantitatively. This transition gave rise to the comparative foreign policy literature, which maintained an emphasis on decision making and scientific analysis, but moved away from case study analysis in favor of events data.
Comparative Foreign Policy and Events Data Analysis
The demand for foreign policy research that was scientific, generalizable, and policy relevant caught nascent foreign policy analysts unprepared. Where other areas of political science could respond to the challenge presented by the behavioral revolution with numerical data already at their disposal, the traditional fodder for diplomatic analysis – histories, documents, interviews, biographies, and memoirs – were less easily reduced into the sort of data necessary for rigorous, quantitative hypothesis testing.
This reality set foreign policy analysis somewhat behind other areas of political science because it had to overcome two distinct obstacles. First, new data had to be collected that was better suited to statistical analyses. Second, methods had to be developed with which to analyze these data within a behavioral framework. Among others, Rosenau (1966; 1968), McClelland (1970), and Brecher et al. (1969) took up these early challenges.
These early foreign policy analysts sought to develop a quantifiable unit of foreign policy interaction. McClelland conceived of this core unit of analysis as the foreign policy “event,” which is simply a formalized observation of a conflictual or cooperative interaction between states. McClelland’s intention was to fill the gap between the traditional narrative approach to foreign policy analysis and empirical techniques that relied upon discrete quantifiable data that could be explored in statistical analyses (Schrodt 1994). In effect, the foreign policy event takes a qualitative observation of foreign policy interaction and reduces it to a numerical or categorical form suited for statistical analysis.
The process of generating events data was and is time-consuming and costly. It is most commonly accomplished through the content analysis of thousands of newspaper reports on the interactions among nations in light of a previously defined set of criteria or codebook. Each observation uncovered in this way is then assigned some numerical score or a categorical code, which can then be analyzed quantitatively (Schrodt 1994). This potentially lengthy process requires that the researcher accomplish some or all of the following: identify sources, identify a period of analysis, create or borrow a coding scheme, train coders, generate the data, and check for reliability.
Foreign policy scholars have generated a significant number of important events datasets that remain central to quantitative methods of foreign policy analysis. The best of them are impressive collections offering decades-long periods of analysis, coverage of many countries (if not the entire international system) and standards of intercoder reliability well in excess of 80 percent (Burgess and Lawton 1972). The paragraphs that follow describe a subset of the available data. Particular attention is given to projects that were seminal to the methodological development of the field and those that generated datasets still widely used by contemporary scholars.
The World Event/Interaction Survey (WEIS)
The World Event/Interaction Survey Project began at the University of Southern California under the direction of Charles McClelland as a research project on the characteristics and processes of the international system (McClelland and Hoggard 1969). The initial WEIS dataset records the flow of action and response between countries (as well as non-governmental actors such as NATO and the United Nations) captured from a daily content analysis of the New York Times from January 1966 through December 1978. This reliance on the New York Times produces a well-known bias toward western perspectives, which was acknowledged from the outset by McClelland and his co-authors. However, they argued that by using a single source they were better able to remove the “noise” surrounding observations. Furthermore, the inclusion of non-state actors raises important methodological issues with regard to the basic unit of analysis. This question has taken on increased salience with the rise in concern about terrorist activities by non-state international entities.
The basic unit of analysis in the dataset is the interaction, which is simply a verbal or physical exchange between nations ranging from agreements to threats to military force. Each of these observations is coded to identify the actors, target, date, action category, and arena. The WEIS databank also provides brief qualitative textual descriptions of each event. These narratives provide context, which facilitates the process of identifying and understanding outliers and applying statistical findings back to political reality – both important for successful events analysis. The initial WEIS effort has been continuously updated and is presently current through 1993 (Tomlinson 1993). Other projects, such as the Kansas Event Data System, have applied WEIS coding rules to new research.
WEIS data has been widely used in the FPA literature, both by McClelland and his students and by outsiders who took advantage of these public domain data to test their own questions. The applications are diverse, underlining the versatility of well-designed events datasets. Several early examples are noted by Rummel (1979): Tanter (1974) used these data to understand the dynamics of the two major Berlin crises of the Cold War (1948–1949 and 1961); Kegley et al. (1974) explored patterns of international conflict and cooperation; while many others began the ongoing process of understanding the relationships among key contextual variables such as relative development, size, and political system, on international conflict, cooperation, and systemic stability (Rosenau 1974). Applications continue to this day. For example, Reuveny and Kang (1996b) utilized WEIS data in their exploration of causality in the relationship between international trade and conflict.
The Conflict and Peace Data Bank (COPDAB)
The COPDAB project was designed by Azar and colleagues (Azar 1980; 1982; Azar et al. 1972) as a longitudinal dataset of international and domestic events developed through content analysis of daily newspapers. In an advance over WEIS methods, COPDAB data is drawn from a wide variety of international and regional media outlets, thereby avoiding some potential bias issues.
COPDAB coders scored each event on a 16-point ordinal scale ranging from cooperative interactions to full-scale violence. The resulting dataset covers the interactions of 135 countries from 1948 to 1978 and can be analyzed at levels of aggregation ranging from the day to the year. Each record includes nine variables: date of event, actor initiating the event, target of the event, issue area(s), contextual information about the incident, and the source of the information about the event. The COPDAB dataset is particularly useful for those interested in the interactions between interstate and civil conflict and cooperation, as complementary datasets exist for both international and domestic events.
While the WEIS and COPDAB datasets are clearly conceptually related, scholars have disagreed about their compatibility (Howell 1983; Vincent 1983; Goldstein and Freeman 1990). The underlying definitions of conflict and cooperation are quite similar; however, coding differences introduce the potential for inconsistencies. Reuveny and Kang (1996a) explore this issue in detail with a series of statistical tests and time-series analyses. They argue that COPDAB and WEIS are indeed compatible for the overlapping period between 1966 and 1978. Building on this logic, they combine the WEIS and COPDAB series to create a larger events dataset covering the period from 1948 to 1993 that is potentially useful for scholars interested in working with a longer period of analysis.
International Crisis Behavior Project (ICB)
Although the final two projects outlined here (the International Crisis Behavior project and the Correlates of War [COW] project) are often excluded from discussions of foreign policy analysis, they are clearly a continuation of events research and are among the most frequently updated and widely used events datasets. The distinctive feature of the ICB and COW datasets is that they primarily focus on international conflict and therefore lack the range of conflictual and cooperative events that characterize the data projects discussed to this point. Researchers should note, however, that the ICB project does provide some indirect data on cooperation.
Michael Brecher and Jonathan Wilkenfeld launched the International Crisis Behavior project in 1975 with the goal of creating a comparative resource for those studying the concept of “international crisis.” There are two defining conditions for a crisis, which are built on work done by Azar (of the COPDAB project): “(1) a change in type and/or an increase in intensity of disruptive, that is, hostile verbal or physical, interactions between two or more states, with a heightened probability of military hostilities; that, in turn, (2) destabilizes their relationship and challenges the structure of an international system – global, dominant, or subsystem” (Brecher and Ben-Yehuda 1985).
The ICB project is congruent with many of the core concepts in FPA – for example, in the operationalization of key elements of decision maker perception. This is perhaps unsurprising, as many of the ICB’s primary researchers are steeped in the FPA tradition. To take one example, Michael Brecher’s (1974) book on Israeli foreign policy decisions, which pre-dates his work on the ICB project, is often cited as a seminal contribution to FPA that seeks to characterize a nation’s psychological and cultural environment as an access point to an understanding of its foreign policy.
As of January 2009, the core systemic dataset that results from this definition codes 452 incidents from the end of World War I through 2006 (version 9.0). Each crisis is coded for a number of variables, ranging from characterizations of decision maker perception to operationalizations of structural and environmental factors as well as crisis characteristics and outcomes.
The ICB project is unusual in that it proceeds simultaneously at multiple complementary levels. There are independent actor and system level datasets that allow the researcher to explore distinctions between systemic and national level explanations for crisis emergence and behavior. In addition, the project provides qualitative data in the form of a brief narrative description of each crisis, 9 in-depth volumes comprising 15 in-depth case studies; and 14 other unpublished studies. These serve as an aid to the researcher interested in adding additional nuance to statistical findings generated from quantitative analysis.
Correlates of War Project (COW)
Like the ICB project, the COW project does not attempt to capture multiple tiers of conflict and cooperation, but rather focuses on conflict. Two definitions were developed by the COW project in the 1980s, namely, a “militarized interstate dispute” (MID), and a “militarized interstate crisis” (MIC). The former is defined as: “[A] set of interactions between or among states involving threats to use military force, displays of military force, or actual uses of military force […] these acts must be explicit, overt, non-accidental, and government sanctioned” (Gochman and Maoz 1984). This “evolves into a militarized interstate crisis when a member of the interstate system on each side of the dispute indicates by its actions its willingness to go to war to defend its interests or to obtain its objectives.” These are steps two and three along a four-step ladder of growing belligerence, beginning with an “interstate dispute” and culminating in an “interstate war” (Leng and Singer 1988).
The majority of scholars currently working with COW events data use the MID dataset. The current version of the dataset contains 2331 militarized disputes from 1816 to 2001 coded for duration, outcome, and level of fatality. In addition, there are several complementary datasets on various metrics of international interaction (ranging from alliance to power to geography) that are associated with the broader COW project and can be easily mapped onto the MID dataset. This body of quantitative data is perhaps the most widely used at the present time – particularly among scholars interested in conflict.
Trade Dependence and Foreign Policy Compliance
An important strand of the behavioral work of the 1970s and 1980s addressed the relationship between trade dependence and foreign policy compliance. While this was far from the only research question to draw on quantitative data, the methodological challenges that confronted it were representative of those faced by quantitative FPA in general and are therefore worthy of some attention. Several scholars working in this area (e.g., Richardson and Kegley 1980; Moon 1983; 1985) argued that relatively smaller and weaker states adopt the foreign policies of their dominant trading partners. Thus, economic dependence severely constrains the independent decision making of leaders in states that are economically reliant on larger patrons. However, consensus on this conclusion was elusive, in large part because of how difficult it is to measure the two key concepts – dependence and compliance. The inevitable result was that discussion of the relationship became bogged down in issues of definition and operationalization. This is symptomatic of a larger issue in the quantitative study of foreign policy. Because the operationalization of the amorphous concepts in foreign policy necessitates discretion from the researcher, it is easy to critique the underlying assumptions that gave rise to the data, not to mention the model. Furthermore, if more than one scholar takes on a question in FPA, they typically settle on different operationalizations of the same underlying phenomenon. A high profile example of this trend can be found in the proliferation of events datasets on conflict and cooperation that has already been discussed. The unfortunate result is that many studies are not comparable or cumulative to the degree we find in the hard sciences.
Events Data – Methodological Challenges
Events data analysis poses a number of methodological challenges that should be taken into account by those analyzing foreign policy. The first of these issues relates to the very core of the events data endeavor – that is, the idea that foreign policy incidents can be reduced to a single quantifiable value. Despite the best efforts of the designers of the data projects described here, it remains difficult to effectively accomplish a cardinal or even ordinal ranking of disparate foreign policy events. However, many of the statistical approaches widely used in political science require cardinal level data, or at least data spaced at even thresholds. As a result, those seeking to generate statistical models of events data need to be particularly careful to apply methods that rely upon defensible assumptions about the nature of the underlying data.
Researchers should also be aware of methodological issues that may arise from the relative sparseness of positive observations in events data. The degree to which this is a problem depends on the type of model and the level of aggregation that is used, but if one considers the daily probability of a foreign policy event it is apparent that null observations would dominate the dataset. King and Zeng (1999; 2001) demonstrate that bias and inappropriately inflated statistical significance may arise in models of zero-inflated data. This is particularly problematic in instances where these null data contain no real information. There are several potential solutions to this problem should it arise. Tomz, King, and Zeng (1999) suggest a rare events correction for logistic analysis, which they have made available as an addition to the widely used STATA software. A less sophisticated check for rare events bias is to simply drop a random subset of null observations in order to confirm that findings derived from the remaining sample are consistent with the original result.
The non-independence of foreign policy events presents an additional methodological challenge. Non-independence simply means that positive foreign policy interactions tend to contribute to future positive interactions, while negative events are associated with subsequent negative events. At first appearance this might seem obvious, but this reality undercuts an assumption of independence that underpins most statistical models used in quantitative foreign policy analysis. Beck, Katz, and Tucker (1998) did much of the work that brought this problem to the attention of the discipline and they suggest a solution that entails generating a natural cubic spline with knots at the first and second derivative.
FPA scholars working with events data should also guard against selection bias (sometimes referred to as selection effects) when designing research, as inattention to this methodological challenge can significantly skew findings from both quantitative and qualitative tests. Selection bias typically arises from pre- or post-sampling that preferentially includes or excludes a particular type of observation from the sample that is subsequently used in testing. This is particularly easy to do when working with data on foreign policy because it is relatively easy to identify events, but difficult to tease out non-events. The trouble is that without an accurate characterization of non-events it is impossible to say anything about the causes or incidence of the events. To take one prominent recent example of this methodological challenge, Robert Pape’s recent work on the causes of suicide terror (2005) has come under fire for “sampling on the dependent variable” (Ashworth et al. 2008). Because Pape limits his sample to incidents of suicide terror, he effectively leaves out the instances in which such attacks did not occur. As a result, his research design prevents him from effectively speaking of when suicide terror does and does not occur.
Beyond issues related to the application of statistical methods to events data, there is an additional conceptual concern regarding the unit of analysis that should command attention from foreign policy researchers. Because FPA concerns the foreign policy of states, but sees this policy as emerging from the actions of individuals, traditional units of analysis are blurred. The foreign policy event is the result of the interaction and interplay between leaders, organizations, institutions, and states; however, many of the microfoundational theories that underpin the FPA endeavor are cast at the level of the individual decision maker. As a result, events analysis brings with it the nascent challenge of explaining how individual actions aggregate to the foreign policy actions of states. To put the issue more succinctly, while FPA theories distinguish themselves from mainline international relations by opening the black box of the state, the empirical data collected by scholars interested in events analysis typically returned to the state as the central unit of analysis.
There are also very practical concerns to bear in mind – simple tasks related to data manipulation remain some of the primary challenges confronting researchers interested in working with events data. It can be a nontrivial task to gather and combine data on foreign policy events with the various explanatory and control variables that are required for regression analysis. Researchers confronted with these difficulties should be aware of the EUGene software developed by Scott Bennett and Allan Stam (2000). EUGene is a basic data management tool that simplifies quantitative analysis of foreign policy interactions. The software offers several advantages. First, it allows for relatively easy transition between commonly used units of analysis – country–year, dyad–year, and directed dyad–year. Second, the software is capable of easily combining many of the events datasets discussed here with basic demographic and geopolitical data including data uploaded by the user.
Finally, there is the issue of collecting new events data. The substantial early investments in projects like WEIS and COPDAB were made at the high point of governmental and institutional enthusiasm for events datasets – both datasets were products of the National Science Foundation’s well funded Data Development for International Research (DDIR) project. However, DDIR funding and government and private support for events data collection projects in general declined markedly by the mid-1990s. While this decline had many causes, it was in part brought on by the difficulties that comparative foreign policy had delivering on its early promise. It proved far more challenging than expected to build policy relevant quantitative models with predictive capacity. The relative decline in interest on the part of traditional funding sources raises the issue of how new events data might be generated. Computer coding of electronically stored sources, which will be discussed in greater detail later in this essay, has emerged as one way to address this dilemma.
Qualitative Methods of Foreign Policy Analysis
The behavioral revolution and Cold War politics proved fertile ground for the emergence of FPA. However, the first major challenge for the young field also stemmed from this dual heritage. The problem was that these divergent intellectual pedigrees gave rise to methodological requirements that were at times mutually exclusive – on the one hand, an imperative from behavioralism for generalizability, and, on the other, a low tolerance for error on the part of Cold Warriors seeking to immediately inform policy with scientific findings. The emerging recognition of this tension and the seemingly unavoidable high error terms in quantitative models of foreign policy brought an end to the exuberance among academics and the US government for quantitative, events-driven foreign policy research. Policy makers backed away from direct involvement in the FPA endeavor, while academics tempered their commitment to events data and quantitative methods. What emerged was a second generation of FPA methodology, one that largely abandoned universalized theory-building in favor of historical methods and qualitative analysis (Neack et al. 1995).
The primary weapon in the arsenal of second-generation FPA researchers is the case study. However, this transition should not be viewed as a complete departure from that which came before it. Many of these scholars place particular emphasis on developing case study methodologies driven by social science principles, with the explicit goal of building techniques that provide intellectual rigor comparable to that of quantitative approaches. The result has been a robust discussion of the role and execution of qualitative methods.
It is admittedly artificial to divide methods of foreign policy analysis by “generation,” as this implies clean transitions that are in reality far more blurred. While the concept of generational change is useful for understanding the broad developments in the field, the reader should be aware that there are many exceptions to the general rule. Alongside second generation case studies were a wide range of quantitative approaches that, while often abandoning the drive toward universalized theory that characterized previous work in comparative foreign policy, stressed both the outputs and the outcomes of foreign policy processes and actions. Similarly, careful qualitative analysis of foreign-policy decision making has always been an element of foreign policy analysis, and therefore cannot only be considered to have followed sequentially on the quantitative work done in comparative foreign policy (although it did take on renewed prominence).
Case Study Analysis
There is no shortage of examples of the excellent use of case study methodology in foreign policy analysis. Graham Allison’s Essence of Decision (1971) is often cited as a seminal piece of research in this area with an innovative methodological approach. While Allison’s volume is concerned with a single incident – the Cuban missile crisis – the book is not a single case study, but rather three. Allison explored the US decision making process in the context of three competing explanatory theories: a rational actor model, an organizational process model, and a government politics model. Each of these three explanatory models receives independent analysis in a separate section of the book. Allison (1971:258) argues that these three models combine to provide a clear understanding of decision making in the context of the Cuban Missile Crisis: “Model I fixes the broader context, the larger national patterns, and the shared images. Within this context, Model II illuminates the organizational routines that produce the information, options, and action. Model III focuses in greater detail on the individuals who constitute a government and the politics and procedures by which their competing perceptions and preferences are combined.”
Another important strand of qualitative foreign policy analysis draws on work from political psychology to theoretically inform case study analysis of the foreign policy decision making process. These efforts began with “operational code analysis,” which involves determining how decision makers’ core beliefs shape their foreign policy reactions (George 1969; Holsti 1970). Operational codes include decision makers’ beliefs about the likelihood of violence, their ability to shape or prevent it, as well as leadership strategies and styles.
Robert Axelrod applies a related technique, termed cognitive mapping, to understand the influence of leadership beliefs on foreign policy interactions. Cognitive mapping entails defining a decision maker’s stated goals and then determining the causal linkages between these goals as a way of predicting likely behavior. Several applications of this technique can be found in an edited volume titled Structure of Decision (Axelrod 1976). A more recent application of cognitive mapping can be found in Johnston’s (1995) work on Chinese–American relations.
This early work developed into a substantial body of foreign policy analysis based more broadly on the psychology of decision makers, a method that figures prominently in analyses conducted at the individual level. Larson (1985) is a leading example of this sort of scholarship. In her book, Origins of Containment, she traces the path of Cold War politics in the context of the cognitive psychology of American policy makers.
A great deal of work has been done in recent years to improve and formalize case study methodology. One such volume, King, Keohane, and Verba’s Designing Social Inquiry (1994), has been influential (and controversial) enough that it is often referred to simply by the initials of its authors – KKV. King, Keohane, and Verba draw on their diverse methodological backgrounds to argue that the core logic of causal inference and control should apply as much to qualitative work as it does to quantitative research. They suggest that, by applying the logic of statistics, it is possible to produce theoretically robust and generalizable results while increasing certainty in the validity of qualitative findings.
Bennett and George’s (2005) more recent work on case study analysis has also emerged as an important contribution to the development of robust qualitative methods. This book lays out methods for designing case studies that are maximally useful for the formulation of policy, which remains a fundamental goal of foreign policy analysis. Bennett and George suggest greater emphasis on within-case analysis, process tracing, and theory building. While these suggestions differ markedly from those of KKV, the underlying goal is quite similar – to create scientific case studies from which lessons can be systematically drawn. In this sense, both volumes speak convincingly to the aforementioned tension between nuance and generalizability that plagues methods of foreign policy analysis.
This issue of generalizability has developed into the core methodological challenge surrounding case study analysis both in foreign policy analysis and in political science more generally. While systematic knowledge of foreign policy interactions does not necessarily require the numerical comparability that comes with quantitative research, some degree of generalizability remains important to the independent identity of foreign policy analysis, as it is this forward-looking element that separates the sub-discipline from diplomatic history. However, comparisons across cases are difficult for two reasons. First, case studies require such a depth of knowledge and investment of time that it is unusual for a scholar to accomplish more than a handful of them on any given question, though there are important exceptions (e.g., Brecher 2008). Second, the comparatively loose structure of case studies can hinder comparison, as many analyses fail to address the same subjects on the same terms. One way that these challenges can be overcome is through collaboration within a consistent framework.
An example of such collaboration can be found in a relatively recent volume edited by Beasley et al. (2002). The volume brings together qualitative work from 15 independent researchers systematically exploring the foreign policies of 13 states. Through the coordinating efforts of the editor, the volume maintains a degree of comparability across the cases while drawing on the deep knowledge of the individual contributors. As a result, the reader is able to engage in comparative analysis within a coherent theoretical framework, allowing for the quick identification of patterns and outliers. There are several examples of similarly structured volumes, and they indicate an important role for collaboration as an approach to boosting the sample sizes of qualitative analyses and thereby the generalizability of findings. The result is “comparative foreign policy,” but of a qualitative variety.
Another interesting solution to the issue of case comparability is found in the qualitative research that has emerged from the qualitative side of the International Crisis Behavior Project, which was already mentioned in the context of events data analysis. These case studies, though they were written over many years and appear in a variety of different outlets, follow a similar format and concern themselves with a consistent set of issues. As a result, they are an explicitly cumulative effort. With each new case study, the body of comparable knowledge increases and this expansion is accompanied by improvement in the robustness of findings.
Gathering Qualitative Data
Those interested in applying case study methodology will need raw material with which to build their analysis. For many questions, considerable ground can be covered using basic library research techniques and secondary sources. However, some of the most fruitful case studies (in terms of their contribution to the existing body of knowledge) bring new information to light. There are several methods of obtaining original qualitative data. The sections that follow will briefly discuss four methods that have become central in foreign policy analysis: archival research, content analysis, interviews, and focus groups.
Original source material can be a crucial element of a quality case study. Typically, scholars uncover such information through archival research. Relevant foreign policy materials are commonly found in the document collections housed in presidential libraries, national archives, and universities. While the basic concept behind archival research is self-explanatory, the actual process of gaining access to collections and navigating their contents can intimidate the neophyte. There are a number of guides to archival methods that can alleviate such anxiety. Directions for identifying and searching appropriate archival sources as well as tips for navigating the archives themselves can be found in Marc Trachtenberg’s (2006) recent volume on methods – Appendix II will be of particular interest to those seeking to utilize archival methods. Hill (1993), Larson (2001), and Lustick (1996) provide additional detail on the nuances of archival research.
Content analysis is a hybrid method that has played a longstanding and important role in quantitative and qualitative foreign policy analysis. The section of this essay on events data already discussed the ways in which content has been used to generate quantitative data for statistical analysis. For example, some of the earliest approaches to events data generation coded the content of elite communication (Winham 1969). However, more detailed content analyses have also been used to generate the raw material for case studies or other qualitative analyses. Ole Holsti (1969) was a pioneer of this method, while, more recently, Steve Walker and his students at Arizona State have developed a typology and quantitative content analysis scheme for operational code analysis (Walker et al. 1998). Those interested in additional detail on the mechanics of content analysis should consult Weber (1985), Neuendorf (2002), and West (2001).
Because the role of the individual figures so prominently in foreign policy analysis, interviews can be a particularly valuable method for accessing information about the mechanics of the decision making process. Interviews enable FPA scholars to delve deeply into the idiosyncrasies of the foreign policy process, gleaning deep insights from decision makers and those around them. Over time, FPA scholars have developed a robust set of interview methods designed to enable researchers to maximize the acquisition of information without introducing biases into findings.
There are a number of excellent examples of innovative interview methods in foreign policy analysis, which can serve as models for those interested in interview research. Prime among them are FPA classics such as Yuen Foong Khong’s Analogies at War (1992). Based on a series of interviews with senior officials (and archival research), Khong argues that leaders routinely reference the past when making foreign policy decisions and that this cognitive bias can profoundly alter decision making. Schoutlz (1987) does similar interview work in the context of US policy toward Latin America. More recently, Silber and Little (1995) draw on a series of interviews to uncover the foreign policy interactions at play in the collapse of the former Yugoslavia. Berg (2001), Brenner et al. (1985), McCracken (1988), Mishler (1986), and Seidman (1998) provide useful, in-depth tutorials on interview methods.
Focus group research is a derivative of interview methodology in which the researcher attempts to facilitate an organized discussion among the participants. In foreign policy analysis this typically takes the form of a meeting of experts in a particular foreign policy area, or participants in a prior foreign policy decision. Focus group methods can be particularly informative because the emerging consensus that comes from such discussions pools the knowledge of the participating individuals and therefore can overcome some of the potential biases of recollection and self-inflation that accompany individual interviews. However, concerns arise as well, due to some of the very same pathologies that FPA scholars have identified in the context of group decision making. For example, Janis’s (1972) concept of groupthink can take hold in such settings, with focus group members avoiding controversy and settling instead on a comfortable consensus, even if this consensus is out of step with reality. Along similar lines, the value of elite focus groups can suffer due to deference to higher-ranking participants and domination of the discussion by more talkative individuals who might overshadow important contributions by those less inclined to assert themselves (Krueger 2000).
Third Generation Methods of Foreign Policy Analyses
Neack, Hey, and Haney’s (1995) concept of generational change, to which this review has adhered thus far, captures only part of the methodological richness of FPA. There have long been methods of foreign policy analysis that fall outside this strict quantitative/qualitative divide, and there has been considerable recent growth in these alternative methods. Meanwhile, the distinction between quantitative and qualitative approaches to FPA has become increasingly blurred as the relative advantages of each approach have become more widely recognized. These events auger the arrival of a third generation of FPA scholarship that combines innovative quantitative and qualitative methods, thereby bridging the internal contradictions that split the second wave from the first and unifying a variety of methods of foreign policy analysis.
Several methods of foreign policy analysis are available to aspiring third generation foreign policy analysts seeking to move beyond events data and case studies including: computer assisted coding, experiments, simulation, surveys, network analysis, and prediction markets. The sections that follow will briefly introduce each of these methods, though the list is by no means exhaustive.
Computer assisted coding of electronically stored information offers several advantages and represents an important methodological innovation that is likely to play an increasingly significant role in the future of foreign policy analysis. First, machine coding can be more reliable than human coding simply because it removes the possibility of individual error and the resulting questions of intercoder reliability. Second, machine coding is extremely rapid. Where earlier events datasets were generated over periods of many years, computers can sift through huge quantities of data in minutes. The result is that machine coding greatly reduces the cost of events data generation – effectively bringing control over such data to the masses (Gerner et al. 1994; Schrodt and Gerner, 1994). However, the obvious benefits of machine coding are accompanied by two important caveats: the initial programming that creates the coding rules must be accurate, and the raw data must exist in a machine readable format (Gerner et al. 1994). Advocates of human coding often counter that the low cost and speed of machine coding are accomplished at the expense of accuracy and nuance.
At present, the best example of a machine coded events project is the Kansas Event Data System (KEDS). This project is among the most active events datasets, due in part to the relatively low cost and speed of generating data in this manner. KEDS provides a computer program that enables users to specify and create personalized events datasets with a variety of output options. The researchers on the KEDS team use this software to code news reports and generate political event data focusing on the Middle East, the Balkans, and West Africa; however, this approach can be extended to other regions or the international system as a whole.
The machine coding community, including members of the KEDS project, is particularly interested in predictive models built on the unique capacities of this technology (Schrodt, 1979; 1994; Gerner et al. 1994; Schrodt and Gerner 2000; Shellman forthcoming). Machine coding not only partially circumvents the need for large financial investments in events data by reducing the required labor and time, but also has the potential to address some of the concerns about the lack of predictive capacity that caused the decline in external funding in the first place. Because machine coding concentrates the researcher’s effort on developing decision rules rather than on the coding itself, once underway these programs can generate empirical data in real time. Such models that draw on continuously updated data effectively serve as early warning systems capable of identifying when political phenomena of interest are likely to occur. For example, Shellman and Stewart use machine coding to predict incidents of forced migration, which they applied with some success in Haiti (Shellman and Stewart 2007). This particular application of events data remains at the cutting edge of the FPA literature and will likely continue to be a productive avenue for future research.
Experiments and Simulation in Foreign Policy Analysis
Like all social science, foreign policy analysis struggles methodologically with the issues of control and causality. The quantitative and qualitative methods already discussed took hold in foreign policy analysis in part because the gold standard of the scientific method – experimental control – is typically off limits either for practical or for ethical reasons. However, with careful attention to design and feasibility, there are applications for experimental methods in the study of foreign policy, and where there are not, researchers have begun to turn their attention to simulation, which can achieve some of the same objectives. To take one recent example, Christensen and Redd (2004) examine how the context of foreign-policy decision making affects choice and assess this relationship in a controlled experiment conducted on undergraduates. They find that, at least in this context, the way in which information is presented directly affects the decision maker’s evaluation.
In recent years the nuts and bolts of experimental methods have drawn increasing attention. Along these lines, McDermott (2002) provides an interesting discussion of the origins and practice of experimental methods in political science, as well as the unique challenges this approach presents. One such challenge that should be considered carefully by those designing experiments meant to speak to foreign policy behavior is the trade-off between internal and external validity in experiments. Internal validity indicates that the proposed relationship between the independent and dependent variables is the true causal one. When such validity is high it means that extraneous variables and alternative explanations have been ruled out. While typically very difficult to achieve in the social sciences, high internal validity results from proper randomization in an experiment. External validity speaks to the degree to which a proposed relationship is generalizable to a broader set of cases or the world at large. Thus, experimental methods are powerful because they are high in internal validity; however, a leap occurs when we attempt to generalize experimentally derived results to actual political behavior.
This leap can be particularly worrisome when it is from an experimental finding generated from a non-elite individual – for example, an undergraduate student as was the case in the Christensen and Redd study – to a foreign policy decision maker. In such cases, the assumption of external validity may not be reasonable. Mintz, Redd, and Vedlitz (2006) explore this issue in detail, replicating an experiment on the subject of counterterrorism with a group of college student and a group of military officers. The authors find significant differences between these groups, suggesting that experimental subjects cannot be expected to play the role of foreign-policy decision makers without careful regard for their actual background. However, while these scholars argue that average individuals can tell us very little about the behavior of elites, they do find it more acceptable to use subjects like students as a sample of the public at large.
Simulation, a close relative of experimental methods, has its roots in the longstanding practices of war gaming and diplomatic analysis. However, recent efforts in this area draw extensively on advances in computing power and the internet. Research in this area builds on early work by the Inter-Nation Simulation (INS) project (Guetzkow et al. 1963), and slightly later efforts by Hermann (1969) and Alker and Brunner (1969).
The International Communication and Negotiation Simulations (ICONS) project is an ongoing extension of this early work that allows political practitioners and students to develop decision making and foreign policy skills through computer aided interactive simulation. Jonathan Wilkenfeld and Richard Brecht developed ICONS in the 1980s, building on Noël’s (1969) early POLIS simulations. As presently formulated, the ICONS project is more about training than research, but the technique presents an intriguing methodological opportunity for those interested in testing theories of foreign policy interactions in a controlled environment.
Survey Research in Foreign Policy Analysis
When it is focused at the elite level, as it often is, survey research in foreign policy analysis directly relates to the previously discussed interview methods. This stands in some contrast to the way in which survey research is conducted in other areas of political science. For example, in American politics there is a long tradition of survey research designed to pinpoint public opinion on a myriad of topics. In order to accomplish this, researchers are obliged to reach as representative a sample of the population as possible. In contrast, FPA’s focus on elite perception and behavior as a determinant of foreign policy leads to the wider usage of elite interviews.
While surveys lack the depth of an interview, they offer the corresponding advantage of breadth. First, by aggregating information from a more significant number of sources, a survey can minimize some of the idiosyncratic error that can plague interview methodology. Second, in a survey analysis it is easier to control for secondary variables that might influence the recollection or reporting of subjects. Finally, surveys can both contribute to qualitative analysis, and serve to generate high-quality data for aggregate analysis.
Holsti and Rosenau’s (1979; 1980) work on post-Vietnam attitudes is an excellent example of what can be accomplished in foreign policy analysis with elite surveys. Holsti and Rosenau were interested in the degree to which historical experience altered the perceptions and beliefs of opinion leaders and decision makers. Their expectation was that the Vietnam conflict significantly altered the perspective of those who drew their primary experience from that conflict rather than World War II. To answer this question they extensively surveyed groups that they believed to comprise the national leadership structure – military personnel, foreign service officers, business executives, labor leaders, clergy, media, etc. – and found significant differences between occupations and within generations.
Surveys can be particularly valuable when conducted repeatedly over several years, as this allows for longitudinal analysis – something that is crucial if one is interested in changes over time. Both the Chicago Council on Foreign Relations and the Pew Research Center for the People and the Press conduct quadrennial surveys of government, academic, military, religious, and scientific “influentials” in order to measure the content of and changes in elite opinion. These surveys, and others that could be conducted along similar lines, are an underutilized resource for foreign policy analysis. Presser et al. (2004) and Rea and Parker (2005) are useful resources for those seeking additional detail on the mechanics of survey research and questionnaire design.
FPA scholars can also benefit from the recent explosion of interest among political scientists in network analysis. Social network analysis, which is simply the mapping and measuring of relationships among entities in a complex system, is a useful tool for modeling foreign policy relationships because it incorporates both bilateral connections and wider connections among the larger group. Because of this, the technique analysis allows FPA scholars to understand relational data – the contacts, ties, connections, and transfers between decision makers that cannot be cleanly reduced to properties of the leaders themselves (Scott 1991). Furthermore, a network theoretic framework consistently captures the role of third parties in foreign policy interactions, which prove to be crucial to understanding outcomes.
Relational approaches have long been an underlying element in the study of foreign policy. For example, Anne-Marie Slaughter (2004) writes on the relationship between elite networks and international conflict. However, quantitative social network analysis first began to make significant inroads into political science in the 1990s primarily through the study of “policy networks” (Marin and Mayntz 1992; Marsh and Rhodes 1992), though there are earlier, pioneering examples (e.g., Eulau and Siegel 1981; Tichy et al. 1979). These studies, as well as later work in international relations (e.g., (Hammarström and Birger 2002; Wilkinson 2002; Montgomery 2005; Heffner-Burton and Montgomery 2006; Maoz 2006; Ward 2006), provide models for future work with foreign policy networks. In short, relational thinking and social network analysis have already contributed to the clarification of a number of puzzles in political science and present a potentially powerful way of approaching foreign policy analysis.
Prediction markets are information exchanges built to generate forecasts using a price mechanism. Futures generated from predictions of upcoming events are traded, such that their value is tied to a particular outcome. The result of this arrangement is that the market prices of these futures can be interpreted as the predicted probability of that outcome. There is a significant body of research that establishes the ability of markets to reduce error in predictions. By aggregating the bets of many individuals, these markets effectively use the price setting mechanism to uncover the consensus about a future foreign policy event in much the same way that the stock market predicts the economic performance of a company or oil futures respond to the expected scarcity of that resource. Pennock et al. (2001) demonstrate that in many cases prediction markets systematically outperform the estimates of even the best individual analysts. There are only a few examples of longstanding prediction markets that handle political futures. These include Intrade, which floats, among many other things, a diverse group of political contracts, and the longer running Iowa Electronic Market, which is an academically oriented project designed for evaluating the probability of election outcomes.
Prediction markets have been applied sparingly in international relations and foreign policy analysis, but have tremendous potential for future application because they offer an interactive mechanism with which individual foreign policy experts can aggregate their knowledge and opinions. Interestingly, given the methodological diversity that characterizes FPA as a sub-discipline, the method by which each expert who trades futures on a prediction market reaches his or her own conclusion is irrelevant. Thus, a prediction market can provide an alternative way to combine and generalize both deep qualitative knowledge and quantitative findings. Furthermore, this approach presents a novel way of dealing with error and uncertainty.
Prospective researchers in this area should note that some early applications of this approach have not gone smoothly. The Defense Advanced Research Projects Agency (DARPA) recently abandoned a promising plan to use a futures market to forecast the probability of important foreign policy events such as regime change and terrorist attacks when the media picked up on the program and it became controversial. Despite a robust literature on the efficacy of such markets, politicians and segments of the public seized upon the effort as being unethical or even nonsensical (Looney 2003). The unwanted attention led DARPA, which usually operates well beneath the public radar, to cancel the project almost immediately. It remains an open question whether this approach will become more politically feasible – seemingly a necessity because these markets generally require a significant initial investment, presumably by a government or university. However, private markets such as Intrade, which is a for-profit enterprise, seem to be a plausible alternative. Foreign policy futures, such as the probability of an Israeli attack on Iran, are traded regularly on Intrade and provide useful information about expectations. Moreover, futures on the outcome of the last presidential election vied with polling data for public and media attention in the lead up to the 2008 US presidential election suggesting that familiarity with these markets may be rising.
Remaining Methodological Challenges
Methods of foreign policy analysis have developed markedly over the past few decades, but challenges remain. An unavoidable tension persists between the accuracy needed for policy relevance and the scope needed for generalizability. As the grand theories of foreign policy interaction motivated those who launched the FPA enterprise proved elusive, the discipline increasingly turned to the nuanced examinations of cases. However, if taken too far this trend is a threat to the unique identity of FPA because it blurs the distinction with longstanding traditions of historical analysis. This survey of available methods suggests that a partial solution to this dilemma lies in bringing quantitative analysis and underutilized “third generation” methods back into the FPA fold by reintegrating them into the well-developed qualitative tradition. The goal should be to develop a healthy mix of methods that applies each approach to the questions which each is best equipped to address.
Additional attention should also be given to determining the degree to which current methods of foreign policy analysis allow predictive or prescriptive conclusions. In recent years, enthusiasm for FPA has been fueled in part by the failure of most international relations scholarship to accurately foresee key events in the international system – specifically the decline of the Soviet Union and the end of the Cold War. The argument is made that Cold War politics, because they were in some sense stable or at equilibrium, were better suited to elegant and parsimonious models of the systemic behavior of state actors. In contrast, the more chaotic world we presently inhabit is characterized by fluidity driven by human agents and therefore is best understood using the methods of foreign policy analysis (Hudson and Vore 1995). This is a reasonable hypothesis; however, prediction is a difficult game in the social sciences and it remains unclear whether FPA is indeed superior in this arena. In short, with a few notable exceptions such as the KEDS project, methods of foreign policy analysis lack predictive capacity and, when they are able to predict, are often unable to clearly state the degree of certainty surrounding these forecasts. More can and should be done to improve this capacity.
Foreign policy analysts should also give deeper consideration to the issues that accompany the choice of the unit of analysis in their models. FPA derives much of its explanatory power from its ability to speak to the individual’s role in the foreign policy process, but the dependent variables that these efforts attempt to explain are often the interactions between states. The result is a gap in our understanding of the process of aggregation by which the behavior of leaders results in the actions and reactions of states. This aggregation problem is widely noted, but additional work is required to complete our understanding of this element of the foreign policy process. Improvements in this linkage between theory and test, as well as a consistent unit of analysis (individual or foreign policy event) are particularly crucial for robust quantitative analysis, as it is in part the inability of the subfield to resolve this basic issue that stifled earlier research on events data.
Finally, more must be done to reengage foreign policy analysis with the core of international relations research. FPA scholars typically claim the first and second image as their domain, but fail to engage with those in mainline international relations who also work in this area. In the lead essay of the first issue of Foreign Policy Analysis, Valerie Hudson (2005) convincingly makes the case that FPA has the potential to reshape the entire discipline of international relations by focusing attention on the workings of the fundamental unit of analysis – the political decision maker. However, despite the call to arms, more often than not FPA scholars labor in relative isolation. Some of these divisions emerge from methodological issues and can therefore be resolved.
In sum, the future of foreign policy analysis appears to be bright. There is reason to believe that longstanding methodological battles that characterized it are drawing to a close with the recognition that multiple methods have their place in the study of foreign policy. In addition, new methods and questions are emerging that are likely to contribute to our understanding of the foreign policy process.
- Alker, H.R., and Brunner, R.D. (1969) Simulating International Conflict – Comparison of 3 Approaches. International Studies Quarterly 13 (1), 70–110.
- Allison, G.T. (1971) The Essence of Decision: Explaining the Cuban Missile Crisis. Boston: Little, Brown.
- Ashworth, S., Clinton, J.D., Meirowitz, A., and Ramsay, K. (2008) Design, Inference, and the Strategic Logic of Suicide Terrorism. American Political Science Review 102 (2), 269–73.
- Axelrod, R. (1976) Structure of Decision: The Cognitive Maps of Political Elites. Princeton: Princeton University Press.
- Azar, E.E. (1980) The Conflict and Peace Data Bank (COPDAB) Project. Journal of Conflict Resolution 24, 143–52.
- Azar, E.E. (1982) The Codebook of the Conflict and Peace Data Bank (COPDAB). College Park: Center for International Development, University of Maryland.
- Azar, E.E., Cohen, S.H., Jukam, T.O., and McCormic, J.M. (1972) Problem of Source Coverage in Use of International Events Data. International Studies Quarterly 16 (3), 373–88.
- Beasley, R.K., Kaarbo, J., Lantis, J.S., and Snarr, M.T. (2002) Foreign Policy in Comparative Perspective: Domestic and International Influences on State Behavior. Washington, DC: CQ Press.
- Beck, N., Katz, J.N., and Tucker, R. (1998) Taking Time Seriously: Time-Series-Cross-Section Analysis with a Binary Dependent Variable. American Journal of Political Science 42 (4), 1260–88.
- Bennett, A., and George, A.L. (2005) Case Studies and Theory Development in the Social Sciences. Cambridge: MIT Press.
- Bennett, D.S., and Stam, A. (2000) EUGene: A Conceptual Manual. International Interactions 26, 179–204.
- Berg, B.L. (2001) Qualitative Research Methods for the Social Sciences, 4th edn. Boston: Allyn and Bacon.
- Brecher, M. (1974) Decisions in Israel’s Foreign Policy. London: Oxford University Press.
- Brecher, M. (2008) International Political Earthquakes: Crises and Conflicts Before, During and After the Cold War. Ann Arbor: University of Michigan Press.
- Brecher, M., and Ben-Yehuda, H. (1985) System and Crisis in International Politics. Review of International Studies 11 (1), 17–36.
- Brecher, M., Steinberg, B., and Stein, J. (1969) A Framework for Research on Foreign Policy Behavior. Journal of Conflict Resolution 13 (1), 75–94.
- Brenner, M., Brown, J., and Canter, D. (eds.) (1985) The Research Interview. London: Academic Press.
- Burgess, P.M., and Lawton, R.W. (1972) Indicators of International Behavior: An Assessment of Events Data Research. Beverly Hills: Sage.
- Christensen, E.J., and Redd, S.B. (2004) Bureaucrats Versus the Ballot Box in Foreign Policy Decision Making: An Experimental Analysis of the Bureaucratic Politics Model and the Poliheuristic Theory. Journal of Conflict Resolution 48 (1), 69–90.
- Eulau, H., and Siegel, J.W. (1981) Social Network Analysis and Political Behavior: A Feasibility Study. The Western Political Quarterly 34 (4), 499–509.
- George, A.L. (1969) Operational Code – Neglected Approach to Study of Political Leaders and Decision-Making. International Studies Quarterly 13 (2), 190–222.
- Gerner, D.J., Schrodt, P.A., Francisco, R.A., and Weddle, J.L. (1994) Machine Coding of Event Data Using Regional and International Sources. International Studies Quarterly 38 (1), 91–119.
- Gochman, C.S., and Maoz, Z. (1984) Military Interstate Disputes, 1816–1976: Procedures, Patterns, and Insights. Journal of Conflict Resolution 28 (4), 585–615.
- Goldstein, J.S., and Freeman, J. (1990) Three Way Street, Strategic Reciprocity in World Politics. Chicago: University of Chicago Press.
- Guetzkow, H., Alger, C.F., Brody, R.A., Noel, R.C., and Snyder, R. (1963) Simulations in International Relations: Developments for Research and Teaching. Englewood Cliffs: Prentice-Hall.
- Hammarström, M., and Birger, H. (2002) The Diffusion of Military Intervention: Testing a Network Position Approach. International Interactions 28, 335–77.
- Heffner-Burton, E., and Montgomery, A.H. (2006) Power Positions: International Organizations, Social Networks, and Conflict. Journal of Conflict Resolution 50 (1), 3–27.
- Hermann, C. (1969) Crises in Foreign Policy: A Simulation Analysis. Indianapolis: Bobbs Merrill.
- Hill, M.R. (1993) Archival Strategies and Techniques. Beverly Hills: Sage Publications.
- Holsti, O.R. (1969) Content Analysis for the Social Sciences and Humanities. Reading: Addison Wesley.
- Holsti, O.R. (1970) The “Operational Code” Approach to the Study of Political Leaders: John Foster Dulles’ Philosophical and Instrumental Beliefs. Canadian Journal of Political Science / Revue Canadienne de Science Politique 3 (1), 123–57.
- Holsti, O.R., and Rosenau, J.N. (1979) Vietnam, Consensus, and the Belief Systems of American Leaders. World Politics 32 (1), 1–56.
- Holsti, O.R., and Rosenau, J.N. (1980) Does Where You Stand Depend on When You Were Born? The Impact of Generation on Post-Vietnam Foreign Policy Beliefs. Public Opinion Quarterly 44 (1), 1.
- Howell, L.D. (1983) A Comparative-Study of the WEIS and COPDAB Data Sets. International Studies Quarterly 27 (2), 149–59.
- Hudson, V.M. (2005) Foreign Policy Analysis: Actor-Specific Theory and the Ground of International Relations. Foreign Policy Analysis 1 (1), 1–30.
- Hudson, V.M., and Vore, C.S. (1995) Foreign Policy Analysis Yesterday, Today, and Tomorrow. International Studies Quarterly 39, 209–38.
- Janis, I.L. (1972) Victims of Groupthink. Boston: Houghton Mifflin Company.
- Johnston, A.I. (1995) Cultural Realism: Strategic Culture and Grand Strategy in Chinese History. Princeton: Princeton University Press.
- Kegley, C., Jr, Salmore, S.A., and Rosen, D.J. (1974) Convergences in the Measurement of Interstate Behavior. In P.J. McGowan (ed.) Sage International Yearbook of Foreign Policy Studies, vol. 2. Beverly Hills: Sage Publications, pp. 309–39.
- Khong, Y.F. (1992) Analogies at War: Korea, Munich, Dien Bien Phu, and the Vietnam Decisions of 1965. Princeton: Princeton University Press.
- King, G., Keohane, R.O., and Verba, S. (1994) Designing Social Inquiry. Princeton: Princeton University Press.
- King, G., and Zeng, L.C. (1999) Logistic Regression in Rare Events Data. At http:/gking.harvard.edu/files/0s.pdf, accessed July 2009.
- King, G., and Zeng, L.C. (2001) Explaining Rare Events in International Relations. International Organization 55 (3), 693–715.
- Krueger, R. (2000) Focus Groups: A Practical Guide for Applied Research, 3rd edn. Thousand Oaks: Sage Publications.
- Larson, D.L. (2001) Sources and Methods in Cold War History: The Need for a New Theory-Based Archival Approach. In C. Elman and M.F. Elman (eds.) Bridges and Boundaries: Historians, Political Scientists, and the Study of International Relations. Cambridge: MIT Press, pp. 327–50.
- Larson, D.W. (1985) Origins of Containment: A Psychological Explanation. Princeton: Princeton University Press.
- Leng, R.J., and Singer, J.D. (1988) Militarized Interstate Crises – the Bcow Typology and its Applications. International Studies Quarterly 32 (2), 155–73.
- Looney, R.E. (2003) DARPA’s Policy Analysis Market for Intelligence: Outside the Box or Off the Wall? Strategic Insights 2 (9).
- Lustick, I.S. (1996) History, Historiography, and Political Science: Multiple Historical Records and the Problem of Selection Bias. American Political Science Review 90 (3), 605–18.
- Maoz, Z. (2006) Network Polarization, Network Interdependence, and International Conflict, 1816–2002. Journal of Peace Research 43 (4), 391–411.
- Marin, B., and Mayntz, R. (eds.) (1992) Policy Networks: Empirical Evidence and Theoretical Considerations. Boulder: Westview Press.
- Marsh, D., and Rhodes, R.A.W. (eds.) (1992) Policy Networks in British Government. New York: Oxford University Press.
- McClelland, C.A. (1970) Some Effects on Theory from the International Event Analysis Movement. Working paper. University of Southern California.
- McClelland, C.A., and Hoggard, G. (1969) Conflict Patterns in the Interactions Among Nations. In J.N. Rosenau (ed.) International Politics and Foreign Policy. New York: Free Press, pp. 711–24
- McCracken, G. (1988) The Long Interview. London: Sage.
- McDermott, R. (2002) Experimental Methodology in Political Science. Political Analysis 10 (4), 325–42.
- Mintz, A., Redd, S.B., and Vedlitz, A. (2006) Can We Generalize from Student Experiments to the Real World in Political Science, Military Affairs, and International Relations? Journal of Conflict Resolution 50 (5), 757–76.
- Mishler, E.G. (1986) Research Interviewing: Context and Narrative. Cambridge: Harvard University Press.
- Montgomery, A.H. (2005) Proliferation Determinism or Pragmatism? How to Dismantle an Atomic Bomb Network. International Security 30 (2), 153–87.
- Moon, B.E. (1983) The Foreign Policy of the Dependent State. International Studies Quarterly 27 (3), 315–40.
- Moon, B.E. (1985) Consensus or Compliance? Foreign-Policy Change and External Dependence. International Organization 39 (2), 297–329.
- Neack, L., Hey, J.A.K., and Haney, P.J. (1995) Generational Change in Foreign Policy Analysis. In L. Neack, J.A.K. Hey, and P.J. Haney (eds.) Foreign Policy Analysis. Contiguity and Change in its Second Generation. Englewood Cliffs: Prentice Hall.
- Neuendorf, K.A. (2002) The Content Analysis Guidebook. Beverly Hills: Sage.
- Noël, R.C. (1969) The Polis Laboratory. American Behavioral Scientist 12 (6), 30–35.
- Paige, G. (1968) Korean Decision, June 24–30, 1950. New York: Free Press.
- Pape, R.A. (2005) Dying to Win: The Strategic Logic of Suicide Terrorism. New York: Random House.
- Pennock, D.A., Lawrence, S., Giles, C.L., and Nielsen, F.A. (2001) The Real Power of Artificial Markets. Science 291 (5506), 987–8.
- Presser, S., Rothgeb, J.M., Couper, M.P., et al. (2004) Methods for Testing and Evaluating Survey Questionnaires. New York: Wiley.
- Rea, L.M., and Parker, R.A. (2005) Designing and Conducting Survey Research: A Comprehensive Guide. San Francisco: Jossey-Bass.
- Reuveny, R., and Kang, H. (1996a) International Conflict and Cooperation: Splicing COPDAB and WEIS Series. International Studies Quarterly 40 (2), 281–305.
- Reuveny, R., and Kang, H. (1996b) International Trade, Political Conflict/Cooperation, and Granger Causality. American Journal of Political Science 40 (3), 943–70.
- Richardson, N.R., and Kegley, C.W., Jr (1980) Trade Dependence and Foreign Policy Compliance: A Longitudinal Analysis. International Studies Quarterly 24 (2), 191–222.
- Rosenau, J.N. (1966) Pre-Theories and Theories of Foreign Policy. In J. Rosenau (ed.) Scientific Study of Foreign Policy. New York: Free Press, pp. 95–151.
- Rosenau, J.N. (1968) Comparative Foreign Policy – Fad, Fantasy, or Field. International Studies Quarterly 12 (3), 296–329.
- Rosenau, J.N. (ed.) (1974) Comparing Foreign Policies: Theories, Findings, and Methods. New York: Halsted Press.
- Rummel, R.J. (1979) Understanding Conflict and War, vol. 4: War, Power, Peace. Beverly Hills: Sage Publications.
- Schoutlz, L. (1987) National Security and United States Policy Towards Latin America. Princeton: Princeton University Press.
- Schrodt, P.A. (1979) Forecasting in International-Relations – Theory, Methods, Problems, Prospects – Choucri, N., Robinson, T.W. American Political Science Review 73(4), 1208–10.
- Schrodt, P.A. (1994) Event Data in Foreign Policy Analysis. In L. Neack, P.J. Haney, and J.A.K. Hey (eds.) Foreign Policy Analysis: Continuity and Change in its Second Generation. New York: Prentice Hall.
- Schrodt, P.A., and Gerner, D.J. (1994) Validity Assessment of a Machine-Coded Event Data Set for the Middle-East, 1982–92. American Journal of Political Science 38 (3), 825–54.
- Schrodt, P.A., and Gerner, D.J. (2000) Cluster-Based Early Warning Indicators for Political Change in the Contemporary Levant. American Political Science Review 94 (4), 803–17.
- Scott, J. (1991) Social Network Analysis: A Handbook. London: Sage.
- Seidman, I. (1998) Interviewing as Qualitative Research: A Guide for Researchers in Education and the Social Sciences, 2nd edn. New York: Teachers College Press.
- Shellman, S.M. (forthcoming) Machine Coding Nonstate Actors’ Behavior in Intrastate Conflict. Political Analysis.
- Shellman, S.M., and Stewart, B. (2007) Predicting Risk Factors Associated with Forced Migration: An Early Warning Model of Haitian Flight. Civil Wars 9 (2), 174–99.
- Silber, L., and Little, A. (1995) The Death of Yugoslavia. London: Penguin.
- Slaughter, A.-M. (2004) A New World Order. Princeton: Princeton University Press.
- Snyder, R.C., Bruck, H., and Sapin, B.M. (1954) Decision-Making as an Approach to the Study of International Relations. Princeton: Princeton University Press.
- Tanter, R. (1974) Modelling and Managing International Conflicts: The Berlin Crises. Beverly Hills: Sage Publications.
- Tichy, N.M., Tushman, M.L., and Fombrun, C. (1979) Social Network Analysis for Organizations. The Academy of Management Review 4 (4), 507–19.
- Tomlinson, R.G. (1993) Monitoring WEIS Event Data in Three Dimensions. In R.L. Merritt, R.G. Muncaster, and D.A. Zinnes (eds.) International Event-Data Developments: DDIR Phase II. Ann Arbor: University of Michigan Press, pp. 55–85.
- Tomz, M., King, G., and Zeng, L.C. (1999) RELOGIT: Rare Events Logistic Regression (Version 1.1). Cambridge. At http:/gking.harvard.edu/stats.shtml, accessed July 2009.
- Trachtenberg, M. (2006) The Craft of International History. Princeton: Princeton University Press.
- Vincent, J.E. (1983) Weis Vs Copdab – Correspondence Problems. International Studies Quarterly 27 (2), 161–8.
- Walker, S.G., Schafer, M., and Young, M.D. (1998) Systematic Procedures for Operational Code Analysis: Measuring and Modeling Jimmy Carter’s Operational Code. International Studies Quarterly 42 (1), 175–89.
- Ward, H. (2006) International Linkages and Environmental Sustainability: The Effectiveness of the Regime Network. Journal of Peace Research 43 (2), 149–66.
- Weber, R.P. (1985) Basic Content Analysis. Beverly Hills: Sage.
- West, M.D. (2001) Theory, Method, and Practice in Computer Content Analysis. Westport: Ablex.
- Wilkinson, D. (2002) Civilizations as Networks: Trade, War, Diplomacy, and Command-Control. Complexity 8 (1), 82–6.
- Winham, G.R. (1969) Quantitative Methods in Foreign Policy Analysis. Canadian Journal of Political Science 2 (2), 187–99.
Links to Digital Materials
International Crisis Behavior Project. At www.cidcm.umd.edu/icb/, accessed July 2009. The ICB project provides quantitative and qualitative data on international crises. The core systemic dataset currently contains 452 incidents from the end of World War I through 2006. The link above provides access to ICB data, codebooks, citations, and a variety of other useful materials.
Correlates of War Project. At www.correlatesofwar.org/, accessed July 2009. The COW project website provides widely used events data on militarized interstate disputes as well as several other datasets that may be appropriate for FPA scholars. Data, documentation, and codebooks are available through the link.
EUGene Software. At www.eugenesoftware.org/, accessed July 2009. The link provides access to the EUGene software package, as well as manuals and documentation (all free of charge). EUGene allows for relatively easy transition between commonly used units of analysis – country–year, dyad–year, and directed dyad–year. In addition, the software enables the researcher to combine many of the events datasets discussed in this essay with basic demographic and geopolitical data including data uploaded by the user.
Kansas Event Data System. At http:/web.ku.edu/keds/, accessed July 2009. KEDS provides a computer program that enables users to specify and create personalized events datasets. Detailed descriptions of machine coding methods, as well as several datasets and codebooks, are available on the KEDS web page.
The Consortium on Qualitative Research Methods. At www.maxwell.syr.edu/moynihan/programs/cqrm/index.html, accessed July 2009. CQRM promotes qualitative research methods in the social sciences. They hold an annual training institute that may be useful for FPA scholars interested in expanding their knowledge of qualitative methods. In addition, they maintain a database of syllabae on qualitative methods.
Intrade. At www.intrade.com/, accessed July 2009. Intrade is the leading prediction market for political futures. The site has prediction markets for a wide range of political and financial outcomes that may be of interest to FPA scholars.
I would like to acknowledge the helpful comments of Michael Glosny, Deborah Larson, Rachel Augustine Potter, and two anonymous reviewers. All errors are my own.