Introduction

In the past few decades, experiments have been successfully introduced in many fields of economics, such as industrial organization (e.g., Chamberlin, 1948; Sauermann & Selten, 1959; Plott, 1982), labor economics (e.g., Fehr, Kirchsteiger, & Riedl, 1993; Kagel, Battalio, Rachlin, & Green, 1981), and public economics (e.g., Andreoni, 1988; Bohm, 1984; Marwell & Ames, 1981). Despite the fact that the use of experiments was first advocated by leading health economists long ago (Frank, 2007; Fuchs, 2000; Newhouse et al., 1981), their introduction and employment has been relatively slow to be widely accepted in health economics, policy, and management.

Recently, however, two special issues in the Journal of Economic Behavior & Organization (Cox, Green, & Hennig-Schmidt, 2016) and in Health Economics (Galizzi & Wiesen, 2017) and a number of dedicated special sessions in major field conferences (e.g., the European Association of Health Economics, EuHEA, the International Health Economics Association, iHEA) indicate the increasing acceptance of experiments by the health economics, policy, and management communities.

The rise in interest in using experiments in health economics has coincided with the parallel growing interest in applying behavioral economics to health, as witnessed by an increasing number of books and articles on the topic (Bickel, Moody, & Higgins, 2016; Hanoch, Barnes, & Rice, 2017; Loewenstein et al., 2017; Roberto & Kawachi, 2015).

Among health policymakers and practitioners, the use of insights from behavioral economics, and, in particular, of “nudges” has recently led many governments around the world to set up behavioral or “nudge units” within their civil services, starting from the Behavioural Insights Team in the UK Cabinet Office, to the analogous initiatives within the UK Department of Health, the National Health Service (NHS), and Public Health England, in the governments of Australia, Canada, Denmark, Finland, France, Israel, Italy, the Netherlands, New Zealand, Norway, Singapore, and the United States, and in the European Commission (Dolan & Galizzi, 2014a; Oliver, 2017; Sunstein, 2011).

We start with a simple operational definition of “behavioral experiments in health,” arguably the first such definition. We then identify ten key areas of potential debate about behavioral experiments in health that we think deserve explicit discussion. In what follows, we address one by one each of these ten areas of possible debate and controversy by answering ten corresponding questions. By doing so, we review the state of the art of the different streams and areas for applications of the growing field of behavioral experiments in health; we discuss which significant questions remain to be addressed; and we highlight the rationale and the scope for the further development of behavioral experiments in health in the years to come.

In a nutshell, “behavioral experiments in health” make use of a broad range of experimental methods typical of experimental and behavioral economics to investigate individual and organizational behaviors and decisions related to health and healthcare.

The behaviors and decisions considered in behavioral experiments in health therefore usually take place, or are framed, in a health, healthcare, or medical setting or context.

The term behavioral in “behavioral experiments in health” first requires a clarification. Common to experimental economics and behavioral science, the outcomes of behavioral experiments in health are “behavioral” in that they consist of directly observable and measurable behavioral responses or directly revealed preferences, rather than self-reported statements. For example, subjects in behavioral experiments in health are typically observed in real health or healthcare field situations, or, if not, they face real consequences for their choices or behaviors through aligned monetary and nonmonetary incentives. Behaviors and decisions of participants in a behavioral experiment in health are thus typically “natural”—that is they take place in naturalistic situations—or “incentive-compatible” in the usual experimental economics sense that participants bear some real behavioral consequences for their choices in the experiment (e.g., Cassar & Friedman, 2004Friedman & Sunder, 1994; Smith, 1976, 1982). This defining feature makes behavioral experiments in health distinct from “stated preference experiments,” such as contingent valuation studies, or “discrete choice experiments” (DCEs), which have long been used in health economics, and which do not typically consider real behavior or incentive-compatible choice situations (e.g., de Bekker-Grob, Ryan, & Gerard, 2012; Ryan & Farrar, 2000; Ryan, McIntosh, & Shackley, 1998).

Furthermore, methodologically, behavioral experiments in health purportedly cover the entire continuum spectrum of experimental methods spanning the lab to the field, passing through online and mobile experiments, and experiments pursuing “behavioral data linking” (for more, see Questions Three and Ten).

Finally, and following the usual methodological convention in experimental economics, behavioral experiments in health do not deceive subjects. Some behavioral experiments in health can, nonetheless, entail some degree of “obfuscation” when, in the attempt to minimize possible “experimenter demand effects” (Zizzo, 2010), subjects are not told about the exact purpose and research question of the experiment. This is in line with the spirit of those experiments that intend to minimize the alteration of, and interference with, naturally occurring behavior by not telling subjects that they are part of an experiment (i.e., in the spirit of “natural field experiments” according to the taxonomy by Harrison & List, 2004, discussed in Question Three; or of “lab-field” experiments, as in Dolan & Galizzi, 2014a).

To sum up, five characterizing features of behavioral experiments in health are therefore: (i) the fact that the decisions and behaviors are health-related; (ii) the fact that, whenever possible, the outcomes of the decisions in the experiment are “behavioral” in the sense of consisting of directly observable and measurable behavioral responses, or of bearing real consequences for the decision makers; (iii) the open-minded consideration of principles and insights from both behavioral economics and conventional economics, as well as their combination and integration; (iv) the use of a broad range of experiments spanning the lab to the field, passing through online and mobile experiments, as well as “behavioral data linking” experiments; and (v) the tendency to avoid deception, which, however, does not prevent the use of obfuscation, natural field experiments, and lab-field experiments.

We next review the existing literature by addressing ten areas of current debate about behavioral experiments in health. A few of these areas apply to behavioral experiments more generally and, when this is the case, we explicitly note it.

Question One: What Can Behavioral Experiments Tell Us That Non-Experimental Methods in Health Cannot Tell Us Already?

First of all, theory, experiments, and econometrics are complements to, not substitutes for, each other (Falk & Heckman, 2009; Galizzi, Harrison, & Miraldo, 2017b); Harrison, Lau, & Rutström, 2015. In particular, the way in which behavioral experiments are sometimes contrasted with econometric analysis is misleading. In fact, running any type of controlled behavioral experiments in health (see more in question three) is just the first step of the data collection process that should then feed into an appropriate econometric analysis of the experimental data. The broad range of behavioral experiments in health allows the researcher to gather rich data to delve empirically into the behavioral nuances and mechanisms of an observed change in health-related behavior. Indeed, as witnessed by the field of “behavioral econometrics,” experiments and econometric analysis are complementary, not substitute, methods (Andersen, Harrison, Lau, & Rutström, 2008a, 2010, 2014; Harrison, Lau, & Rutström, 2015; Hey & Orme, 1994). A similar point holds for the theoretical underpinnings of a behavioral experiment in health.

Second, a key advantage of behavioral experiments is the ability to tightly control experimental conditions. In physics and social sciences, testing theory is a basic component of experiments and the scientific method relies upon explicit tests of theory (Charness & Fehr, 2015; Charness & Kuhn, 2011). While secondary data are often rich and abundant, at the same time they might be confounded by a variety of environmental factors. For example, a health economist who aims to test the effect of incentives inherent in performance pay on the physicians’ quality of medical care using secondary data might end up with confounded results because institutions such as public monitoring and reporting of physicians’ quality were introduced at the same time. Using secondary data, disentangling these factors seems prohibitively challenging, if not impossible. Taking a more general perspective, the key strength of behavioral experiments is the ability to test a specific theoretical model. One can then compare the behavioral predictions of the model to what happens. If a specific theory is rejected, it is then relatively neat to test competing explanations. For example, rational decision-making theory might not be suitable to explain the inconsistent choices of insurance plans in the United States (e.g., Abaluck & Gruber, 2011). Competing behavioral decision-making theories might then be called upon whose alternative explanations can then be tested in controlled experiments. This is consistent with the open-minded approach of behavioral experiments in health, which consider principles and insights from both behavioral economics and conventional economics.

Third, another reason to run experiments is the unique opportunity to study behavior and practices analyzed in the theoretical health economics models that are difficult to observe using field data. An example is the effect of referral payments from specialists to primary care physicians on primary care physicians’ referral behavior. While health economic theory (e.g., Pauly, 1979) suggests that referral fees enhance efficiency, payments for referrals are largely forbidden, in almost all Western healthcare markets. In a lab experiment, Waibel and Wiesen (2017) explicitly test model predictions on physicians’ diagnostic effort and referral decisions and find that the introduction of referral payments increase efficiency, although not to the levels predicted by theory. Another example is unethical behavior in healthcare, for example, diagnosis-related group (DRG) upcoding. Admittedly, at an aggregate level there is plenty of evidence that DRG upcoding exists (e.g., Jürges & Köberlein, 2015; Silverman & Skinner, 2004). However, what drives unethical behaviors is largely unknown. A study by Hennig-Schmidt, Jürges, and Wiesen (2017) complements field studies on DRG upcoding by analyzing dishonest behavior in a framed experiment in neonatology and by linking dishonest behavior to individuals’ characteristics to explore what drives dishonesty. They find that audits and fines significantly reduce dishonesty and that subjects’ personality traits and integrity relate to dishonest behavior. A further area in health, to which behavioral experiments have contributed, is a better understanding of the behavioral effects of professional norms. Exogenously changing professional norms in the field seems prohibitively challenging, and (if possible) drawing inferences seems difficult due to numerous confounding factors. In an online experiment with a large medical student sample, Kesternich, Schumacher, and Winter (2015) analyze the effect of making the Hippocratic oath salient on patient-regarding altruism and distributional preferences. In a series of experiments with physicians (from internal medicine and pediatrics), Ockenfels and Wiesen (2018) investigate the effect of a professional framing on physicians’ dishonest behavior (on behalf of themselves and others). Evidence from behavioral experiments in health that are “well-grounded” in theory is therefore not only useful to contrast behavior with model predictions but also to further stimulate the debate among healthcare policymakers on practices with little or no field evidence.

Fourth, one of the key strengths of experiments, in general, is that a researcher can empirically study the causal effects of different institutions, as defined by their rules, actors, and incentives. Thus, behavioral experiments seem ideal to serve as a test bed for analyzing the effect of institutional changes related to healthcare. Understanding behavioral mechanisms in health-related decisions is imperative before designing and implementing large-scale behavioral interventions in the field or ad hoc healthcare policy interventions, as there might be unknown or unintended effects for providers and patients alike. In this sense, excluding behavioral experiments from the research toolkit of a health economist would be somewhat similar to ignoring animal studies for medical or drug research:

While results from animal studies do not always apply to humans, the ability to test many hypotheses cheaply under carefully controlled conditions provides an indispensable tool for the development of models that work in the real world.

(Charness & Kuhn, 2011, p. 233)

Fifth, behavioral results from experiments might not only be insightful to better understand actual health-related decision-making and behavior, but also to inform the development of behavioral economics theories in health contexts (e.g., Hansen, Anell, Gerdtham, & Lyttkens, 2015; Kőszegi, 2003, 2006; Frank, 2007). The observation of actual human behavior in experiments enables the researcher to identify behavioral deviations from theory and thus to extend health economic theories by taking into account issues like human motivation or behavioral phenomena like emotions or (patient-regarding) altruism. Two prominent research areas in which theory and experiments have already fruitfully complemented each other are: the matching markets for organ donations and for physicians and healthcare professionals (e.g., Herr & Normann, 2016; Kessler & Roth, 2012, 2014a, 2014b; Li, Hawley, & Schnier, 2013; Roth, 2002; Roth & Peranson, 1999); and the design of mixed systems of public and private healthcare finance (e.g., Buckley et al., 2012, 2016).

In sum, running behavioral experiments in health allows the researcher to better understand the causal effects of health-related policy interventions on individual and organizational behavior and to contrast findings with predictions from theoretical models. Behavioral experiments therefore nicely complement and bridge non-experimental methods, in particular theory and empirical econometric analysis, and could therefore help bring closer together the different health research communities (Galizzi, 2017; Galizzi, Harrison, & Miraldo, 2017).

Question Two: Are Behavioral Experiments Really New to Health Economics, Policy, and Management?

No. First, health economists are particularly well aware of the importance of using randomized controlled experiments. Modern evidence-based medicine and pharmacology are all based on randomized controlled trials (RCTs), starting from the pioneering work on scurvy by James Lind in 1747, to the first published RCT in medicine by Austin Bradford Hill and colleagues in 1948. Thanks to the groundbreaking contributions of Charles Sanders Peirce, Jerzy Neyman, Ronald A. Fisher, and others, modern science has long considered randomized controlled experiments as fundamental and important scientific methods. Far from novel, the idea of using randomized controlled experiments has been advocated for decades even for policy applications (Burtless, 1995; Ferber & Hirsch, 1978; Rubin, 1974).

Second, arguably one of the most influential studies in health economics is indeed based on a large-scale randomized controlled experiment. The RAND Health Insurance Experiment conducted in the United States between 1974 and 1982, in fact, analyzed the effects of randomly allocated co-payment rates and health insurance contracts on healthcare costs and utilization of healthcare (Manning et al., 1987; Newhouse et al., 1981). As a major finding, Joseph P. Newhouse and colleagues documented that cost sharing reduced the overutilization of medical care while it did not significantly affect the quality of care received by participating patients.

The spirit and the main features of the RAND Health Insurance Experiment later inspired the design of the Oregon Health Insurance Experiment. The latter was conducted in 2008 with uninsured low-income adults in Oregon. Adults allocated to the treatment group were given the chance to apply for Medicaid (via a lottery). This allowed researchers to analyze the effects of expanding access to public health insurance (Medicaid), for example, on the healthcare use and health of low-income adults. The researchers found that the treatment group had substantively and statistically significantly higher healthcare utilization and a better self-reported health than the control group (Finkelstein et al., 2012; Finkelstein & Taubman, 2015).

The launch of the Behavioural Experiments in Health Network (BEH-net) in 2015 can be seen as the response to a fast-increasing demand to systematically use behavioral experiments in health economics, policy, and management. The network aims precisely at integrating and bringing closer together the research communities at the interface between experimental and behavioral economics, and health economics.

Question Three: What Types of Experiments are Considered When Referring to Behavioral Experiments in Health?

There is an important initial conceptual distinction between behavioral experiments in health and RCTs. Many health practitioners and policymakers, in fact, tend to automatically associate behavioral experiments with RCTs.

In the health policy debate, the term RCT is sometimes used to denote large-scale field experiments conducted with entire organizations (e.g., hospitals, villages) without necessarily allowing the stakeholders in those organizations to explicitly express their views or their consent to the proposed manipulations. This is a major conceptual and practical difference with respect to proper RCTs in medicine or pharmacology, where subjects are always explicitly asked to give informed consent prior to taking part in RCT, and are allowed to drop out, with important ethical, political, and logistical implications. The term RCT is therefore conceptually inappropriate and practically misleading in a health economics, policy, and management context, since it conveys the false impression that subjects have been made aware of being part of an experiment and have been consulted and given their consent to it, when actually this may not be the case.

Moreover, even in the above narrow and inappropriate connotation, RCTs are only one specific type of experiment, namely field experiments. As mentioned, however, behavioral experiments in health purportedly cover the entire spectrum of experiments from the lab to the field. Harrison and List (2004) proposed an influential taxonomy of experiments along the lab-field spectrum that illustrates the diversity of experiments: (i) conventional lab experiments involve student subjects, abstract framing, a lab context, and a set of imposed rules; (ii) artefactual field experiments depart from conventional lab experiments in that they involve nonstudent samples; (iii) framed field experiments add to artefactual field experiments a field context in the commodity, stakes, task, or information; and, finally, (iv) natural field experiments depart from framed field experiments in that subjects undertake the tasks in their natural environment and subjects do not know that they are taking part in an experiment.

The main idea behind natural field experiments is equivalent to von Heisenberg’s “uncertainty principle” in physics: the mere act of observation and measurement necessarily alters, to some extent, what is being observed and measured. In key areas for health economics, for example, there may be potential experimenter demand effects, where participants change behavior due to cues about what represents “appropriate” behavior for the experimenter (Levitt & List, 2007), for example, when deciding on provision of medical services; Hawthorne effects, where simply knowing they are part of a study makes participants feel important and improves their effort and performance (Levitt & List, 2011); and John Henry effects, where participants who perceive that they are in the control group exert greater effort because they treat the experiment like a competitive contest and they want to overcome the disadvantage of being in the control group (Cook & Campbell, 1979).

More recently, other types of experiments have been conducted in experimental economics, beyond lab, artefactual field, frame field, and natural field experiments. For example, virtual experiments combine controlled experiments with virtual reality settings (Fiore, Harrison, Hughes, & Rutström, 2009). While not yet applied to health economics contexts, virtual experiments are a promising approach to make trade-offs more salient and vivid in health and healthcare decision-making.

Lab-field experiments consist of a first-stage intervention under controlled conditions (in the lab) linked to a naturalistic situation (in the field) where subjects are not aware that their behavior is observed. Lab-field experiments have been used to look at the unintended “behavioral spillover” effects of health incentives (Dolan & Galizzi, 2014b, 2015; Dolan, Galizzi, & Navarro-Martinez, 2015) or at the external validity of lab-based behavioral measures (Galizzi & Navarro-Martinez, 2019).

Virtual experiments and lab-field experiments are part of the growing efforts to bridge the gap between the lab and the field in health economics applications (Hennig-Schmidt, Selten, & Wiesen, 2011; Kesternich et al., 2015). They are also part of the more general “behavioral data linking” approach (Galizzi, Harrison, & Miraldo, 2017), that is, the linkage of behavioral economics experiments with longitudinal surveys, administrative registers, biomarkers banks, apps, mobile devices, scan data, and other big data sources (Andersen et al., 2015; Galizzi, Harrison, & Miniaci, 2017). Data linkage poses new ethical, practical, and logistical challenges when it seeks to link surveys and behavioral experiments with health records and healthcare registers (Galizzi, Harrison, & Miraldo, 2017). Nonetheless, there is currently an extraordinary, and still largely untapped, potential to apply the experimental approach to an unprecedented host of health and healthcare contexts, and by linking and augmenting behavioral experiments in health with the very rich data sources available in health (see more in Question Ten).

Taken together, there is not one type of experiment for potential health economics and policy purposes. Rather, the broad spectrum of different types of experiments spanning the lab to the field can prove useful and complementary for health applications, as well as the most recent online, mobile, and “behavioral data linking” experiments.

Question Four: Is There a Preferred Type of Behavioral Experiment in Health?

There is currently no consensus on which specific type of behavioral experiment is superior. The choice of the specific type of experiment depends on the specific research question. Lab, field, online, mobile, and “behavioral data linking” experiments all have strengths and weaknesses, and their relative merits have been systematically discussed elsewhere (Bardsley et al., 2009; Falk & Heckman, 2009; Guala, 2005; Harrison, 2013; Harrison & List, 2004; Kagel, 2015; Levitt & List, 2007, 2009; Smith, 2002).

For example, it is generally reckoned that lab experiments allow for high internal validity because of their ability to tightly control the environment and frame, to minimize confounding factors, to closely simulate conditions of theoretical models, and to replicate past experiments. Furthermore, they provide insights into possible patterns prior to moving into the field, they uncover the mechanisms underlying decisions and behavior, and they require significantly fewer financial, time, and logistical resources than field experiments.

On the other hand, it is generally reckoned that field experiments can enhance the external validity of experimental results (see more about this in Question Five), because observations are made with subjects, environments, situations, tasks, rules, and stakes that are closer to the ones occurring in the real world (Brookshire, Coursey, & Schulze, 1987; Galizzi & Navarro-Martinez, 2019). Field experiments, however, come with less control and with several other limitations when used for policy purposes (Harrison, 2014). Moreover, they are inherently more difficult to replicate. This is a major limitation given the increasing attention to the replicability of experimental results in economics, psychology, and health sciences (Burman, Reed, & Alm, 2010; Camerer et al., 2016; Dolan & Galizzi, 2014a; Galizzi, Harrison, & Miraldo, 2017; Open Science Collaboration, 2015).

More generally, it is important to reiterate the point that the different types of experiments are complementary, not substitutes (Falk & Heckman, 2009; Galizzi, Harrison, & Miraldo, 2017; Harrison, Lau, & Rutström, 2015). Recent behavioral experiments in health pick up on this important point and combine different types of experiments and thereby test for the validity of findings from lab experiments with conventional student samples. For example, Brosig-Koch, Hennig-Schmidt, Kairies-Schwarz, and Wiesen (2016a) combine lab and artefactual field experiments to analyze the effect of fee-for-service and capitation regimes for medical service provision on the behaviors of medical students, nonmedical students, and physicians. Across the board, they found that all subject pools responded to incentives similarly—namely, patients were overtreated under fee-for-service and undertreated under capitation. Physicians, however, responded less to the incentives inherent in these two payment schemes. In experiments with medical and non-medical students, Hennig-Schmidt and Wiesen (2014) and Brosig-Koch, Hennig-Schmidt, Kairies-Schwarz, and Wiesen (2017a) report similar findings. Wang, Iversen, Hennig-Schmidt, and Godager (2017) compare behavior of physicians and medical students in China and Germany. Comparing lab experimental data from medical students with artefactual field experimental data from a subsample of a representative sample of German resident physicians, Brosig-Koch, Hennig-Schmidt, Kairies-Schwarz, and Wiesen (2017b) find that performance pay crowds out patient-regarding motivation for both subject pools.

Taken together, the experimental studies that systematically account for potential differences in the subject pools indicate that the direction of a treatment effect does not differ between (medical or nonmedical) students and medical professionals. Importantly, however, the intensity of a behavioral effect might vary across subject pools. Moreover, experimental designs in health typically abstract from the complexity of real-world settings in order to “isolate” treatment effects. A few experimental studies even employ neutral framings of the decision situation, presumably for reasons of control and salience of the incentives: see, for example, Green (2014) on provider incentives; Huck, Lünser, Spitzer, and Tyran (2016) on health insurance choice; and Mimra, Rasch, and Waibel (2016) on specialists’ second opinions. As experiments are “scalable,” adding more realism (health context) to these settings by employing medical frames seems desirable: Ahlert, Felder, and Vogt (2012) and Kesternich et al. (2015) find behavioral differences in the intensity of subjects’ responses when comparing neutral and medical frames.

In sum, researchers and healthcare policymakers alike might clearly need to be cautious in drawing conclusions about real-world settings when taking findings from behavioral experiments in health at face value. One might argue, however, that this key point applies to any type of behavioral experiments in health, not just to lab experiments or to field experiments. Moreover, the existing evidence indicates somewhat similar main directions of experimental treatment effects across subject pools and it is therefore useful to inform debates in healthcare policy and management. For a more general discussion of external validity and generalizability of experimental findings, see the next question.

Question Five: Can We Trust the External Validity of Behavioral Experiments in Health?

The point on the external validity of behavioral experiments in health is too important to be unduly misrepresented or oversimplified, as is often done in the research and policy debate. Most issues related to the external validity of experiments are actually not unique to behavioral experiments in health, but are common to all the economic experiments in general, and for this reason we answer this question more generally.

The main observation is that external validity means different things from different points of view. In a first connotation, external validity refers to the within-subjects question of whether the outcomes of a behavioral experiment in health are representative of the corresponding outcomes of interest that would occur outside of the behavioral experiment for the same pool of subjects. From this perspective, as mentioned, external validity is often contrasted with internal validity on the presumed ground that there is always an inherent trade-off between internal and external validity while one moves from the lab end to the field end of the spectrum in the Harrison and List (2004) taxonomy. This, however, is not always nor necessarily the case. In fact, to start with, if rigorously designed and conducted, all randomized controlled behavioral experiments in health are internally valid, whether they are lab, artefactual field, framed field, or natural field experiments. So, it is simply not true that internal validity is necessarily higher in lab than in natural field experiments. Moreover, it is also not true that the external validity (in the above explained connotation) is necessarily higher in natural field than in lab experiments. It obviously depends on how close is the correspondence between the outcomes measured in the behavioral experiment and the outcomes of interest that would occur outside of the behavioral experiments. In other words, it depends on what the final outcomes of interest are, and ultimately, what the research question is. For example, imagine that the main outcome of interest is about looking at how many calories subjects eat in a buffet, or how much time they wait until lighting up the next cigarette, which are both likely to be the outcomes of some automatic or “visceral” decision-making occurring without any conscious deliberation (Loewenstein, 1996). Then, a natural field experiment, where subjects do not know they are part of an experiment, would be a natural setting for observing such behaviors (Dolan & Galizzi, 2014b). On the other hand, however, imagine that the main outcomes of interest are how subjects trade off and choose between different private health insurance schemes, or which groceries subjects purchase in an online supermarket, two highly deliberate decisions that are both likely to take place in online settings even when they occur outside of a behavioral experiment. Then, a conventional lab or an online experiment would be a natural setting for observing such behaviors.

In a second, slightly different, connotation, external validity refers to the, still within-subjects, question of whether the outcomes of a behavioral experiment in health are good predictors of the corresponding outcomes of interest that would occur outside of the behavioral experiment for the same pool of subjects. For example, are healthy food choices in a behavioral experiment good predictors of a subject’s healthy diet? Are experimental decisions about drugs, treatments, and health insurance good predictors of analogous decisions outside the experiment? It is true that, in principle, the experimental decisions can move closer to the decisions taken outside the experiment as one moves from lab experiments to natural field experiments in the Harrison and List (2004) taxonomy. But it is also true that this depends, again, on what the ultimate outcomes of interest and research questions are. And also, in principle, this type of external validity question can affect the entire spectrum of behavioral experiments in health, from lab to natural field experiments. In fact, the only rigorous strategy to empirically address this question is to design and implement a longitudinal augmentation of the original behavioral experiment that enables researchers to follow up over time in more naturalistic settings the same pool of subjects. When implemented in a systematic and transparent way, this strategy also allows the researcher to overcome a major limitation of the very few external validity analyses to date, namely the fact that they typically are ad hoc analyses. The typical analysis, in fact, reports the correlation between one specific experimental outcome and one specific variable outside the experiment and, when such a correlation is found to be statistically significant, it concludes that what is found in the experiment is externally valid. Such an approach, however, lacks systematization because it fails to provide full information on the whole set of pairwise correlations between all the experimental outcomes and all the variables outside the experiment, be they significant or not significant. Only a systematic and transparent testing and reporting of all such correlations would be the litmus test of the external validity of a behavioral experiment. Such exercises are rare in experimental economics and virtually not existent in health.

In a nonhealth context, for example, Galizzi and Navarro-Martinez (2019) systematically assess and report the associations between the whole set of eight social preferences experimental games and five different prosocial variables outside of the experiment, and they find that only one out of 40 pairwise correlations is statistically significant (and none is if there is proper correction for multiple hypothesis testing). They then relate their finding to a systematic review and meta-analysis of all the published and unpublished studies that have previously tested the external validity of those same experimental games, and conclude that the often proclaimed external validity of social preferences games is not supported by the empirical evidence: only the 39.7% of the reported pairwise correlations and 37.5% of the reported regressions find a statistically significant association between an experimental outcome and a variable outside the experiment.

The behavioral experiments in the health literature still lack a similar systematic and transparent approach to this dimension of the external validity question and, given the importance of this exercise for both research and policy purposes, we encourage more research in this direction (Galizzi, Machado, & Miniaci, 2016a). For example, the lack of transparency and systematization is the main explanation behind the current sterile debate about whether lab-based behavioral economics measures for risk, time, and social preferences are externally valid in the health context. Very few studies in behavioral health economics have tied their hands by publishing a pre-analysis plan or a public protocol that clearly and explicitly states at the outset of the analysis which health behaviors in the field will be associated with the lab-based behavioral economics measures. Most studies only report ad hoc subsets of the correlations and regressions between those measures and the health behaviors, and do not report nor discuss how these results relate and compare to the whole set of statistically significant and not significant associations. Moreover, systematic replication is almost inexistent in behavioral experiments in health. It is even argued by strong proponents of the lab experiments that preferences can only be measured in the lab. This is tantamount to stating that the question of whether lab-based measures for those preferences are externally valid is a non-falsifiable question, which is the opposite of an evidence-based scientific approach to this key matter. If behavioral experiments in health are about to leave infancy for adulthood, they should better take seriously the lessons learned in the neighboring disciplines of medicine and health studies, where collective knowledge is systematically cumulated only through transparent replications, systematic reviews, and meta-analyses, as epitomized by major collective research infrastructures, such as the Cochrane Collaboration or the Campbell Collaboration.

A third connotation of external validity has to do with whether the outcomes of a behavioral experiment in health are representative of the corresponding outcomes of the population of interest. This “out-of-sample” connotation of external validity clearly requires that the pool of subjects involved in the behavioral experiment in health has been drawn from a representative sample of the population of interest. So, for example, if the behavioral experiment aims at concluding something about decisions or behaviors of medical doctors, it should involve a representative sample of medical doctors, while if it aims at inferring anything at a population level, the behavioral experiment should involve a representative sample of the population.

Moreover, the debate about the external validity of behavioral experiments in health should be more generally conducted within the broader framework of the debate in terms of generalizability—that is, the question of which other populations, settings, contexts, or domains the findings from an experiment can be generalized to (Al-Ubaydli & List, 2015). Importantly, the generalizability question equally applies to the whole spectrum of behavioral experiments in health, from lab to natural field experiments.

There are three conceptually distinct threats to generalizability of behavioral experiments in health. The first threat comes essentially from participation bias. Unlike natural field experiments, lab, artefactual field, and framed field experiments recruit subjects through an explicit invitation to take part in an experiment. As a result, there is bias because subjects who choose to participate in experiments may be inherently different in their underlying characteristics from subjects who choose not to take part. Health researchers and policymakers should therefore be aware that, because of the participation bias, even if the initial sample of subjects is indeed representative of the target student population, the resulting subsample of actual respondents may not be.

The second threat comes from the fact that the environment, context, and frame of the experimental decisions and tasks in the lab may not be representative of real situations encountered by subjects in natural health and healthcare settings. This limitation can be easily overcome by redesigning tasks and contexts to more closely match naturalistic situations that subjects are more familiar with in real life—that is, to design framed field experiments in the sense of Harrison and List (2004). This strategy has been extensively employed in other application areas in experimental economics (e.g., Harrison & List, 2008; Harrison, List, & Towe, 2007) and has already been explored in the health economics area (e.g., Eilermann et al., 2017; Hennig-Schmidt et al., 2011; Hennig-Schmidt & Wiesen, 2014; Galizzi et al., 2016).

The last threat to generalizability is that experimental subjects may not be representative of the general population, especially when they are students or medical students (Levitt & List, 2007). To overcome this limitation, behavioral economists have started running artefactual field experiments with representative samples of the population (Andersen et al., 2008a, 2014; Galizzi, Machado, & Miniaci, 2016; Galizzi, Harrison, & Miniaci, 2017; Harrison et al., 2007; Harrison, Lau, & Williams, 2002). This is a promising avenue for behavioral experiments in health, given that the goals and priorities in designing health policies and health systems are typically set at a population level (Michie, 2008).

From the broader generalizability perspective, we can hardly see why the results of a natural field experiment with, say, female nurses in Tanzania, or health insurance customers in the rural Philippines, should be considered as more generalizable than a lab experiment with medical students in Germany, or an artefactual field experiment with a representative sample of the population in the United Kingdom. Too often, similar claims even forget to state what the population of interest is for the study.

More generally, Falk and Heckman (2009) state that causal knowledge requires a controlled variation. Whether a variation from, for example, a natural field experiment or a more controlled lab experiment is more informative depends on the research question and is still debated among researchers in social sciences. It is important to acknowledge, again, that empirical methods and different sources of data are complements. For example, both behavioral experiments, spanning the lab to the field, and econometric analyses of secondary data can all improve the state of knowledge in health economics research, with the issue of generalizability of results being applicable to all of them.

Taken together, behavioral and experimental health economists should take seriously the external validity and generalizability challenges by open-mindedly using all types of experiments in the lab-field spectrum, by embracing a transparent and systematic approach in gathering and reporting evidence, including reporting of all the statistically significant and not significant correlations and regressions (rather than cherry-picked subsets of the positive results). We see this as a fundamental requisite for behavioral experiments in health as a field moving, in the years to come, from infancy to adulthood.

Question Six: What About Experiments to Elicit Preferences in Health?

One of the above-discussed defining features of behavioral experiments in health is that they entail directly observable and measurable behavioral responses. For example, experimental decisions, tasks, and measures to elicit preferences and willingness-to-pay are incentive-compatible in the sense that subjects pay some real consequence in terms of monetary or nonmonetary outcomes for the choices they make. This raises the question of whether or not behavioral experiments in health also include the experimental studies that aim at eliciting health-related preferences.

Some distinctions should be made on this point. On the one hand, there is a vast literature in health economics that uses popular experimental methods, such as the Standard Gamble (SG) or the Time Trade Off (TTO), to elicit preferences for hypothetical health states (Attema & Brouwer, 2012; Bleichrodt, 2002; Bleichrodt & Johannesson, 2001). Given the hypothetical nature of the choices about different health states, these experiments are similar in nature to the already discussed “stated preference experiments,” such as contingent valuation studies or discrete choice experiments (DCEs), which do not typically consider real behavior or incentive-compatible choice situations. Using the same argument, therefore, the experiments in this literature should not be considered behavioral experiments in health.

On the other hand, there is also a small, but growing, literature looking at the relationships between incentive-compatible experimental measures of risk and time preferences and health-related behaviors. Harrison, Lau, and Rutström (2010), for example, elicit risk and time preferences of a representative sample of the Danish population and find no difference in the likelihood of smokers and nonsmokers exhibiting hyperbolic discounting, no significant association of smoking with risk aversion among men, and no significant association of smoking with discount rates among women. Galizzi and Miraldo (2017) measure the risk preferences of a convenience sample of students and find that, while there is no association between smoking or body mass index (BMI) with the estimated risk aversion, the latter is significantly associated with the Healthy Eating Index, an indicator of overall nutritional quality. Harrison, Hofmeyr, Ross, and Swarthout (2015) elicit risk and time preferences of a convenience sample of students at the University of Cape Town and find that smokers and nonsmokers differ in their baseline discount rates, but do not significantly differ in their present bias, risk aversion, or subjective perception of probabilities. In a longitudinal experiment with a representative sample of the UK population, Galizzi, Machado, and Miniaci (2016) systematically assess the external validity of different measures of risk preferences linked to the UK Longitudinal Household Survey (UKHLS), and find that the experimental measures are not significantly associated with subjects’ BMI and eating, smoking, or drinking habits. Several other ad hoc analyses have associated risk and time preferences with heavy drinking (Anderson & Mellor, 2008), BMI (Sutter, Kocher, Glätzle-Rützler, & Trautmann, 2013), and the uptake of vaccinations, preventive care, and medical tests (Bradford, 2010; Bradford et al., 2010; Chapman & Coups, 1999).

Given that all these latter experiments use incentive-compatible methods to elicit risk and time preferences, they should be considered behavioral experiments in health. A common aspect of the latter group of experiments, however, is that they measure individual risk and time preferences over (risky or intertemporal) monetary outcomes, and then link these to health-related behaviors. But what about the studies that elicit individual risk and time preferences for health outcomes, rather than for monetary outcomes?

We see the experiments eliciting risk and time preferences in health as an interesting middle ground between stated preferences experiments and behavioral experiments. When it comes to the measurement of risk and time preferences in the health domain, in fact, the current community of behavioral health experimentalists interprets behavioral experiments in health with a fair degree of tolerance, flexibility, and open-mindedness, and considers the elicitation of risk and time preferences in health a research field that is closely aligned with, and affine to, the core interests and methods of behavioral experiments in health.

This is not because the community disagrees with the traditional experimental economics view that answers to hypothetical questions can significantly differ from responses to incentive-compatible tests because “talk is cheap” if there are no real behavioral consequences (Battalio, Kagel, & Jiranyakul, 1990; Cummings, Elliott, Harrison, & Murphy, 1997; Cummings, Harrison, & Rutström, 1995; Harrison, 2006; Holt & Laury, 2002;). Moreover, behavioral health experimentalists are all well aware that, from a theoretical perspective, risk and time preferences are fundamental individual characteristics at the core of health behavior and decision-making (Gafni & Torrance, 1984). Risk and time preferences, in fact, directly inform the principles and practices of cost-effectiveness analysis (CEA) and cost-utility analysis (CUA) in healthcare, and the assumptions beyond the quality adjusted life years (QALY), the measure of health benefits that is commonly employed in CEA and CUA and that relies on the above-mentioned SG and TTO methods (Bleichrodt, Wakker, & Johannesson, 1997).

Rather, it is because at the moment the literature on behavioral experiments in health lacks a systematic body of state-of-the-art consensus methods to measure health-related preferences with real nonmonetary consequences. Given the fundamental role of risk and time preferences in the health context, it is actually surprising that there is no consensus to date on a “gold standard” measurement methodology.

A multitude of different methods have been proposed to measure risk and time preferences in health contexts, which are heterogeneous in terms of underlying theoretical frameworks, methodological features, and links to formal econometric analysis (Galizzi, Harrison, & Miraldo, 2017). A major challenge in converging to a consensus methodology to measure risk and time preferences in health is related to the fact that, to date, the different proposed methods are substantially disconnected. On the one hand, the current methods to measure preferences for health outcomes only entail hypothetical scenarios. On the other hand, all the incentive-compatible methods to measure preferences with real consequences are based on monetary outcomes. From both a conceptual and an empirical point of view, however, it is unclear whether individual risk and time preferences are stable across the health and the monetary domains (Chapman, 1996).

There have been a number of exploratory analyses of whether these preferences are indeed stable across the finance and the health domains. Galizzi, Miraldo, and Stavropoulou (2016), for example, summarize the relatively limited number of studies that compare risk taking across the health and other domains, and find that, despite the broad heterogeneity of methods and frames used in the literature, there is general evidence that there are differences across domains, and that these differences also emerge when real consequences are at stake.

The elicitation of risk and time preferences with incentive-compatible methods in the health domain is a promising and challenging task and a collective priority in the research agenda of behavioral experimentalists in health. We expect this methodological and substantial gap to be filled soon by the increasingly collaborative community of behavioral experimentalists in health.

Question Seven: Which Topics Are Addressed by Behavioral Experiments in Health?

A first area of experimental research that has recently received considerable attention are “nudges,” that is, changes in the “choice architecture” made to induce changes in health behavior, mainly at an unconscious or automatic level (Thaler & Sunstein, 2008). In the spirit of “asymmetric paternalism” (Loewenstein, Asch, & Volpp, 2013; Loewenstein, Brennan, & Volpp, 2007), many behavioral experiments have in fact applied nudges to health and healthcare behavior spanning from risky behaviors in adolescents (Clark & Loheac, 2007) to exercise (Calzolari & Nardotto, 2016), from food choices (Milkman, Minson, & Volpp, 2014; Schwartz et al., 2014; VanEpps, Downs, & Loewenstein, 2016a; Schwartz, Riis, Elbel, & Ariely, 2012; 2016b) to drugs compliance (Vervloet et al., 2012), from medical decision-making (Ansher et al., 2014; Brewer, Chapman, Schwartz, & Bergus, 2007; Schwartz & Chapman, 1999) to dentists’ services (Altman & Traxler, 2014).

There is, however, much more in behavioral experiments in health than just nudging (Galizzi, Harrison, & Miraldo, 2017; Oliver, 2017). Behavioral experiments in health can, in fact, uncover the behavioral mechanisms beyond the change in health behavior, and thus inform the design and the implementation of a series of other types of health policies, including informational campaigns, salient labeling and packaging of healthy food items, the use of financial and nonfinancial incentives, and the design of effective tax, subsidy, health insurance plans, and regulatory schemes (Galizzi, 2014, 2017).

In fact, a broad spectrum of lab to natural field experiments have already been applied to a variety of health economics, policy, and management areas, well beyond “nudges.” For example, behavioral experiments in health have investigated the effects of different co-payment rates and health insurance contracts on healthcare utilization and costs (Manning et al., 1987; Newhouse et al., 1981); the effects of public health insurance coverage on healthcare utilization and health outcomes (Baicker et al., 2013; Finkelstein et al., 2012; Finkelstein & Taubman, 2015; Finkelstein, Taubman, Allen, Wright, & Baicker, 2016); the effects of different providers’ incentives and the role of altruism (Ahlert et al., 2012; Brosig-Koch et al., 2016a, 2016b, 2017a, 2017b; Fan, Chen, & Kann, 1998; Godager & Wiesen, 2013; Green, 2014; Hennig-Schmidt et al., 2011; Hennig-Schmidt & Wiesen, 2014; Kesternich et al., 2015; Kokot, Brosig-Koch, & Kairies-Schwarz, 2017); the role of audit, transparency, compliance, and gender bias in healthcare management (Godager, Hennig-Schmidt, & Iversen, 2016; Hennig-Schmidt et al., 2017; Jakobsson, Kotsadam, Syse, & Øien, 2016; Lindeboom, van der Klaauw, & Vriend, 2016); the role of different healthcare financing policies (Buckley, Cuff, Hurley, McLeod, Mestelman, & Cameron, 2012; Buckley, Cuff, Hurley, Mestelman, et al., 2015, 2016); two-part tariffs for physician services (Greiner, Zhang, & Tang, 2017); provider competition (Bosig-Koch, Hehenkamp, & Kokot, 2017; Han, Kairies-Schwarz, & Vomhof, 2017); the matching markets for organ donations and for physicians and healthcare professionals (Herr & Normann, 2016; Kessler & Roth, 2012, 2014a, 2014b; Li, Hawley, & Schnier, 2013; Roth, 2002; Roth & Peranson, 1999); the role of subsidies for diagnostic tests and new health products (Cohen, Dupas, & Schaner, 2015; Duflo, Dupas, & Kremer, 2015; Dupas, 2014a, 2014b; Dupas, Hoffman, Kremer, & Zwane, 2016); the choice of health insurance (Buckley, Cuff, Hurley, McLeod, Nuscheler, & Cameron, 2012; Huck et al., 2016; Kairies-Schwarz, Kokot, Vomhof, & Weßling, 2017; Kesternich, Heiss, McFadden, & Winter, 2013; Schram & Sonnemans, 2011); the economic and behavioral determinants of vaccination (Binder & Nuscheler, 2017; Böhm, Betsch, & Korn, 2016; Böhm, Meier, Korn, & Betsch, 2017; Bronchetti, Huffman, & Magenheim, 2015; Massin, Ventelou, Nebout, Verger, & Pulcini, 2015; Milkman, Beshears, Choi, Laibson, & Madrian, 2011; Tsutsui, Benzion, & Shahrabani, 2012); the effects of different types of HIV risk information and of SMS interventions on HIV treatment adherence (Dupas, 2011; Rana et al., 2015); the use of financial incentives for smoking cessation (Gine, Karlan, & Zinman, 2010; Halpern et al., 2015; 2016; Volpp et al., 2009), physical exercise (Charness & Gneezy, 2009; Royer, Stehr, & Sydnor, 2015), weight loss (John et al., 2011;John, Loewenstein, & Volpp, 2012; Kullgren et al., 2013, 2016; Rao, Krall, & Loewenstein, 2011; Volpp et al., 2008), healthy eating (Loewenstein, Price, & Volpp, 2016), warfarin adherence (Kimmel et al., 2012, 2016; Volpp et al., 2008), glucose control (Long, Jahnle, Richardson, Loewenstein, & Volpp, 2012), home-based health monitoring (Sen et al., 2014), mental exercises (Schofield, Loewenstein, Kopisc, & Volpp, 2015), immunization coverage (Banerjee, Duflo, Glennerster, & Kothari, 2010), nursing services (Banerjee, Duflo, & Glennerster, 2007), and medical drugs (Samper & Schwartz, 2013); the unintended carryover and spillover effects of financial incentives and nudges in health (Chiou, Yang, & Wan, 2011; Dolan et al., 2015; Dolan & Galizzi, 2014b, 2015; Müller et al., 2009; Wisdom, Downs, & Loewenstein, 2009); the behavioral effect of decision support systems and feedback (Cox et al., 2016; Eilermann et al., 2017); the elicitation of risk and time preferences in health and their links with health-related behaviors (Allison et al., 1998; Anderson & Mellor, 2008; Attema, 2012; Attema, Bleichrodt, & Wakker, 2012; Attema & Brouwer, 2010, 2012, 2014; Attema & Versteegh, 2013; Bleichrodt & Johannesson, 2001; Bradford, 2010; Bradford, Zoller, & Silvestri, 2010; Cairns, 1994; Cairns & van der Pol, 1997; Chapman, 1996; Chapman & Coups, 1999; Chapman & Elstein, 1995; Dolan & Gudex, 1995; Galizzi, Machado, & Miniaci, 2016; Galizzi & Miraldo, 2017; Galizzi, Miraldo, & Stavropoulou, 2016; Galizzi, Miraldo, Stavropoulou, & van der Pol, 2016; Garcia Prado, Arrieta, Gonzalez, & Pinto-Prades, 2017; Harrison et al., 2015; Harrison, Lau, & Rutström, 2010; Michel-Lepage, Ventelou, Nebout, Verger, & Pulcini, 2013; Sutter et al., 2013; Szrek, Chao, Ramlagan, & Peltzer, 2012; van der Pol & Cairns, 1999, 2001, 2002, 2008, 2011), the elicitation of preferences for retransplantation (Ubel & Loewenstein, 1995), and end-of-life decisions (Halpern et al., 2013).

Nudges are just one of the many areas of applications. They tend to be relatively effective in the health contexts where people suffer from “internalities,” costs that we incur because we fail to account for our future selves (Herrnstein, Loewenstein, Prelec, & Vaughan, 1993). Many other health situations are, however, also affected by externalities. Other policy tools, such as taxes, subsidies, and regulatory interventions, have been documented to deal effectively with externalities in health markets (Bhargava & Loewenstein, 2015; Galizzi, 2017). The application of behavioral experiments to these policy areas is at the moment almost nonexistent, and we foresee an increase of applications in this key area.

Another related aspect for both research and policy purposes is the rationale informing the legitimacy of nudging people. There is a major gap in the literature in measuring underlying preferences before the nudging interventions take place, and in monitoring their evolution (if any) before and after being nudged. This would help in understanding heterogeneity in behavioral change, as well as in identifying the behavioral channels, mechanisms, and mediating factors that are activated when individuals’ behavior is nudged. At the same time, it would inform the design of target-specific nudges, incentives, and behavioral regulatory tools (thus advancing the state-of-the-art evidence beyond knowing just “what works”). The issue of which set of preferences should be considered for drawing a welfare analysis of nudges and other behavioral interventions is one of the most relevant and pressing open questions from both a conceptual and an empirical perspective, as well as another area of promising development for the next waves of behavioral experiments in health.

Question Eight: How Do Framing and Subject Pool Matter in Behavioral Experiments in Health When Analyzing Healthcare Professionals’ Behavior?

While a neutral framing of the experimental decision is appropriate in an experiment on decision-making in games of strategic interactions, a medical framing appears natural for behavioral experiments on decision-making in medical contexts. Kesternich et al. (2015) show that framing in a health context affects subjects’ behavior in modified dictator and trilateral distribution games. In particular, in their health frame, subjects decide on the role of physicians in the provision of medical services with consequences for real patients outside the lab (similar to Hennig-Schmidt et al., 2011) in the modified dictator game. In a trilateral distribution game, consequences for the insured bearing the cost of medical service provision are also added.

More generally, recent practice in behavioral experiments in health is aligned with the belief that unless you frame a decision situation—for example in a medical or health frame—a researcher cannot be sure how subjects in an experiment have perceived the decision situation (Galizzi & Navarro-Martinez, 2019; Harrison & List, 2008). It may thus be crucial in chosen effort tasks to set subjects in the context the experimenter wants to study in order to avoid the possibility that subjects may impose a context on the abstract experimental task that is different from the experimenter’s intended context (e.g., Engel & Rand, 2014; Harrison & List, 2004). As Harrison and List (2004) notice, “it is not the case that abstract, context-free experiments provide more general findings if the context itself is relevant to the performance of the subjects” (p. 1022).

It remains unclear, however, whether a change in behavior due to framing more or less accurately reflects true behavior of healthcare professionals. One may argue that a healthcare professional in a health-framed study may be more willing to forgo earnings to avoid looking bad (“experimenter demand effect”; Zizzo, 2010). In practice, healthcare professionals in a neutrally framed decision situation could even become less responsive to how choices directly affect patients than when facing a series of choices in a framed task. Further, individuals may value their health and the health of others differently than any other good. Therefore, both unframed and framed experiments may misinterpret choices by healthcare professionals. For these reasons, it is important to study whether a medical framing in experiments more accurately reflects behavior in healthcare delivery (Cox, Green, & Hennig-Schmidt, 2016; Kesternich, Schumacher, & Winter, 2015).

Further, different subject pools used in health-related experiments (nonmedical students, medical students, physicians) may significantly change behavioral results, with nonmedical students exerting less patient-regarding altruism (Brosig-Koch et al., 2017a; Hennig-Schmidt & Wiesen, 2014). Considering fraudulent behavior in a routine task in neonatal intensive care units (entry of weights in the birth reports), Hennig-Schmidt et al. (2017) found some evidence for more honest behavior of medical students compared to economics students. A few studies with healthcare professionals and medical students in developing countries correlate neutrally framed social preferences with actual health-related behaviors (e.g., Brock, Lange, & Leonard, 2016; Kolstad & Lindkvist, 2012).

Taken together, three promising avenues for behavioral experiments in health on this issue are (i) rigorously and systematically testing the behavioral effects of framing and subject pools; (ii) extending initial findings from the laboratory to field experiments, ideally with healthcare professionals (Cox, Sadiraj, Schnier, & Sweeney, 2016; Eilermann et al., 2017; Leeds et al., 2017); and (iii) linking findings from behavioral experiments to actual health-related behaviors. In this sense, it seems thus appropriate to call, again, for more systematic evidence—also from healthcare systems in developed countries—to be able to gather more conclusive predictions of providers’ behavior in the field.

Question Nine: Is Health Really Different from Other Policy Domains?

The specificity of health as a policy domain is self-evident. On the one hand, health is a very special area of policy application for obvious ethical and political reasons, and even more so for the application of randomized controlled behavioral experiments. Health, moreover, is a research and policy area that is uniquely rich in data: think about the millions of yearly entries in healthcare records and administrative registers (e.g., the Hospital Episode Statistics in the United Kingdom); the large epidemiological cohorts and clinical RCTs; and the complex databanks containing the genetic and epigenetic profiling at a population level. It is also unclear from both a conceptual and an empirical point of view whether behaviors and decisions in the health domain merely reflect behaviors and decisions in other domains in life, for example in the financial domain. As mentioned, the small experimental literature on cross-domain preferences seems to suggest that preferences are not stable across the health and monetary domains. Also the literature on the use of financial incentives in health finds that their effects are less straightforward and universally applicable than in other fields of applications (Gneezy, Meier, & Rey-Biel, 2011). A more specific example is healthcare provider incentives. Recent experimental findings suggest that healthcare providers’ behavior seems to be affected by pay-for-performance pay, but that this might also lead to adverse effects, such as motivation crowding-out (e.g., Brosig-Koch et al., 2016a, 2017b; Oxholm, 2016). The latter pattern is, for example, not observed in other working domains (admittedly with different performance schemes), such as in field experiments with teachers (e.g., Muralidharan & Sundararaman, 2011). Therefore, a note of caution is in order when extrapolating lessons from experiments in other fields and generalizing them to the health domain.

On the other hand, behavioral health economists should be careful in advocating a complete disconnection of health applications from other areas of application of behavioral and experimental economics. On the contrary, they should continue arguing that there is much that can be learned from health applications that is useful to other policy domains. This can also help to reduce the substantial disengagement between economic and medical journals. For example, generalist economic journals seem to regularly publish behavioral experiments in, for example, education, financial savings, and energy consumption more regularly than in health, which are sometimes dismissed as “too field-specific.” That health is of more, not less, general interest than other subfields of economics is directly confirmed by the stellar impact factor and international reputation and visibility of the medical journals.

Question Ten: What Can Behavioral Experiments in Health Tell Us About Long-Term Effects?

It is true that, at the moment, there is very little evidence on the long-term carryover effects and on the cross-behavioral spillover effects of nudges, incentives, and other health policy interventions (Dolan & Galizzi, 2014a, 2015; Dolan et al., 2015). This is also due to the fact that, in practice, it is difficult to design behavioral experiments that follow up subjects over time for periods of time longer than a couple of hours (in the lab) or a few weeks or months (in the field), or that track all the complex ramifications of an initial policy intervention on the whole set of targeted and nontargeted health behaviors.

There is, more generally, a sort of major gap and disconnection between two key sources of empirical evidence in health economics. On the one hand, the behavioral experiments in health are typically conducted with small samples of subjects and almost invariably are centered on a single observation window or a single data collection point. On the other hand, very comprehensive longitudinal data sets exist in health in forms of administrative records for healthcare access (e.g., the Hospital Episode Statistics in the United Kingdom), biomarkers banks (e.g., the UK Biobank in the United Kingdom), or medical records and biomarkers for epidemiological cohorts (e.g., Constances in France).

The time seems ripe to systematically link and integrate these two major data sources. The already mentioned recent spring of experiments on “behavioral data linking” has shown that it is indeed feasible to link and merge behavioral economics experiments with other data sources, such as longitudinal surveys, online panels, administrative records, biomarkers and epigenetics banks, apps and mobile devices, smart cards and scan data, clinical RCTs, and other big data sources (Andersen et al., 2015; Galizzi, Harrison, & Miniaci, 2017). Given the inherent data-richness of health as a research and policy domain, we expect behavioral data linking to become a key building block of the next generation of behavioral experiments in health. This will contribute to further integrating and cross-fertilizing insights, tools, and methods from behavioral, experimental, and health economics, and to shaping up a groundbreaking interdisciplinary area at the interface between behavioral, medical, and data sciences.

Conclusions

This article reviews the state of the art of behavioral experiments in health by critically discussing ten key areas of potential debate and misconception, by highlighting their theoretical and empirical rationale and scope, and by discussing the significant questions which remain.

As our discussions indicate, there are many areas within health economics where experimental methods can be applied fruitfully. To date, in fact, a broad spectrum of behavioral experiments from the lab to the field have already been applied to numerous different health-related areas, including, for example, the effects of different co-payment rates and health insurance contracts on healthcare utilization and costs; the effects of public health insurance coverage on healthcare utilization and health outcomes; the effects of different providers’ incentives and the role of altruism; the role of audit, transparency, compliance, and gender bias in healthcare management; the role of different healthcare financing policies; the matching markets for organ donations and for physicians and healthcare professionals; the role of subsidies for diagnostic tests and new health products; the choice of health insurance; the economic and behavioral determinants of vaccination; the effects of different types of HIV risk information and of SMS interventions on HIV treatment adherence; the use of financial incentives and nudges for smoking cessation, physical exercise, weight loss, healthy eating, warfarin adherence, glucose control, home-based health monitoring, mental exercises, immunization coverage, and medical drugs; the unintended carryover and spillover effects of financial incentives and nudges in health; the behavioral effect of decision support systems and feedback; the elicitation of risk and time preferences in health and their links with health-related behaviors; and the elicitation of preferences for retransplantation and end-of-life decisions. Many other health-related areas are expected to follow in the next years in both developing and developed countries.

Tailoring and fine-tuning the broad spectrum of lab, field, online, mobile, and “behavioral data linking” experiments in order to address pressing health policy challenges and key research questions is, both methodologically and substantially, one of the most promising and exciting areas of applications of behavioral experiments to health economics. Also, via the new international networks, the next cohort of behavioral experiments in health is likely to originate from a closer collaboration among behavioral and experimental economists, health economists, medical doctors, and decision makers in health policy and management. This forthcoming generation of behavioral experiments in health will likely scale up the current endeavors to systematically link behavioral economics measures to other data sources, in which health is naturally rich. In the years to come, the promise and the research and policy impact of behavioral experiments in health are only destined to grow.