Healthcare Decision-Making and Multi-Criteria Decision Analysis

Most important decisions in the health sector involve confronting trade-offs between multiple, often conflicting, objectives, usually subject to a resource constraint (e.g., a fixed budget). An example is choosing which health “technologies” (i.e., drugs, devices, procedures, etc.) to fund—a process known as health technology assessment (HTA). Decision makers usually handle the trade-offs between objectives by evaluating the alternatives under consideration according to criteria relevant to the overarching decision—in order to, depending on the application, rank (or prioritize) or choose between the alternatives. For example, Golan, Hansen, Kaplan, and Tal’s (2011) review of the HTA literature identified criteria for prioritizing new health technologies associated with each technology’s clinical benefits and appropriateness, efficiency (including cost-effectiveness), and patient need, as well as equality, solidarity, and other ethical or social values.

Decision makers can apply their criteria in either an implicit or an explicit fashion. In the last decade or so, multi-criteria decision analysis (MCDA)—sometimes also referred to as multi-criteria decision-making (MCDM)—has been increasingly used to support healthcare decision-making (Diaby, Campbell, & Goeree, 2013). In general terms, MCDA (or MCDM), a subdiscipline of operations research with foundations in economics, psychology, and mathematics, is concerned with decision-making situations in which multiple criteria are to be explicitly evaluated and, in effect, combined, in order to rank or choose between alternatives (Belton & Stewart, 2001).

Of course, applying criteria to evaluate alternatives is a natural decision-making approach as old as human history. However, “traditional” decision-making—and how most people make their everyday decisions, too—usually involves weighting the criteria and evaluating trade-offs in an intuitive or holistic manner. By contrast, MCDA is concerned with more formally structuring and solving decision problems, usually involving the explicit weighting of criteria and trade-offs between them.

A prominent example of an application of MCDA to healthcare decision-making that has received a lot of attention in recent years and that is the main subject of this article is HTA. For example, in 2012, the U.K. organization responsible for assessing the value for money of new medicines, the National Institute for Health and Clinical Excellence (NICE), considered the use of “structured decision-making” (i.e., MCDA) in its formal methods review (NICE, 2012), ultimately rejecting MCDA as not representing a clear improvement on existing deliberative processes. Several HTA bodies have piloted the use of MCDA, including, most recently, the Swedish Dental and Pharmaceutical Benefits Agency (Angelis, 2018). In 2014, the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) convened the MCDA Emerging Good Practices Task Force “charged with establishing a common definition for MCDA in healthcare decision-making and developing good practice guidelines for conducting MCDA to aid health care decision-making” (Thokala et al., 2016, p. 2; see also Marsh et al., 2016).

The recent advent of “value assessment frameworks” in the United States has increased interest in MCDA’s potential to help weigh up the competing aspects of value being considered (see Norman, Chalkidou, & Culyer, 2018, and other articles in the same issue of Value in Health). The value frameworks aim to support decision-making about new technologies in the U.S. health system and are intended to measure and to communicate the value of pharmaceuticals and other healthcare technologies for decision-making, using multiple attributes of value (Neumann, Willke, & Garrison, 2018). There has also been a flurry of articles (including this one!)—both in favor of MCDA (Drake, Trujillo de Hart, Monlean, Toto, & Valetim, 2017) and expressing doubts about its suitability (Campillo-Artero, Puig-Junoy, & Culyer, 2018)—and books (Marsh, Goetghebeur, Thokala, & Baltussen, 2017) on the subject of MCDA in healthcare decision-making.

In addition to HTA, other examples of health MCDA applications include: prioritizing R&D portfolios in pharmaceutical companies (Phillips & Bana E Costa, 2007); allocating budgets across disease areas, such as the Health Foundation’s (2018) “socio-technical allocation of resources” (STAR) toolkit; prioritizing patients for elective (nonurgent) surgery (Hansen et al., 2012); disease classification and diagnosis (Johnson et al., 2014); prioritizing antibiotic-resistant diseases for R&D (Tacconelli et al., 2018); supporting patients and clinicians in selecting treatments (Dolan, 2008); and weighing up the benefits and risks of new medicines to support licensing decisions (Phillips, Fasolo, Zafiropoulos, & Beyer, 2011).

These examples are obviously quite different from each other; however, they all include the four key elements common to all MCDA applications in general: alternatives, criteria, weights, and decision makers. That is, all seven applications above (plus HTA) involve ranking or selecting alternatives—R&D investments, patients, diseases, treatments, or new medicines—based on the application of criteria and weights (representing the relative importance of the criteria) according to the preferences of decision makers (and, potentially, other stakeholders).

This article aims to explain the steps involved in a MCDA process, including an overview of methods, and to contribute to the research agenda for increasing MCDA’s use in healthcare decision-making. Because of HTA’s prominence—because all health systems have to grapple with choosing health technologies to fund—and the recent attention paid to HTA, most of the article’s discussion is in terms of HTA; nonetheless, the ideas discussed apply to other health MCDA applications as well, and they are also addressed.

The origins and emergence of MCDA are discussed briefly, and this is followed by a discussion of weighted-sum models, the main approach for health applications. Then, the main steps and methods available for implementing an MCDA process are explained. After consideration of the arguments against the use of MCDA for HTA and the counterarguments, methodological and normative issues associated with operationalizing MCDA for HTA are discussed. The article ends with a summary of conclusions.

Origins and Emergence of MCDA

Historically, the first example of a formal MCDA approach is usually recognized as being Benjamin Franklin’s “moral or prudential algebra” from 1772 (Franklin, 1973). Franklin’s approach, which he explained in a letter to his friend Joseph Priestley about how to make important decisions, involved successively trading off—in effect, weighting—the pros and cons of one alternative relative to another in order to identify the preferred one. Although this simple approach is effective for decisions involving just two alternatives, it does not scale up well for more alternatives.

More contemporary methods for choosing between more than just two alternatives (and involving multiple decision makers) were presented in Ralph Keeney’s and Howard Raiffa’s seminal book Decisions with Multiple Objectives: Preferences and Value Tradeoffs (Keeney & Raiffa, 1993), published first in 1976. In 1979, Stanley Zionts helped popularize the abbreviation MCDM (for multiple criteria decision-making) with his article for managers: “MCDM—If not a Roman numeral, then what?” (Zionts, 1979). Other significant events from MCDM’s history are discussed in Köksalan, Wallenius, and Zionts, (2011).

Nowadays, MCDA is supported by specialized software (Oleson, 2016; Weistroffer & Li, 2016), which is often commercially available as a web application. The software is designed to assist decision makers “at various stages of the decision-making process, including problem exploration and formulation, identification of decision alternatives and solution constraints, structuring of preferences, and trade-off judgments” (Weistroffer & Li, 2016, p. 1302). MCDA software is discussed in the section “Steps in the MCDA Process.”

Consistent with the modern-day emergence of MCDA since the 1970s, MCDA has been widely used in many nonhealth sectors, such as energy, the environment, the military, management, and local and central government, for example. By contrast, healthcare decision makers have been relatively slow to adopt MCDA. However, increasing attention is being paid to MCDA now, although as far as health economists are concerned, the attention is sharply divided between advocacy for, and criticism of, using MCDA for HTA.

Weighted-Sum Models

Most health MCDA applications (and nonhealth MCDA applications) involve value-measurement approaches based on weighted-sum models—also variously known as additive, linear, scoring, point-count, and points models (or systems), or in the MCDA scientific literature as additive multi-attribute value models (Belton & Stewart, 2001). In short, such models involve decision makers explicitly weighting the criteria for the decision problem being addressed and rating the alternatives on the criteria, with each alternative’s “performance” on the criteria aggregated via a linear (i.e., additive) equation to produce the alternative’s “total score,” by which the alternatives are ranked relative to each other.

More specifically, an alternative’s score on each criterion (usually in the range 0 to 100), representing the alternative’s degree of achievement on the criterion, is multiplied by the criterion’s relative weight (summing across the criteria to unity), representing the relative importance of the criterion to the decision maker. These weighted scores are then summed across the criteria to produce the alternative’s total score (in the range 0 to 100). Based on their total scores, alternatives are ranked relative to each other.

A common equivalent representation of the linear equations described above is a schedule of “point values” for each criterion—a “points system”—where each criterion is demarcated into mutually exclusive and exhaustive performance categories (e.g., low, medium, high). The point value for each category on a criterion represents the combined effect of the criterion’s relative importance (weight) and its degree of achievement as reflected by the category. (In other words, consistent with the linear equations described above, a category’s point value can be obtained by multiplying the weight for the criterion by the score for the category from the equation; likewise, a point value can be “decomposed” into the score on the criterion’s performance category and the criterion’s weight.) Each alternative is “scored” on the points system and the alternative’s point values for each criterion are summed across the criteria, again to produce a total score by which the alternatives are ranked. Arguably, a points system (a simple schedule of point values) is easier to implement manually than an equation (a set of weights and scores that need to be multiplied together).

In practical terms, a major attraction of weighted-sum models—in either their linear equation or points system forms—is the simplicity by which each alternative’s performance on each criterion is aggregated to produce a total score for each alternative. (Nonlinear functions for aggregating the criteria, including multiplicative functions, are also possible but are very rare.)

More importantly, such simple models have been found nearly universally in very many studies of health and nonhealth applications to be more accurate than the unaided (i.e., intuitive or holistic) judgments of decision makers (Kahneman, 2011). According to Hastie and Dawes (2010, p. 52), weighted-sum models are

surprisingly successful in many applications. We say ‘surprisingly,’ because many judges claim that their mental processes are much more complex than the linear summary equations would suggest—although empirically, the equation does a remarkably good job of ‘capturing’ their judgment habits.

Hastie and Dawes (p. 52) also explained that:

The mind is in many essential respects a linear weighting and adding device. In fact, much of what we know about the neural networks in the physical brain suggests that a natural computation for such a ‘machine’ is weighting and adding, exactly the fundamental processes that are well described by linear equations.

Finally, it is also worthwhile mentioning (mainly for completeness) that other MCDA-based approaches not based on aggregative functions for combining alternatives’ performance on criteria (i.e., via weights) are also potentially available as alternatives to weighted-sum models—albeit such alternative approaches are unlikely to be particularly useful.

The simplest alternative approach to weighted-sum models is a table—sometimes referred to as a “performance matrix”—for reporting the alternatives’ performance on the criteria (i.e., where each criterion is a column in the table/matrix). When one alternative dominates the others on all criteria, or where the trade-offs involved in selecting an alternative are clear and uncontroversial, decision makers can use such a table to reach their decision. Of course, most decision problems are more complicated than this! Most important health applications involve confronting nontrivial trade-offs between criteria; therefore, merely tabulating alternatives’ performance on the criteria is insufficient for resolving most decision problems (although such a table is useful at step 3 in the MCDA process outlined in the next section).

A more sophisticated MCDA approach capable of evaluating trade-offs between criteria, and that is not based on aggregative functions for combining alternatives’ performance on criteria (via weights), is the group of “outranking” methods, including PROMETHEE (Vincke & Brans, 1985), GAIA (Brans & Mareschal, 1994), and the ELECTRE family of methods (Roy, 1991). (The names are acronyms. PROMETHEE stands for Preference Ranking Organization METHod for Enrichment of Evaluations, GAIA stands for Geometrical Analysis for Interactive Aid, and ELECTRE stands for ELimination Et Choix Traduisant la REalité, which translates as “elimination and choice expressing reality.”) In essence, outranking methods involve decision makers’ pairwise rankings of alternatives relative to each other on each criterion in turn and then the combining of the pairwise-ranking results (but not via weights) in order to obtain a measure of support for judging each alternative as the top-ranked alternative overall. Despite their use for nonhealth applications, outranking methods are rarely used for healthcare decision-making—probably due to their complexities and the nonintuitive nature of their inputs and algorithms relative to weighted-sum models (Belton & Stewart, 2001).

Steps in the MCDA Process

Step 7: Supporting Decision-Making

The MCDA results can be presented in tables or in graphs for decision makers to review. As for any analysis based on uncertain data inputs, it is important to check the robustness of the results via sensitivity analysis to plausible variations in alternatives’ performances (step 3), their scores on the criteria (step 4), and criteria weights (step 5).

In some applications, other considerations in addition to the MCDA results may also be relevant. For example, decisions with implications for healthcare budgets, such as HTA decision-making about new technologies, involve weighing up the benefits of each alternative, as represented by their MCDA total score, relative to their costs—and perhaps also considering technologies’ quality of evidence and additional “X-factors” not elsewhere included, such as strategic or legal factors (Golan & Hansen, 2012). Graphically, the technologies’ total scores can be presented on one axis, with cost data on the other axis, in order to reveal (Pareto) “efficiency frontiers” consisting of “dominant” technologies, in the sense that, compared to them, no other technologies have both lower costs and higher benefits. The advisability of keeping costs separate from other benefit-related criteria (instead of combining them) is returned to in the section “How Should Cost or Cost-Effectiveness Be Included in MCDA?”

As an illustration, Figure 1 reproduces the “value for money chart,” including an efficiency frontier, from Golan and Hansen (2012), in the context of HTA in Israel. Applying this framework to prioritize technologies involves decision makers, mulling over alternative affordable combinations to arrive at the “optimal portfolio” of technologies. Using a process of trial and error, decision makers should aim to maximize the aggregate benefits from the technologies chosen, subject to the budget constraint and given the technologies’ total cost, quality of evidence, and X-factors.

Open in new tab

Figure 1. Value for money chart, with 18 illustrative technologies for Israel.

Technology (indication, number of potential patients): t1 = smoking-cessation drugs (smokers, 6,000), t2 = Taxotere (head and neck cancer, 200), t3 = Herceptin (breast cancer—adjuvant treatment, 700), t4 = Elaprase (Hunter syndrome, 3), t5 = Visudyne (age-related macular degeneration, 1,050), t6 = left-ventricular assist devices (terminal heart failure, 12), t7 = statins (hypercholesterolemia, 5,600), t8 = pain relief (neuropathic pain, 14,250), t9 = Revlimid (multiple myeloma—3rd-line treatment, 200), t10 = dental care (children, 20,000), t11 = growth hormone (short-statured children, 3,900), t12 = Avastin [Bevacizumab] (colon cancer, 700), t13 = overactive bladder drugs (urinary urge incontinence, 21,000), t14 = Fuzeon (HIV, 45), t15 = long-acting insulins (diabetes, 10,000), t16 = contraceptives (adolescent girls, 20,000), t17 = Erbitux (colon cancer, KRAS mutation negative, 210), t18 = Humira (psoriatic arthritis, 60).

Source: Golan & Hansen (2012, Figure 2).

Finally, it is worthwhile to emphasize that MCDA is intended to serve as a tool to help people reach a decision—their decision (made by humans), not the tool’s decision. Decision makers need to understand the MCDA results, including any significant limitations of the analysis, and use the results, to a greater or lesser extent, to support them in reaching their decision. Where appropriate, the MCDA results can be used to communicate and to justify the final decision to stakeholders.

Which Scoring and Weighting Methods Should Be Used?

As summarized in Table 2, a potentially bewildering array of scoring and weighting methods is available for users to choose from. All methods (and the software implementing them) have their relative strengths and weaknesses—such that choosing the “best” MCDA method is itself a multi-criteria decision problem!

Issues to consider when thinking about which methods to use include: how well the methods elicit trade-offs between criteria, the time and resources required to implement alternative methods, the cognitive burden imposed on participants, whether skilled facilitators are required, the need for additional data processing and statistical analysis, the validity of the underlying assumptions with respect to decision makers’ preferences, and whether the outputs produced will satisfy decision makers’ objectives (De Montis, De Toro, Droste-Franke, Omann, & Stagl, 2004).

It is beyond the scope of this article to canvass how the methods summarized in Table 2 vary across these dimensions. Confronting these issues is, however, a priority if healthcare decision makers are to have confidence in the methods used, and guidance for practitioners in these respects would be useful. These issues are not unique to healthcare decision-making; they are common to MCDA applications in other sectors, too, such as the environment, project management, and the military. Therefore, it is likely that results and expertise are available and transferable from other sectors to healthcare.

Drawing lessons from elsewhere would provide a good starting point for health MCDA research, but doing so is unlikely to be sufficient. Methodological and normative issues peculiar to healthcare decision-making will still remain, as discussed later. First, though, arguments recently advanced against using MCDA for HTA and counterarguments are considered.

Arguments About Using MCDA for HTA, and Counterarguments

As already noted, MCDA methods have been used for a wide range of health decision-making other than HTA—e.g., prioritizing patients for treatment, disease classification, R&D prioritization, treatment decision-making, etc.—all of which appear to be generally well accepted. By contrast, opinion among health economists about the merits of using MCDA for HTA is sharply divided between advocacy and criticism, with criticism coming especially from researchers who support the use of conventional economic evaluation techniques—mostly cost-utility analysis—for HTA (Campillo-Artero, Puig-Juno, & Culyer, 2018). The concerns raised by health economists in this context are noteworthy because they appear to be unique to HTA; analogous arguments against using MCDA are largely absent from other nonhealth sectors. Therefore, it is worthwhile to consider the arguments commonly raised about using MCDA for HTA and the counterarguments.

The principal advantages of using MCDA for HTA include those of MCDA generally: structuring the decision problem leads to better decisions, avoiding the heuristics and other biases—e.g., relying on “gut feeling” (see Calnan, Hashem, & Brown, 2017, for a description of a NICE committee process)—that can affect complex decisions, especially ones involving uncertainty. Arguments specifically relating to the use of MCDA for HTA include that making the criteria being used in decision-making more explicit sends clearer signals to the health and life sciences industries about what society values. Furthermore, because MCDA makes the weights/trade-offs between different criteria explicit, this ensures more transparent and consistent decision processes among committees (e.g., NICE has four HTA committees sitting in parallel) and over time.

Of course, the arguments in favor of MCDA assume that it is desirable that decision makers are explicit, consistent, and transparent about the basis for their decisions. This assumption might be contested; for example, Mechanic (1997) argued against explicit approaches to “rationing” in healthcare, in favor of “muddling through elegantly” (p. 83), noting that “explicit rationing brings into a public forum conflicting needs and preferences, resulting in acrimony and political mobilization” (p. 86). Furthermore, decision makers may not want to be fully explicit about the basis for their decisions: they may fear that revealing the judgments exercised in each case may expose them to legal or constitutional challenges from external parties, particularly from those with vested commercial interests.

MCDA is also argued to facilitate greater participation in decision-making by stakeholders, and potentially to provide a more systematic means by which the preferences of patients or the general public can be taken into account. However, Walker (2016, p. 124) cautioned: “The scoring and weighting of factors is intended to stimulate debate, but it could also shut it down, while giving more power to the people preparing the scoring and weighting.” Similarly, Garratini and Padula (2018, p. 3) questioned “whether the scoring and weighting of all key factors necessarily stimulates more debate among decision makers (the best expected MCDA result) rather than strengthening the role of technicians who manage the procedures.” Such concerns can be expected to diminish as understanding of MCDA becomes more widespread (and better communicated to decision makers and stakeholders), including wider understanding of MCDA’s scoring and weighting methods (supported by software that is increasingly user-friendly).

The first argument against using MCDA for HTA is the claim that HTA fails to address the opportunity cost of resource use. For example, according to Campillo-Artero, Puig-Juno, and Culyer (2018, p. 147): “An efficient allocation of limited resources between alternative interventions cannot neglect opportunity cost. The multidimensionality of ‘value’ provides no protection from this since opportunity cost includes multidimensional benefits forgone.” The challenge of addressing opportunity cost is inherent in HTA; it is not unique to using MCDA to inform these decisions. HTA based on (conventional) economic evaluation also struggles to identify an appropriate cost-effectiveness “threshold” by which to judge value for money (Vallejo-Torres et al., 2016).

Also, if opportunity cost is defined purely in terms of quality-adjusted life years (QALYs) forgone (via a cost-effectiveness threshold), but other benefit criteria are taken into account qualitatively alongside QALYs in the appraisal of new technologies—e.g., as is the case in NICE’s use of social value judgments (NICE, 2008), alongside its cost-effectiveness threshold—then accounting for opportunity cost remains partial, nonsystematic, and potentially distortionary for allocative efficiency. (The issue of how to capture the opportunity cost of decisions in MCDA is also discussed in the section “How Can the Opportunity Cost of Decisions Be Captured in MCDA?”)

The second argument against using MCDA for HTA is that most MCDA methods currently advocated for use in HTA rely on weighted-sum models—whereby each alternative’s “performance” on the criteria is aggregated via a linear (i.e., additive) equation. Such models can be problematic if the criteria are misspecified such that they overlap (leading to double counting) or they are not “structurally independent.” Marsh, Sculpher, Caro, and Tervonen (2018) illustrated this issue with respect to EVIDEM, a MCDA approach designed and promoted specifically for use in healthcare decision-making (Goetghebeur et al., 2008), and which includes both cost-effectiveness and other cost and effectiveness criteria.

Weighted-sum models also present problems of preference dependence. A review by Marsh, Lanitis, and Neasham (2014) suggested that over 40% of MCDAs designed to support reimbursement decisions included both severity and health gain as criteria; but if health gain is valued more highly as the severity of a disease increases, this violates preference independence and means that weighted-sum models will not be appropriate. Of course, the above-mentioned shortcomings arise from misspecification of the criteria in weighted-sum models; this can be overcome by correctly specifying the criteria, as explained at step 2 of the MCDA process outlined earlier.

The third argument against using MCDA for HTA is that MCDA is “too complex” and risks reducing complex decisions into algorithms that are not well understood by decision makers. For example, Garratini and Padula (2018, p. 3) worried that MCDA’s “apparent numerical precision might be misleading for stakeholders, giving the false impression that the final results are objective numbers, scientifically produced.” As emphasized in the discussion of step 7 of the MCDA process, it is important that decision makers understand the MCDA results they use to communicate and to justify the final decision to stakeholders. Such understanding includes appreciating any significant limitations of the analysis, including the robustness of the results. MCDA’s methods are fundamentally no more complex and difficult to communicate to stakeholders than, for example, the methods underpinning cost-utility analysis (CUA), such as are used for calculating QALYs; it is just that MCDA is currently less well known than CUA in the health sector.

Moreover, the complexity inherent in HTA decision-making (in general) is a reason for use of MCDA to structure decision-making, in order to avoid the simplifying heuristics and biases known to affect complex decision processes. Arguably, MCDA has further advantages over relying purely on deliberative processes, given that HTA decision-making usually involves group/committee processes, where decisions may involve interpersonal tactics and strategies (see Calnan, Hashem, & Brown, 2017, for a description of a NICE HTA committee’s deliberation).

Issues Associated With MCDA for HTA

Conclusions

MCDA is well suited to supporting healthcare decision-making, including for HTA (and other health applications, too). In general terms, “good practice” when implementing an approach based on the explicit weighting of criteria and scoring of alternatives (weighted-sum models) includes: carefully structuring the decision problem being addressed, ensuring that appropriate criteria are specified, measuring alternatives’ performance accurately, using valid and reliable methods for scoring alternatives on the criteria and weighting the criteria, and presenting the MCDA results, including sensitivity analysis, to decision makers to support their decision-making.

Future research into MCDA methodological issues peculiar to healthcare would be worthwhile, including helping practitioners to select the most appropriate approaches, including scoring and weighting methods (supported by software), for particular types of applications. As the use of MCDA in healthcare increases, best practice guidelines will likely emerge to help address some of the challenges. A first attempt at such guidance has been provided by ISPOR (Marsh et al., 2016; Thokala et al., 2016), although the usefulness of the guidance offered is limited by its generality—ISPOR’s guidance did not address issues peculiar to using MCDA for HTA and budget allocation, where there seems to be divided opinion among health economists.

Finally, further research is required to evaluate the effectiveness of MCDA in healthcare decision-making. There are many examples of the exploratory use of a specific MCDA tool in various decision contexts, which have shown that the use of MCDA is feasible. However, studies to provide head-to-head comparisons of alternative MCDA approaches and methods are virtually nonexistent, as are formal evaluations of the benefits and costs of implementing MCDA. Such studies would provide a better evidence base for people and organizations considering using MCDA to strengthen their healthcare decision-making.