Show Summary Details

Page of

date: 27 May 2019

Multi-Criteria Decision Analysis (MCDA) in Healthcare Decision-Making

Summary and Keywords

Multi-criteria decision analysis (MCDA) is increasingly used to support healthcare decision-making. MCDA involves decision makers evaluating the alternatives under consideration based on the explicit weighting of criteria relevant to the overarching decision—in order to, depending on the application, rank (or prioritize) or choose between the alternatives. A prominent example of MCDA applied to healthcare decision-making that has received a lot of attention in recent years and is the main subject of this article is choosing which health “technologies” (i.e., drugs, devices, procedures, etc.) to fund—a process known as health technology assessment (HTA). Other applications include prioritizing patients for surgery, prioritizing diseases for R&D, and decision-making about licensing treatments. Most applications are based on weighted-sum models. Such models involve explicitly weighting the criteria and rating the alternatives on the criteria, with each alternative’s “performance” on the criteria aggregated using a linear (i.e., additive) equation to produce the alternative’s “total score,” by which the alternatives are ranked. The steps involved in a MCDA process are explained, including an overview of methods for scoring alternatives on the criteria and weighting the criteria. The steps are: structuring the decision problem being addressed, specifying criteria, measuring alternatives’ performance, scoring alternatives on the criteria and weighting the criteria, applying the scores and weights to rank the alternatives, and presenting the MCDA results, including sensitivity analysis, to decision makers to support their decision-making. Arguments recently advanced against using MCDA for HTA and counterarguments are also considered. Finally, five questions associated with how MCDA for HTA is operationalized are discussed: Whose preferences are relevant for MCDA? Should criteria and weights be decision-specific or identical for repeated applications? How should cost or cost-effectiveness be included in MCDA? How can the opportunity cost of decisions be captured in MCDA? How can uncertainty be incorporated into MCDA?

Healthcare Decision-Making and Multi-Criteria Decision Analysis

Most important decisions in the health sector involve confronting trade-offs between multiple, often conflicting, objectives, usually subject to a resource constraint (e.g., a fixed budget). An example is choosing which health “technologies” (i.e., drugs, devices, procedures, etc.) to fund—a process known as health technology assessment (HTA). Decision makers usually handle the trade-offs between objectives by evaluating the alternatives under consideration according to criteria relevant to the overarching decision—in order to, depending on the application, rank (or prioritize) or choose between the alternatives. For example, Golan, Hansen, Kaplan, and Tal’s (2011) review of the HTA literature identified criteria for prioritizing new health technologies associated with each technology’s clinical benefits and appropriateness, efficiency (including cost-effectiveness), patient need, and equality, solidarity, and other ethical or social values.

Decision makers can apply their criteria in either an implicit or an explicit fashion. In the last decade or so, multi-criteria decision analysis (MCDA)—sometimes also referred to as multi-criteria decision-making (MCDM)—has been increasingly used to support healthcare decision-making (Diaby, Campbell, & Goeree, 2013). In general terms, MCDA (or MCDM), a subdiscipline of operations research with foundations in economics, psychology, and mathematics, is concerned with decision-making situations in which multiple criteria are to be explicitly evaluated and, in effect, combined, in order to rank or choose between alternatives (Belton & Stewart, 2001).

Of course, applying criteria to evaluate alternatives is a natural decision-making approach as old as human history. However, “traditional” decision-making—and how most people make their everyday decisions, too—usually involves weighting the criteria and evaluating trade-offs in an intuitive or holistic manner. By contrast, MCDA is concerned with more formally structuring and solving decision problems, usually involving the explicit weighting of criteria and trade-offs between them.

A prominent example of an application of MCDA to healthccare decision-making that has received a lot of attention in recent years and that is the main subject of this article is HTA. For example, in 2012, the U.K. organization responsible for assessing the value for money of new medicines, the National Institute for Health and Clinical Excellence (NICE), considered the use of “structured decision-making” (i.e., MCDA) in its formal methods review (NICE, 2012), ultimately rejecting MCDA as not representing a clear improvement on existing deliberative processes. Several HTA bodies have piloted the use of MCDA, including, most recently, the Swedish Dental and Pharmaceutical Benefits Agency (Angelis, 2018). In 2014, the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) convened the MCDA Emerging Good Practices Task Force “charged with establishing a common definition for MCDA in healthcare decision-making and developing good practice guidelines for conducting MCDA to aid health care decision-making” (Thokala et al., 2016, p. 2; see also Marsh et al., 2016).

The recent advent of “value assessment frameworks” in the United States has increased interest in MCDA’s potential to help weigh up the competing aspects of value being considered (see Norman, Chalkidou, & Culyer, 2018, and other articles in the same issue of Value in Health). The value frameworks aim to support decision-making about new technologies in the U.S. health system and are intended to measure and to communicate the value of pharmaceuticals and other healthcare technologies for decision-making, using multiple attributes of value (Neumann, Willke, & Garrison, 2018). There has also been a flurry of articles (including this one!)—both in favor of MCDA (Drake, Trujillo de Hart, Monlean, Toto, & Valetim, 2017) and expressing doubts about its suitability (Campillo-Artero, Puig-Junoy, & Culyer, 2018)—and books (Marsh, Goetghebeur, Thokala, & Baltussen,2017) on the subject of MCDA in healthcare decision-making.

In addition to HTA, other examples of health MCDA applications include: prioritizing R&D portfolios in pharmaceutical companies (Phillips & Bana E Costa, 2007); allocating budgets across disease areas, such as the Health Foundation’s (2018) “socio-technical allocation of resources” (STAR) toolkit; prioritizing patients for elective (nonurgent) surgery (Hansen et al., 2012); disease classification and diagnosis (Johnson et al., 2014); prioritizing antibiotic-resistant diseases for R&D (Tacconelli et al., 2018); supporting patients and clinicians in selecting treatments (Dolan, 2008); and weighing up the benefits and risks of new medicines to support licensing decisions (Phillips, Fasolo, Zafiropoulos, & Beyer, 2011).

These examples are obviously quite different from each other; however, they all include the four key elements common to all MCDA applications in general: alternatives, criteria, weights, and decision makers. That is, all seven applications above (plus HTA) involve ranking or selecting alternatives—R&D investments, patients, diseases, treatments, or new medicines—based on the application of criteria and weights (representing the relative importance of the criteria) according to the preferences of decision makers (and, potentially, other stakeholders).

This article aims to explain the steps involved in a MCDA process, including an overview of methods, and to contribute to the research agenda for increasing MCDA’s use in healthcare decision-making. Because of HTA’s prominence—because all health systems have to grapple with choosing health technologies to fund—and the recent attention paid to HTA, most of the article’s discussion is in terms of HTA; nonetheless, the ideas discussed apply to other health MCDA applications as well, and they are also addressed.

The origins and emergence of MCDA are discussed briefly, and this is followed by a discussion of weighted-sum models, the main approach for health applications. Then, the main steps and methods available for implementing a MCDA process are explained. After consideration of the arguments against the use of MCDA for HTA and the counterarguments, methodological and normative issues associated with operationalizing MCDA for HTA are discussed. The article ends with a summary of conclusions.

Origins and Emergence of MCDA

Historically, the first example of a formal MCDA approach is usually recognized as being Benjamin Franklin’s “moral or prudential algebra” from 1772 (Franklin, 1973). Franklin’s approach, which he explained in a letter to his friend Joseph Priestley about how to make important decisions, involved successively trading off—in effect, weighting—the pros and cons of one alternative relative to another in order to identify the preferred one. Although this simple approach is effective for decisions involving just two alternatives, it does not scale up well for more alternatives.

More contemporary methods for choosing between more than just two alternatives (and involving multiple decision makers) were presented in Ralph Keeney’s and Howard Raiffa’s seminal book Decisions with Multiple Objectives: Preferences and Value Tradeoffs (Keeney & Raiffa, 1993), published first in 1976 and again in 1993. In 1979, Stanley Zionts helped popularize the abbreviation MCDM (for multiple criteria decision-making) with his article for managers: “MCDM—If not a Roman numeral, then what?” (Zionts, 1979). Other significant events from MCDM’s history are discussed in Köksalan, Wallenius, and Zionts, (2011).

Nowadays, MCDA is supported by specialized software (Oleson, 2016; Weistroffer & Li, 2016) that is often commercially available as a web application. The software is designed to assist decision makers “at various stages of the decision-making process, including problem exploration and formulation, identification of decision alternatives and solution constraints, structuring of preferences, and trade-off judgments” (Weistroffer & Li, 2016, p. 1302). MCDA software is discussed in the section “Steps in the MCDA Process.”

Consistent with the modern-day emergence of MCDA since the 1970s, MCDA has been widely used in many nonhealth sectors, such as energy, the environment, the military, management, and local and central government, for example. By contrast, healthcare decision makers have been relatively slow to adopt MCDA. However, increasing attention is being paid to MCDA now, although as far as health economists are concerned, the attention is sharply divided between advocacy for, and criticism of, using MCDA for HTA.

Weighted-Sum Models

Most health MCDA applications (and nonhealth MCDA applications) involve value-measurement approaches based on weighted-sum models—also variously known as additive, linear, scoring, point-count, and points models (or systems), or in the MCDA scientific literature as additive multi-attribute value models (Belton & Stewart, 2001). In short, such models involve decision makers explicitly weighting the criteria for the decision problem being addressed and rating the alternatives on the criteria, with each alternative’s “performance” on the criteria aggregated via a linear (i.e., additive) equation to produce the alternative’s “total score,” by which the alternatives are ranked relative to each other.

More specifically, an alternative’s score on each criterion (usually in the range 0 to 100), representing the alternative’s degree of achievement on the criterion, is multiplied by the criterion’s relative weight (summing across the criteria to unity), representing the relative importance of the criterion to the decision maker. These weighted scores are then summed across the criteria to produce the alternative’s total score (in the range 0 to 100). Based on their total scores, alternatives are ranked relative to each other.

A common equivalent representation of the linear equations described above is a schedule of “point values” for each criterion—a “points system”—where each criterion is demarcated into mutually-exclusive and exhaustive performance categories (e.g., low, medium, high). The point value for each category on a criterion represents the combined effect of the criterion’s relative importance (weight) and its degree of achievement as reflected by the category. (In other words, consistent with the linear equations described above, a category’s point value can be obtained by multiplying the weight for the criterion by the score for the category from the equation; likewise, a point value can be “decomposed” into the score on the criterion’s performance category and the criterion’s weight.) Each alternative is “scored” on the points system and the alternative’s point values for each criterion are summed across the criteria, again to produce a total score by which the alternatives are ranked. Arguably, a points system (a simple schedule of point values) is easier to implement manually than an equation (a set of weights and scores that need to be multiplied together).

In practical terms, a major attraction of weighted-sum models—in either their linear equation or points system forms—is the simplicity by which each alternative’s performance on each criterion is aggregated to produce a total score for each alternative. (Nonlinear functions for aggregating the criteria, including multiplicative functions, are also possible but are very rare.)

More importantly, such simple models have been found nearly universally in very many studies of health and nonhealth applications to be more accurate than the unaided (i.e., intuitive or holistic) judgments of decision makers (Kahneman, 2011). According to Hastie and Dawes (2010, p. 52), weighted-sum models are

surprisingly successful in many applications. We say ‘surprisingly,’ because many judges claim that their mental processes are much more complex than the linear summary equations would suggest—although empirically, the equation does a remarkably good job of ‘capturing’ their judgment habits.

Hastie and Dawes (p. 52) also explained that:

The mind is in many essential respects a linear weighting and adding device. In fact, much of what we know about the neural networks in the physical brain suggests that a natural computation for such a ‘machine’ is weighting and adding, exactly the fundamental processes that are well described by linear equations.

Finally, it is also worthwhile mentioning (mainly for completeness) that other MCDA-based approaches not based on aggregative functions for combining alternatives’ performance on criteria (i.e., via weights) are also potentially available as alternatives to weighted-sum models—albeit such alternative approaches are unlikely to be particularly useful.

The simplest alternative approach to weighted-sum models is a table—sometimes referred to as a “performance matrix”—for reporting the alternatives’ performance on the criteria (i.e., where each criterion is a column in the table/matrix). When one alternative dominates the others on all criteria, or where the trade-offs involved in selecting an alternative are clear and uncontroversial, decision makers can use such a table to reach their decision. Of course, most decision problems are more complicated than this! Most important health applications involve confronting nontrivial trade-offs between criteria; therefore, merely tabulating alternatives’ performance on the criteria is insufficient for resolving most decision problems (although such a table is useful at step 3 in the MCDA process outlined in the next section).

A more sophisticated MCDA approach capable of evaluating trade-offs between criteria, and that is not based on aggregative functions for combining alternatives’ performance on criteria (via weights), is the group of “outranking” methods, including PROMETHEE (Vincke & Brans, 1985), GAIA (Brans & Mareschal, 1994), and the ELECTRE family of methods (Roy, 1991). (The names are acronyms. PROMETHEE stands for Preference Ranking Organization METHod for Enrichment of Evaluations, GAIA stands for Geometrical Analysis for Interactive Aid, and ELECTRE stands for ELimination Et Choix Traduisant la REalité, which translates as “elimination and choice expressing reality”.) In essence, outranking methods involve decision makers’ pairwise rankings of alternatives relative to each other on each criterion in turn and then the combining of the pairwise-ranking results (but not via weights) in order to obtain a measure of support for judging each alternative as the top-ranked alternative overall. Despite their use for nonhealth applications, outranking methods are rarely used for healthcare decision-making—probably due to their complexities and the non-intuitive nature of their inputs and algorithms relative to weighted-sum models (Belton & Stewart, 2001).

Steps in the MCDA Process

Most MCDA applications are based on weighted-sum models. The steps for creating and applying weighted-sum models common to most applications are summarized in Table 1 and are discussed in turn next. As presented here, the steps are necessarily generic, given the breadth of possible health MCDA applications. Equivalent processes for specific applications are available in the literature, such as prioritizing patients for elective services (Hansen et al., 2012).

Although the steps are presented in sequence here, they do not necessarily need to be performed strictly in that order. Step 4 (scoring alternatives on the criteria) and step 5 (weighting the criteria) are intrinsically linked and can be implemented sequentially, simultaneously, or iteratively, depending on the methods employed, and so the two steps are explained together. Also, earlier steps, such as step 1 (structuring the decision problem) and step 2 (specifying criteria), can be revisited throughout the process as new insights into the particular application emerge and revisions and refinements become desirable.

Though, in principle, the steps can be performed “by hand” (e.g., supported by spreadsheets), many of them, and often entire processes, are supported by specialized MCDA software. Weistroffer and Li (2016) surveyed 69 examples of MCDA software, of which about a third (mostly web applications) are commercially available (Oleson, 2016). The software is especially useful for decision problems involving many alternatives and criteria and when the scoring (step 4) and weighting (step 5) methods used are technically sophisticated. MCDA software frees “the facilitator/analyst and decision maker from the technical implementation details, allowing them to focus on the fundamental value judgments” (Belton & Stewart, 2001, p. 345). Software capable of surveying potentially thousands of people is also useful for applications that involve eliciting and analyzing the preferences of members of the general population.

Table 1. Steps in the MCDA Process

Step

Brief Description

1. Structuring the decision problem

Identify objectives, alternatives, decision makers, any other stakeholders, and the output required.

2. Specifying criteria

Specify criteria for the decision that are relevant to decision makers (and, potentially, other stakeholders).

3. Measuring alternatives’ performance

Gather information about the alternatives’ performances on the criteria.

4. Scoring alternatives on the criteria

Convert performance measures into scores, representing each alternative’s degree of achievement on the criteria.

5. Weighting the criteria

Determine weights for the criteria, representing their relative importance to decision makers.

6. Applying scores and weights to rank alternatives

Multiply alternatives’ scores on the criteria by the weights and sum to get “total scores,” by which the alternatives are ranked.

7. Supporting decision-making

Use MCDA outputs, including sensitivity analysis, to support decision-making—i.e., ranking or selecting alternatives (depending on the application).

Step 1: Structuring the Decision Problem

The first step involves structuring and framing the decision problem being addressed. It is important to clarify the overarching objectives of the decision-making exercise, such as prioritizing technologies for funding, patients for surgery, diseases for R&D, treatments for licensing, etc. Related issues include, where possible, identifying the alternatives under consideration (e.g., technologies, patients, diseases, treatments, etc.), whether the decision is a one-off or repeated application, and the output required from the MCDA (e.g., a ranking or a selection).

These elements should be validated with stakeholders, who, depending on the application, may include patients, clinicians, ethics committees, and members of the general population. In some applications, the decision makers may be the stakeholders (e.g., patients choosing a treatment), whereas in other applications decision makers might be acting on behalf of others (e.g., the HTA committee making decisions in the interests of patients and taxpayers).

Step 2: Specifying Criteria

The second step involves specifying appropriate criteria for the decision: criteria that are valid, reliable, and relevant to decision makers and other stakeholders. For example, HTA authorization decisions (i.e., whether to allow a new pharmaceutical product to enter the market) may be informed exclusively by clinical outcomes related to treatment benefits and side effects. By contrast, patient prioritization decisions may incorporate a broader set of criteria, such as patient need and ability to benefit from treatment, etc.

The criteria should be specified without major overlaps (double-counting) or redundancy (irrelevant criteria; Belton & Stewart, 2001). As much as possible, criteria should be “structurally independent”—i.e., the range of possible ratings on a criterion should be independent of (i.e., not constrained by) ratings on other criteria (even if the criteria are correlated), and vice versa.

Depending on the application, the criteria can be identified from reviews of the literature and similar decisions, and from facilitated discussions, expert opinions, focus groups, and surveys. As in the previous step, stakeholders should be involved in identifying and validating the criteria.

Step 3: Measuring Alternatives’ Performances

Data about the alternatives’ performance in terms of each of the criteria—which can be presented in a “performance matrix” (table)—can be gathered in a variety of ways, ranging from expert opinions to rapid literature reviews, to full systematic reviews and modelling exercises.

The sophistication and intensity of the data-gathering activity depend on the availability of relevant evidence, the decision problem, and also other practical factors, such as the resources available for the job. MCDA is capable of combining quantitative and qualitative data, and also subjective judgments (in the absence of “harder” data) with more traditional scientific evidence in the same application.

Steps 4 and 5: Scoring Alternatives on the Criteria and Weighting the Criteria

Steps 4 and 5 are explained here together because they are intrinsically linked. They can be performed sequentially, simultaneously, or iteratively, depending on the application.

Scoring the alternatives on the criteria involves converting each alternative’s performance on each criterion into a numerical score. The scores are usually normalized so the worst performance on the criterion gets a score of zero and the highest performance gets 100. Scores can be implemented using a continuous scale (0 to 100), or, alternatively, two or more mutually-exclusive and exhaustive performance categories (e.g., low, medium, high) can be employed, each with a point value (e.g., medium = 60). Weighting the criteria involves determining their relative weights—normalized to sum across the criteria to unity—representing each criterion’s relative importance to decision makers.

The significance of the scoring and weighting steps is highlighted by recognizing that even if valid and reliable criteria have been specified (step 2) and the alternatives’ performances have been accurately measured (step 3), but their scores or the weights on the criteria are “wrong” (inaccurate), then, logically, the “wrong” decision, as determined by the ranking of the alternatives’ total scores, is almost certain. Thus, it is important that the alternatives’ scores and the criteria’s weights are represented as validly and reliably as possible.

A variety of scoring and weighting methods are available (Belton & Stewart, 2001), often supported by specialized software, as mentioned earlier. Many of the methods involve pairwise comparisons of alternatives and/or criteria and can be broadly classified as direct and indirect methods, respectively. Issues to consider when thinking about which methods to use when choosing from the potentially bewildering array of methods available are noted in the next section.

Direct methods for scoring alternatives involve decision makers directly expressing how they feel about the relative performance of each alternative on a criterion relative to the other alternatives. These expressions of a decision maker’s preferences are usually represented using either an ordinal scale (e.g., alternatives are rated on a 0 to 100 scale on a criterion) or a ratio scale measurement (e.g., “alternative A is three times as important as alternative B on a criterion”) from which scores are derived (i.e., directly). Similarly, direct methods for weighting the criteria involve decision makers directly expressing how they feel about the relative importance of the criteria, again usually represented in terms of either an ordinal scale (e.g., criteria are rated on a 0 to 100 scale) or a ratio scale measurement (e.g., “criterion A is three times as important as criterion B”) from which criterion weights are derived (directly).

In contrast to direct methods, indirect methods involve decision makers expressing their preferences usually by ranking or choosing between two or more alternatives—real or hypothetical—defined on some or all of the criteria (i.e., “partial” or “full profiles”). From these rankings or choices, scores and weights are derived (indirectly) using regression-based techniques or other quantitative methods.

Direct methods for scoring alternatives on the criteria and weighting the criteria include: direct rating and points allocation (Bottomley, Doyle, & Green, 2000), Simple MultiAttribute Rating Technique (SMART; Edwards, 1977), Analytic Hierarchy Process (AHP; Saaty, 1977), Simple MultiAttribute Rating Technique Exploiting Ranks (SMARTER; Edwards & Barron, 1994), swing weighting, and the “bisection” and “difference” methods (Winterfeldt & Edwards, 1986). Indirect methods include: conjoint analysis (or discrete choice experiments, DCE; Green, Krieger, & Wind, 2001, Ryan, Gerard, & Amaya-Amaya, 2007) and the Potentially All Pairwise RanKings of all possible Alternatives (PAPRIKA) method (Hansen & Ombler, 2008). Brief descriptions of these methods appear in Table 2, and more detailed information is available from the references cited.

Table 2. Direct and Indirect Methods for Scoring Alternatives on the Criteria and Weighting the Criteria

Direct Methods

Name

Description

Direct rating

(a) Scoring alternatives on the criteria

For each criterion, each alternative is rated on a point scale from 0 to 100—e.g., using a visual analogue scale (VAS)—in proportion to its performance on the criterion relative to the other alternatives. Alternatives’ scores on the criterion are normalized (or expressed directly) so that the lowest performer gets 0 and the highest performer gets 100 (as for the points allocation and SMART methods, below).

(b) Weighting the criteria

Each criterion is rated on a point scale from 0 to 100—e.g., using a VAS—in proportion to its importance relative to the other criteria. Weights for the criteria—normalized to sum across the criteria to unity (as for the other methods below, too)—are calculated from ratios of the ratings.

Points allocation

(a) Scoring alternatives on the criteria

For each criterion, a total of 100 points is allocated across the alternatives, in proportion to their relative performance on the criterion.

(b) Weighting the criteria

A total of 100 points is allocated across the criteria, in proportion to their relative importance. Weights for the criteria are calculated from ratios of the points.

SMART

(a) Scoring alternatives on the criteria

For each criterion, the lowest-performing alternative is identified and is given a value of 10 points. The other alternatives are rated relative to this alternative by also assigning points to them.

(b) Weighting the criteria

The least-important criterion is identified and is given a value of 10 points. The other criteria are rated relative to this criterion by also assigning points (of higher value) to them. Weights for the criteria are calculated from ratios of the points.

AHP

(a) Scoring alternatives on the criteria

For each criterion, the alternatives are pairwise compared and their “intensity of importance” relative to each other is expressed on a 1 to 9 ratio scale, usually represented as ranging from “equally preferred” (ratio 1) to “extreme importance” (ratio 9). Scores for the alternatives are calculated from ratios using eigenvalue analysis, normalized to sum across each criterion to unity (note, in this respect AHP is different from the other scoring methods here).

(b) Weighting the criteria

Each level in the hierarchy of criteria and subcriteria (and sub-subcriteria, etc.), as represented in a “value tree,” can be analyzed as a separate decision problem (and then combined multiplicatively). For each level, the criteria are pairwise compared and their “intensity of importance” relative to each other is expressed on the above-mentioned 1 to 9 ratio scale. Weights are calculated from ratios using eigenvalue analysis.

SMARTER

(a) Scoring alternatives on the criteria

SMARTER is not usually used for scoring alternatives on criteria.

(b) Weighting the criteria

The K criteria are ranked in order of their importance. The most-important criterion gets a value of 1, the second-most important criterion gets a value of 2, and so on down to a value of K for the least-important criterion. Weights for the criteria are calculated using the formula $wk=(1K)∑i=kK(1/i)$, where wk is the weight of the kth-ranked criterion, k = 1,2, . . . K (e.g., with four criteria, the weights are: 0.52, 0.27, 0.15, 0.06). Other, similar methods based on rank orders also exist (Riabacke, Danielson, & Ekenberg, 2012).

Swing weighting

(a) Scoring alternatives on the criteria

Swing weighting is not used for scoring alternatives on criteria.

(b) Weighting the criteria

For each criterion, the effects of a “swing” in performance from the worst to the best possible performance is evaluated. The criterion judged to be the most important in terms of the swing gets 100 points. The second-most important criterion is identified and is assigned points relative to the 100 points for the most important criterion. The exercise is repeated for the remaining criteria. Weights for the criteria are calculated from ratios of the points.

Bisection method

(a) Scoring alternatives on the criteria

For each criterion, the lowest- and highest-performing alternatives are identified and are rated 0 and 100, respectively. The performance on the criterion that is halfway between the two extremes is therefore worth 50 (midway between 0 and 100) and is defined. The next two midpoints on the criterion relative to the performances worth 0 and 50 and 50 and 100, respectively, are then defined. These two endpoints and three midpoints are usually sufficient to trace out the approximate shape of the criterion’s “value function.”

(b) Weighting the criteria

The bisection method is not used for weighting criteria.

Difference method

(a) Scoring alternatives on the criteria

For each criterion, the range of possible values for the performance on the criterion is divided into, say, four or five equal intervals (as noted, this method is for criteria that are quantitatively measurable and monotonic). The intervals are ranked in order of importance, thereby indicating the shape of the “value function” (e.g., concave or convex) so it can be traced out approximately.

(b) Weighting the criteria

The difference method is not used for weighting criteria.

Indirect Methods

Name

Description

Conjoint analysis (or DCE), regression-based

(a) Scoring alternatives on the criteria, and (b) Weighting the criteria

“Choice sets” comprising two or more hypothetical alternatives defined on the criteria are ranked, or the top-ranked alternative is identified. Usually, multiple decision makers are surveyed individually with respect to different choice sets. Weights for the criteria are calculated from the aggregated rankings across all participants using regression techniques, such as multinomial logit analysis and hierarchical Bayes estimation.

PAPRIKA

(a) Scoring alternatives on the criteria, and (b) Weighting the criteria

All pairs of real or hypothetically possible alternatives defined on two or more criteria at a time (where each criterion has performance categories) and involving a trade-off are pairwise ranked. Each time a decision maker ranks a pair of alternatives, all other pairs that can be pairwise ranked via transitivity are identified and eliminated, thereby minimizing the number of pairwise rankings explicitly performed. Criteria weights and scores for the criteria’s categories are determined from the explicitly ranked pairs using mathematical methods based on linear programming.

Step 6: Applying Scores and Weights to Rank Alternatives

Having scored the alternatives on the criteria and weighted the criteria, it is easy to calculate their “total scores.” For the direct methods and the PAPRIKA (indirect) method, each alternative’s scores on the criteria are multiplied by the weights, and then the weighted scores are summed across the criteria to get each alternative’s total score. For the conjoint analysis (or DCE) method, the regression technique estimates each alternative’s value (or “utility”) or its probability of being preferred by decision makers.

Step 7: Supporting Decision-Making

The MCDA results can be presented in tables or in graphs for decision makers to review. As for any analysis based on uncertain data inputs, it is important to check the robustness of the results via sensitivity analysis to plausible variations in alternatives’ performances (step 3), their scores on the criteria (step 4), and criteria weights (step 5).

In some applications, other considerations in addition to the MCDA results may also be relevant. For example, decisions with implications for healthcare budgets, such as HTA decision-making about new technologies, involve weighing up the benefits of each alternative, as represented by their MCDA total score, relative to their costs—and perhaps also considering technologies’ quality of evidence and additional “X-factors” not elsewhere included, such as strategic or legal factors (Golan & Hansen, 2012). Graphically, the technologies’ total scores can be presented on one axis, with cost data on the other axis, in order to reveal (Pareto) “efficiency frontiers” consisting of “dominant” technologies, in the sense that, compared to them, no other technologies have both lower costs and higher benefits. The advisability of keeping costs separate from other benefit-related criteria (instead of combining them) is returned to in the section “How Should Cost or Cost-Effectiveness be Included in MCDA?

As an illustration, Figure 1 reproduces the “value for money chart,” including an efficiency frontier, from Golan and Hansen (2012), in the context of HTA in Israel. Applying this framework to prioritize technologies involves decision makers mulling over alternative affordable combinations to arrive at the “optimal portfolio” of technologies. Using a process of trial and error, decision makers should aim to maximize the aggregate benefits from the technologies chosen, subject to the budget constraint and given the technologies’ total cost, quality of evidence, and X-factors.

Click to view larger

Figure 1. Value for money chart, with 18 illustrative technologies for Israel.

Technology (indication, number of potential patients): t1 = smoking cessation drugs (smokers, 6000), t2 = Taxotere (head and neck cancer, 200), t3 = Herceptin (breast cancer—adjuvant treatment, 700), t4 = Elaprase (Hunter syndrome, 3), t5 = Visudyne (age-related macular degeneration, 1050), t6 = left-ventricular assist devices (terminal heart failure, 12), t7 = statins (hypercholesterolemia, 5600), t8 = pain relief (neuropathic pain, 14,250), t9 = Revlimid (multiple myeloma—3rd-line treatment, 200), t10 = dental care (children, 20,000), t11 = growth hormone (short-statured children, 3900), t12 = Avastin [Bevacizumab] (colon cancer, 700), t13 = overactive bladder drugs (urinary urge incontinence, 21,000), t14 = Fuzeon (HIV, 45), t15 = long-acting insulins (diabetes, 10,000), t16 = contraceptives (adolescent girls, 20,000), t17 = Erbitux (colon cancer, KRAS mutation negative, 210), t18 = Humira (psoriatic arthritis, 60).

Source: Golan & Hansen (2012, Figure 2).

Finally, it is worthwhile to emphasize that MCDA is intended to serve as a tool to help people reach a decision—their decision (made by humans), not the tool’s decision. Decision makers need to understand the MCDA results, including any significant limitations of the analysis, and use the results, to a greater or lesser extent, to support them in reaching their decision. Where appropriate, the MCDA results can be used to communicate and to justify the final decision to stakeholders.

Which Scoring and Weighting Methods Should Be Used?

As summarized in Table 2, a potentially bewildering array of scoring and weighting methods is available for users to choose from. All methods (and the software implementing them) have their relative strengths and weaknesses—such that choosing the “best” MCDA method is itself a multi-criteria decision problem!

Issues to consider when thinking about which methods to use include: how well the methods elicit trade-offs between criteria, the time and resources required to implement alternative methods, the cognitive burden imposed on participants, whether skilled facilitators are required, the need for additional data processing and statistical analysis, the validity of the underlying assumptions with respect to decision makers’ preferences, and whether the outputs produced will satisfy decision makers’ objectives (De Montis, De Toro, Droste-Franke, Omann, & Stagl, 2004).

It is beyond the scope of this article to canvass how the methods summarized in Table 2 vary across these dimensions. Confronting these issues is, however, a priority if healthcare decision makers are to have confidence in the methods used, and guidance for practitioners in these respects would be useful. These issues are not unique to healthcare decision-making; they are common to MCDA applications in other sectors, too, such as the environment, project management, and the military. Therefore, it is likely that results and expertise are available and transferable from other sectors to healthcare.

Drawing lessons from elsewhere would provide a good starting point for health MCDA research, but doing so is unlikely to be sufficient. Methodological and normative issues peculiar to healthcare decision-making will still remain, as discussed later. First, though, arguments recently advanced against using MCDA for HTA and counterarguments are considered.

Arguments About Using MCDA for HTA, and Counterarguments

As already noted, MCDA methods have been used for a wide range of health decision-making other than HTA—e.g., prioritizing patients for treatment, disease classification, R&D prioritization, treatment decision-making, etc.—all of which appear to be generally well accepted. By contrast, opinion among health economists about the merits of using MCDA for HTA is sharply divided between advocacy and criticism, with criticism coming especially from researchers who support the use of conventional economic evaluation techniques—mostly cost-utility analysis—for HTA (Campillo-Artero, Puig-Juno, & Culyer, 2018). The concerns raised by health economists in this context are noteworthy because they appear to be unique to HTA; analogous arguments against using MCDA are largely absent from other nonhealth sectors. Therefore, it is worthwhile to consider the arguments commonly raised about using MCDA for HTA and the counterarguments.

The principal advantages of using MCDA for HTA include those of MCDA generally: structuring the decision problem leads to better decisions, avoiding the heuristics and other biases—e.g., relying on “gut feeling” (see Calnan, Hashem, & Brown, 2017, for a description of a NICE committee process)—that can affect complex decisions, especially ones involving uncertainty. Arguments specifically relating to the use of MCDA for HTA include that making the criteria being used in decision-making more explicit sends clearer signals to the health and life sciences industries about what society values. Furthermore, because MCDA makes the weights/trade-offs between different criteria explicit, this ensures more transparent and consistent decision processes among committees (e.g., NICE has four HTA committees sitting in parallel) and over time.

Of course, the arguments in favor of MCDA assume that it is desirable that decision makers are explicit, consistent, and transparent about the basis for their decisions. This assumption might be contested; for example, Mechanic (1997) argued against explicit approaches to “rationing” in healthcare, in favor of “muddling through elegantly” (p. 83), noting that “explicit rationing brings into a public forum conflicting needs and preferences, resulting in acrimony and political mobilization” (p. 86). Furthermore, decision makers may not want to be fully explicit about the basis for their decisions: they may fear that revealing the judgments exercised in each case may expose them to legal or constitutional challenges from external parties, particularly from those with vested commercial interests.

MCDA is also argued to facilitate greater participation in decision-making by stakeholders, and potentially to provide a more systematic means by which the preferences of patients or the general public can be taken into account. However, Walker (2016, p. 124) cautioned: “The scoring and weighting of factors is intended to stimulate debate, but it could also shut it down, while giving more power to the people preparing the scoring and weighting.” Similarly, Garratini and Padula (2018, p. 3) questioned “whether the scoring and weighting of all key factors necessarily stimulates more debate among decision makers (the best expected MCDA result) rather than strengthening the role of technicians who manage the procedures.” Such concerns can be expected to diminish as understanding of MCDA becomes more widespread (and better communicated to decision makers and stakeholders), including wider understanding of MCDA’s scoring and weighting methods (supported by software that is increasingly user-friendly).

The first argument against using MCDA for HTA is the claim that HTA fails to address the opportunity cost of resource use. For example, according to Campillo-Artero, Puig-Juno, and Culyer (2018, p. 147): “An efficient allocation of limited resources between alternative interventions cannot neglect opportunity cost. The multidimensionality of ‘value’ provides no protection from this since opportunity cost includes multidimensional benefits forgone.” The challenge of addressing opportunity cost is inherent in HTA; it is not unique to using MCDA to inform these decisions. HTA based on (conventional) economic evaluation also struggles to identify an appropriate cost-effectiveness “threshold” by which to judge value for money (Vallejo-Torres et al., 2016).

Also, if opportunity cost is defined purely in terms of quality-adjusted life years (QALYs) forgone (via a cost-effectiveness threshold), but other benefit criteria are taken into account qualitatively alongside QALYs in the appraisal of new technologies—e.g., as is the case in NICE’s use of social value judgments (NICE, 2008), alongside its cost-effectiveness threshold—then accounting for opportunity cost remains partial, nonsystematic, and potentially distortionary for allocative efficiency. (The issue of how to capture the opportunity cost of decisions in MCDA is also discussed in the section “How Can the Opportunity Cost of Decisions Be Captured in MCDA?”)

The second argument against using MCDA for HTA is that most MCDA methods currently advocated for use in HTA rely on weighted-sum models—whereby each alternative’s “performance” on the criteria is aggregated via a linear (i.e., additive) equation. Such models can be problematic if the criteria are misspecified such that they overlap (leading to double counting) or they are not “structurally independent.” Marsh, Sculpher, Caro, and Tervonen (2018) illustrated this issue with respect to EVIDEM, a MCDA approach designed and promoted specifically for use in healthcare decision-making (Goetghebeur et al., 2008), and which includes both cost-effectiveness and other cost and effectiveness criteria.

Weighted-sum models also present problems of preference dependence. A review by Marsh, Lanitis, and Neasham (2014) suggested that over 40% of MCDAs designed to support reimbursement decisions included both severity and health gain as criteria; but if health gain is valued more highly as the severity of a disease increases, this violates preference independence and means that weighted-sum models will not be appropriate. Of course, the above-mentioned shortcomings arise from misspecification of the criteria in weighted-sum models; this can be overcome by correctly specifying the criteria, as explained at step 2 of the MCDA process outlined earlier.

The third argument against using MCDA for HTA is that MCDA is “too complex” and risks reducing complex decisions into algorithms that are not well understood by decision makers. For example, Garratini and Padula (2018, p. 3) worried that MCDA’s “apparent numerical precision might be misleading for stakeholders, giving the false impression that the final results are objective numbers, scientifically produced.” As emphasized in the discussion of step 7 of the MCDA process, it is important that decision makers understand the MCDA results they use to communicate and to justify the final decision to stakeholders. Such understanding includes appreciating any significant limitations of the analysis, including the robustness of the results. MCDA’s methods are fundamentally no more complex and difficult to communicate to stakeholders than, for example, the methods underpinning cost-utility analysis (CUA), such as are used for calculating QALYs; it is just that MCDA is currently less well known than CUA in the health sector.

Moreover, the complexity inherent in HTA decision-making (in general) is a reason for use of MCDA to structure decision-making, in order to avoid the simplifying heuristics and biases known to affect complex decision processes. Arguably, MCDA has further advantages over relying purely on deliberative processes, given that HTA decision-making usually involves group/committee processes, where decisions may involve interpersonal tactics and strategies (see Calnan, Hashem, & Brown, 2017, for a description of a NICE HTA committee’s deliberation).

Issues Associated With MCDA for HTA

Notwithstanding the previous section’s ‘debate’ about using MCDA for HTA, the following five questions arise with respect to how MCDA is operationalized for HTA. The normative nature of the issues at the heart of these questions means there are unlikely to be definitively right or wrong answers. The most appropriate way of performing MCDA depends on the nature of the health system and the society within which decision-making is set.

Whose Preferences Are Relevant for MCDA?

In general, the answer to this question depends on the application. For example, in the case of shared patient-clinician decision-making, the preferences of the patient (“the principal”) are clearly the most relevant. By contrast, in the case of national HTA reimbursement decisions, for example, it is less obvious whose preferences should be used. One option is to rely on the preferences of the members of the HTA committee (“agents”), which can be obtained in various ways, such as via discussions during committee meetings or by eliciting committee members’ preferences in advance. Alternatively, criteria and weights could be based on the preferences of members of the general population, collected via surveys (Sullivan & Hansen, 2017).

A precedent for approaches based on population preferences emerges in the way that utilities representing health-related quality of life, as used to calculate QALYs for cost-utility analysis, are obtained. The rationale for considering the preferences of members of the general population is that they are both taxpayers and potential patients (i.e., funders and potential consumers of healthcare). Similarly, the preferences of patients, as another relevant stakeholder group, can be obtained. In practice, the above-mentioned options need not be mutually exclusive: evidence on the preferences of both the public and patients could be used to inform the decisions made by HTA committee members.

Should Criteria and Weights Be Decision-Specific or Identical for Repeated Applications?

HTA decisions may be made in different ways. For example, whereas NICE and PHARMAC consider different technologies at different points during the year, the Israeli Health Basket Committee has an annual process for assessing new technologies for funding. In the former case—which is the more common for HTA—a question arises about how many of the MCDA process steps should be undertaken specifically for each technology considered. For example, where MCDA has been used by the European Medicines Agency to decide whether a new medicine should be licensed, the benefits, risks, and weights applied are specific to each individual medicine considered (Phillips et al., 2011). The individual-technology focus is appropriate in that context, as the decision involves accepting (or rejecting) each medicine, one at a time, without considering the wider implications for resource allocation in the health system.

By contrast, most HTAs entail a series of repeated applications of the same set of criteria and weights across potentially many competing technologies, each of which has consequences for the allocation of resources across the health system. The need to consider opportunity cost in a systematic way, in the pursuit of allocative efficiency, implies that the criteria to be used, and their weights, should be the same across all decisions. However, in practice this is likely to be seen by some decision makers as too restrictive; for example, HTA committee members—and stakeholders, such as industry and patient groups—may prefer greater flexibility.

How Should Cost or Cost-Effectiveness Be Included in MCDA?

Including cost-effectiveness alongside effectiveness, cost, or indeed both, in the MCDA framework results in double-counting, as previously noted. Moreover, there are concerns about treating cost-effectiveness ratios as criteria per se and weighting them. Because a ratio can be formed from many different combinations of costs and health effects, it is not possible to capture preferences for cost and health effects via preferences for the ratio.

It is important to note that cost is not a value criterion, but a measure of what has to be given up to achieve the value criteria. In other words, as discussed earlier, cost should not be combined with other benefit-related criteria, but instead should be kept separate. As represented in Figure 1, a HTA framework based on explicitly considering value for money addressed this concern by comparing technologies’ costs against their health-related benefits, the latter comprised of multiple dimensions aggregated into a single measure using MCDA (Golan & Hansen, 2012).

How Can the Opportunity Cost of Decisions Be Captured in MCDA?

This question is related to the previous one, and also to the first argument against using MCDA for HTA. In the context of prioritizing health technologies subject to a budget constraint, the opportunity cost of the top-ranked technologies is obvious if all (competing) alternative “investments” are included in the group of technologies being considered. By contrast, if MCDA is instead undertaken to inform reimbursement decisions at a national level, it is not always possible to know which interventions would be forgone or disinvested locally in order to free up the necessary budget.

This problem may be addressed by identifying a threshold in terms of units of benefit-related value per unit of expenditure, where such a threshold represents the marginal efficiency of alternative investments when all relevant criteria are taken into account. Identifying such a threshold presents some challenges—although, arguably, the same challenges also exist in attempts to identify a threshold for decisions in terms of cost per QALY, as continues to be disputed with respect to the appropriate methods and evidence required to determine thresholds (Barnsley, Towse, Karlsberg Schaffer, & Sussex, 2013; Claxton et al., 2015). The key consideration for the achievement of allocative efficiency is that, if MCDA is used as a means of constructing an aggregate measure of incremental value, then opportunity cost needs to be considered relative to that measure.

How Can Uncertainty Be Incorporated Into MCDA?

Uncertainty is obviously relevant to healthcare decision makers—especially when decisions relate to new health technologies, where evidence may still be emerging about the benefits and risks and relative effectiveness or costs. When evaluating new technologies, and given the trend toward promoting earlier access to them, there are likely to be significant differences with respect to the quality of their evidence that should be taken into account in decision-making (Golan & Hansen, 2012). Uncertainty is often included as a criterion in health MCDAs (Garau & Devlin, 2017). However, uncertainty is not a criterion per se; instead, it is a measure of confidence that should be incorporated separately into the analysis (Marsh et al., 2016).

Conclusions

MCDA is well suited to supporting healthcare decision-making, including for HTA (and other health applications, too). In general terms, “good practice” when implementing an approach based on the explicit weighting of criteria and scoring of alternatives (weighted-sum models) includes: carefully structuring the decision problem being addressed, ensuring that appropriate criteria are specified, measuring alternatives’ performance accurately, using valid and reliable methods for scoring alternatives on the criteria and weighting the criteria, and presenting the MCDA results, including sensitivity analysis, to decision makers to support their decision-making.

Future research into MCDA methodological issues peculiar to healthcare would be worthwhile, including helping practitioners to select the most appropriate approaches, including scoring and weighting methods (supported by software), for particular types of applications. As the use of MCDA in healthcare increases, best practice guidelines will likely emerge to help address some of the challenges. A first attempt at such guidance has been provided by ISPOR (Marsh et al., 2016; Thokala et al., 2016), although the usefulness of the guidance offered is limited by its generality—ISPOR’s guidance did not address issues peculiar to using MCDA for HTA and budget allocation, where there seems to be divided opinion among health economists.

Finally, further research is required to evaluate the effectiveness of MCDA in healthcare decision-making. There are many examples of the exploratory use of a specific MCDA tool in various decision contexts, which have shown that the use of MCDA is feasible. However, studies to provide head-to-head comparisons of alternative MCDA approaches and methods are virtually nonexistent, as are formal evaluations of the benefits and costs of implementing MCDA. Such studies would provide a better evidence base for people and organizations considering using MCDA to strengthen their healthcare decision-making.

References

Angelis, A. (2018). Evaluating the benefits of new drugs in health technology assessment using multiple criteria decision analysis: A case study on metastatic prostate cancer with the dental and pharmaceuticals benefits agency (TLV) in Sweden. Medical Decision Making: Policy and Practice, 3(2), 1–17.Find this resource:

Barnsley, P., Towse, A., Karlsberg Schaffer, S., & Sussex, J. (2013). Critique of CHE Research Paper 81: Methods for the estimation of the NICE cost effectiveness threshold. Occasional Paper 13/01. London, U.K.: Office of Health Economics.Find this resource:

Belton, V., & Stewart, T. (2001). Multiple criteria decision analysis: An integrated approach. Norwell, USA: Kluwer Academic Publishers.Find this resource:

Bottomley, P., Doyle, J., & Green, R. (2000). Testing the reliability of weight elicitation methods: Direct rating versus point allocation. Journal of Marketing Research, 37(4), 508–513.Find this resource:

Brans, J.-P., & Mareschal, B. (1994). The PROMCALC & GAIA decision support system for multicriteria decision aid. Decision Support Systems, 12(4–5), 297–310.Find this resource:

Calnan, M., Hashem, F., & Brown, P. (2017). Still elegantly muddling through? NICE and uncertainty in decision making about the rationing of expensive medicines in England. International Journal of Health Services, 47(3), 571–594.Find this resource:

Campillo-Artero, C., Puig-Junoy, J., & Culyer, A. (2018). Does MCDA trump CEA? Applied Health Economics and Health Policy, 16(2), 147–151.Find this resource:

Claxton, K., Martin, S., Soares, M., Rice, N., Spackman, E., Hinde, S., . . . Sculpher, M. (2015). Methods for the estimation of the National Institute for Health and Care Excellence cost-effectiveness threshold. Health Technology Assessment, 19(14), 1–503, v–vi.Find this resource:

De Montis, A., De Toro, P., Droste-Franke, B., Omann, I., & Stagl, S. (2004). Assessing the quality of different MCDA methods. In M. Getzner, C. Spash, & S. Stagl (Eds.), Alternatives for environmental valuation, London, U.K.: Routledge.Find this resource:

Diaby, V., Campbell, K., & Goeree, R. (2013). Multi-criteria decision analysis (MCDA) in health care: A bibliometric analysis. Operations Research for Health Care, 2(1–2), 20–24.Find this resource:

Dolan, J. (2008). Shared decision-making—Transferring research into practice: The analytic hierarchy process (AHP). Patient Education and Counseling, 73(3), 418–425.Find this resource:

Drake, J., Trujillo de Hart, J.-C., Monlean, C., Toto, W., & Valetim, J. (2017). Utilization of multiple-criteria decision analysis (MCDA) to support healthcare decision-making FIFARMA, 2016, Journal of Market Access & Health Policy, 5, 1.Find this resource:

Edwards, W. (1977). How to use multiattribute utility measurement for social decision-making. IEEE Systems, Man and Cybernetics, 7(5), 326–340.Find this resource:

Edwards, W., & Barron, F. H. (1994). SMARTS and SMARTER: Improved simple methods for multiattribute utility measurement. Organizational Behavior and Human Decision Processes, 60(3), 306–325.Find this resource:

Franklin, B. (1973) [1772]. To Joseph Priestley. In W. Willcox & D. Bridgwater (Eds.), The papers of Benjamin Franklin: January 1 through December 31, 1772 (pp. 299–300). New Haven, CT: Yale University Press.Find this resource:

Garattini, L., & Padula, A. (2018). Multiple criteria decision analysis in health technology assessment for drugs: Just another illusion? Applied Health Economics and Health Policy, 16, 1–4.Find this resource:

Garau, M., & Devlin, N. (2017). Using MCDA as a decision aid in HTA for coverage decisions. In K. Marsh, M. Goetghebeur, P. Thokala, & R. Baltussen (Eds.), Multi-criteria decision analysis to support healthcare decisions. Berlin, Germany: Springer.Find this resource:

Goetghebeur, M., Wagner, M., Khoury, H., Levitt, R. J., Erickson, L. J., & Rindress, D. (2008). Evidence and value: Impact on DEcisionMaking—The EVIDEM framework and potential applications. BMC Health Services Research, 8(1), 270.Find this resource:

Golan, O., & Hansen, P. (2012). Which health technologies should be funded? A prioritization framework based explicitly on value for money. Israel Journal of Health Policy Research, 1, 44.Find this resource:

Golan, O., Hansen, P., Kaplan, G., & Tal, O. (2011). Health technology prioritization: Which criteria for prioritizing new technologies and what are their relative weights? Health Policy, 102(2–3), 126–135.Find this resource:

Green, P. E., Krieger, A. M., & Wind, Y. (2001). Thirty years of conjoint analysis: Reflections and prospects. Interfaces, 31(Suppl. 3), S56–S73.Find this resource:

Hansen, P., Hendry, A., Naden, R., Ombler, F., & Stewart, R. (2012). A new process for creating points systems for prioritising patients for elective health services. Clinical Governance: An International Journal, 17(3), 200–209.Find this resource:

Hansen, P., & Ombler, F. (2008). A new method for scoring additive multi-attribute value models using pairwise rankings of alternatives. Journal of Multi-Criteria Decision Analysis, 15(3–4), 87–107.Find this resource:

Hastie, R., & Dawes, R. (2010). Rational choice in an uncertain world: The psychology of judgment and decision making. Los Angeles, USA: SAGE.Find this resource:

Health Foundation. (2018). STAR: A tool for commissioners. (consulted December 10, 2018).

Johnson, S., Naden, R., Fransen, J., Van Den Hoogen, F., Pope, J., Baron, M., . . . Gabrielli, A. (2014). Multicriteria decision analysis methods with 1000Minds for developing systemic sclerosis classification criteria. Journal of Clinical Epidemiology, 67(6), 706–714.Find this resource:

Kahneman, K. (2011). Thinking, fast and slow (pp. 209–244). New York, USA: Farrar, Straus and Giroux.Find this resource:

Keeney, R., & Raiffa, H. (1993). Decisions with multiple objectives: Preferences and value tradeoffs. Cambridge, U.K.: Cambridge University Press.Find this resource:

Köksalan, M., Wallenius, J., & Zionts, S. (2011). Multiple criteria decision making: From early history to the 21st century. Singapore: World Scientific.Find this resource:

Marsh, K., Goetghebeur, M., Thokala, P., & Baltussen, R. (Eds.). (2017). Multi-criteria decision analysis to support healthcare decisions. Berlin, Germany: Springer.Find this resource:

Marsh, K., IJzerman, M., Thokala, P., Baltussen, R., Boysen, M., Kaló, Z., . . . Devlin, N. (2016). Multiple criteria decision analysis for health care decision making—Emerging good practices: Report 2 of the ISPOR MCDA Emerging Good Practices Task Force. Value in Health, 19(2), 125–137.Find this resource:

Marsh, K., Lanitis, T., & Neasham, D. (2014). Assessing the value of healthcare interventions using multi criteria decision analysis: A review of the literature. Pharmacoeconomics, 32(4), 345–365.Find this resource:

Marsh, K., Sculpher, M., Caro, J., & Tervonen, T. (2018). The use of MCDA in HTA: Great potential but more effort needed. Value in Health, 21(4), 394–397.Find this resource:

Mechanic, D. (1997). Muddling through elegantly: Finding the proper balance in rationing. Health Affairs, 16(5), 83–92.Find this resource:

National Institute for Health and Clinical Excellence. (2008). Social value judgements: Principles for the development of NICE guidance.Find this resource:

National Institute for Health and Clinical Excellence. (2012). Briefing paper for methods review workshop on structured decision making. Pp 274–354 in the collated documents relating to the NICE 2012 Methods Review.Find this resource:

Neumann, P., Willke, R., & Garrison, L. (2018). A health economics approach to US value assessment frameworks—Introduction: An ISPOR Special Task Force Report. Value in Health, 21(2), 119–123.Find this resource:

Norman, R., Chalkidou, C., & Culyer, A. J. (2018). A health economics approach to US value frameworks: Serving the needs of decision making. Value in Health, 21(2), 117–118.Find this resource:

Oleson, S. (2016). Decision analysis software survey. OR/MS Today, 43, 5.Find this resource:

Phillips, L., & Bana e Costa, C. A. (2007). Transparent prioritisation, budgeting and resource allocation with multi-criteria decision analysis and decision conferencing. Annals of Operations Research, 154(1), 51–61.Find this resource:

Phillips, L. D., Fasolo, B., Zafiropoulos, N., & Beyer, A. (2011). Is quantitative benefit–risk modelling of drugs desirable or possible? Drug Discovery Today, 8(1), e3–e10.Find this resource:

Riabacke, M., Danielson, M., & Ekenberg, L. (2012). State-of-the-art prescriptive criteria weight elicitation. Advances in Decision Sciences, 2012, 1–24.Find this resource:

Roy, B. (1991). The outranking approach and the foundations of ELECTRE methods. Theory and Decision, 31, 49–73.Find this resource:

Ryan, M., Gerard, K., & Amaya-Amaya, M. (Eds.). (2007). Using discrete choice experiments to value health and health care. The Netherlands: Springer.Find this resource:

Saaty, T. L. (1977). A scaling method for priorities in hierarchical structures. Journal of Mathematical Psychology, 15(3), 234–281.Find this resource:

Sullivan, T., & Hansen, P. (2017). Determining criteria and weights for prioritizing health technologies based on the preferences of the general population: A New Zealand pilot study. Value in Health, 20(4), 679–686.Find this resource:

Tacconelli, E., Carrara, E., Savoldi, A., Harbarth, S., Mendelson, M., Monnet, D., . . . WHO Pathogens Priority List working group. (2018). Discovery, research, and development of new antibiotics: The WHO priority list of antibiotic-resistant bacteria and tuberculosis. The Lancet Infectious Diseases, 18(3), 318–327.Find this resource:

Thokala, P., Devlin, N., Marsh, K., Baltussen, R., Boysen, M., Kaló, Z., . . . IJzerman, M. (2016). Multiple criteria decision analysis for health care decision making—An introduction: Report 1 of the ISPOR MCDA Emerging Good Practices Task Force. Value in Health, 19(1), 1–13.Find this resource:

Thokala, P., & Duenas, A. (2012). Multiple criteria decision analysis for health technology assessment. Value in Health, 15(8), 1172–1181.Find this resource:

Vincke, P., & Brans J-P. (1985). A preference ranking organisation method: The PROMETHEE method for multiple criteria decision-making. Management Science, 31(6), 647–656.Find this resource:

Walker, A. (2016). Challenges in using MCDA for reimbursement decisions on new medicines? Value in Health, 19(2), 123–124.Find this resource:

Weistroffer, H. R., & Li, Y. (2016). Multiple criteria decision analysis software. In S. Greco, M. Ehrgott, & J. Figueira (Eds.), Multiple criteria decision analysis: State of the art surveys. New York, USA: Springer.Find this resource:

Vallejo-Torres, L., García-Lorenzo, B., Castilla, I., Valcárcel-Nazco, C., García-Pérez, L., Linertová, R., . . . Serrano-Aguilar, P. (2016). On the Estimation of the Cost-Effectiveness Threshold: Why, What, How? Value in Health, 19(5), 558–566.Find this resource:

von Winterfeldt, D., & Edwards, W. (1986). Decision analysis and behavioral research. Cambridge, U.K.: Cambridge University Press.Find this resource:

Zionts, S. (1979). MCDM—If not a Roman numeral, then what? Interfaces, 9(4), 94–101.Find this resource: