Introduction

Medical devices (MDs) represent an important group of health technologies. In 2011 it was estimated that there were more than 200,000 MDs on the European market (Fraser, Krucoff, Brindis, Komajda, & Smith, 2011). However, MDs represent a very heterogeneous family of technologies consisting of both diagnostic and therapeutic devices and ranging from “low-risk” devices, such as scalpels, to “high-risk” devices, such as cardiac implantable electrical devices. While some devices require only very simplified assessment, others need to be assessed through a full evaluation of safety, efficacy, effectiveness, and cost-effectiveness.

Many of the early applications of economic evaluation in healthcare were related to the use of medical procedures involving devices, such as kidney dialysis (Klarman, Francis, & Rosenthal, 1968), or of expensive diagnostic devices, such as computed tomography (CT) scanners (Office of Technology Assessment, 1978). However, more recent literature in economic evaluation has focused largely on pharmaceuticals, reflecting the growth of formal requirements for new drugs to demonstrate cost-effectiveness in order to be reimbursed in many jurisdictions (Drummond, 2013). For example, in 2016 the National Institute of Health and Care Excellence (NICE) in the United Kingdom, one of the most active health technology assessment organizations worldwide, published 50 technology appraisals of pharmaceuticals, but only 3 appraisals of medical devices and 6 of diagnostic technologies.

This relative lack of economic evaluations of MDs, as compared with pharmaceuticals, has led some authors to question whether devices have any special characteristics that inhibit their assessment and whether, over time, the method guidelines for economic evaluation have become more suited to the assessment of pharmaceuticals (Tarricone, Torbica, & Drummond, 2017a). The challenges in conducting economic evaluation of medical devices are related to two major issues. The first lies in the policy domain and regards current regulatory requirements, which are typically less stringent than those for pharmaceuticals. As a consequence, evidence produced by manufacturers applying for marketing approval is often not sufficient for subsequent economic evaluation analysis. The second issue is purely methodological and reflects the ongoing debate on whether devices have distinctive features that require different or modified methods for economic evaluation to be applied properly. This article discusses the challenges in conducting economic evaluations of MDs from both policy and methodological viewpoints and provides some suggestions on how they might be addressed.

Regulatory Requirements for MDs

One critical issue relates to the fact the regulatory requirements for devices are typically less stringent than those for pharmaceuticals. In order to obtain approval to market, pharmaceuticals normally require two adequately controlled clinical trials. In contrast, the level of clinical evidence required for the marketing of MDs varies by the risk classification of the device. The risk classifications differ slightly by jurisdiction but generally divide devices into three or four groups (Tarricone, Ferré, Torbica, & Drummond, 2014).

Low-risk devices, such as crutches and scalpels, are generally exempt from premarket approval, although they need to be registered and are subject to labeling regulations. In the European Union this means obtaining a CE (Conformité Européenne) mark. Devices of medium risk, such as catheters and infusion pumps, do require premarket approval, and often the manufacturer is required to submit a dossier of relevant clinical and nonclinical data. The data required usually depend on the level of perceived risk. For example, in some jurisdictions (e.g., the European Union), orthopedic implants are classified as “medium risk” but manufacturers would be expected to provide some clinical data and also establish a registry so that the performance of the device could be monitored after it enters the market.

Finally, high-risk devices such as coronary stents, cardiac pacemakers, and implantable defibrillators always require clinical data, although the nature of that data may differ by jurisdiction. In the United States, the Food and Drug Administration often requires a randomized controlled trial (RCT), similar to its requirement for pharmaceuticals, whereas in the EU it may be possible to obtain market approval by conducting an observational study (e.g., a clinical registry). This was the case for transcatheter aortic valve implantation (TAVI), where an RCT was conducted in the United States but an observational study was established in Europe (Tarricone & Drummond, 2011).

Another feature of the regulation of MDs that differs from pharmaceuticals is that manufacturers can claim that their device is “substantially equivalent” to another marketed device for which clinical data are available. An example of this approach is the 510(k) process in the United States (Sorenson & Drummond, 2014). This can greatly assist some manufacturers (known as “fast followers”) but has the effect of not allowing “data exclusivity” to the manufacturer of the original device, as is the case for pharmaceuticals. This can have the effect of reducing the incentives for manufacturers to invest in clinical research, if that research is not mandatory, since the data generated can also be used by their competitors.

A major impact of the current processes for regulating devices is that there is a relative lack of clinical data at the time of product launch, although there remains the possibility of further clinical data collection while the device is in regular clinical use. This limits the possibilities for conducting economic evaluations early in the life cycle of the technology, especially at the time of product launch. Therefore, it has been argued that a different strategy may have to be followed for conducting economic evaluations of MDs, focusing more on assessments in the postmarketing phase (Tarricone et al., 2014). However, conducting economic evaluations in real-life settings poses methodological challenges in the analysis of observational data, which are subject to numerous biases. Furthermore, evidence synthesis becomes even more demanding when data need to be synthesized across studies with different designs (Schnell-Inderst et al., 2017).

Special Characteristics of MDs

It has been suggested that a number of special characteristics of MDs pose additional challenges for their clinical and economic evaluation (Drummond, Griffin, & Tarricone, 2009). These include the fact that (1) there is often an interaction between the device and the user (e.g., a surgeon), meaning that the performance of the device may change over time (the so-called “learning curve”), as a given user gains more experience, (2) MDs are characterized by incremental innovation, where small improvements are made, either through modifications to an existing device or the introduction of new products, (3) the introduction of MDs can have a substantial organizational impact by requiring training or investments in infrastructure, and (4) the pricing of MDs is more dynamic than that for other health technologies because of the extensive use of competitive tendering in the procurement process. In particular, the learning curve may have substantial effects on both effectiveness and costs. A more experienced surgeon may obtain better clinical results than one performing a procedure for the first time. In turn, better clinical outcomes might reduce the total length of hospital stay or the need for subsequent procedures, which could also have an impact on costs. Also, a more skilled surgeon may perform the initial procedure more quickly, which could reduce the amount of expensive operating theatre time required.

There is some evidence to support these assertions. For example, the learning curve for MDs has been observed in several clinical trials. In the CLASICC Trial of laparoscopic-assisted colorectal surgery, the proportion of patients requiring “revisions” (i.e., repeat procedures) fell from 38% in the first year of the trial to 16% in year six (Guillou et al., 2005). Similarly, in a clinical trial of spinal cord stimulation versus percutaneous myocardial laser revascularization for patients with angina, improvements in the outcome from use of the new procedure caused the estimated incremental cost-effectiveness ratio (ICER) to fall from £230,000 per quality-adjusted life-year (QALY) in 2000/1 to £18,000 per QALY in 2002/3 (Dyer et al., 2008). Learning curve effects have also been observed in the use of procedures in regular clinical practice. Using individual patient level data obtained from a hospital discharges database in Germany over the period 2006–2013, Varabyova, Blankert, and Schreyögg (2017) found that, over time, both the in-hospital mortality and length of stay became lower for patients undergoing endovascular aneurysm repair (EVAR), implying that the impact of the learning curve on cost-effectiveness can be more substantial than that on effectiveness alone. Conversely, there was no similar learning effect for a related procedure, fenestrated endovascular repair (fEVAR), possibly because much of the “learning” from performing EVAR procedures was transferable to fEVAR.

The issue of incremental innovation was raised in a clinical and economic evaluation of stapled hemorrhoidectomy conducted for the National Institute for Health and Care Excellence (NICE) in the United Kingdom (Burch et al., 2008). The researchers found that, although clinical trials comparing staple guns versus conventional surgery did exist, most of the trials were conducted using older versions of the gun, which had undergone several modifications. However, it could be argued that as long as the modifications represented improvements to the device’s performance and the learning curve in using the new version of the gun was not substantial, it would be conservative to use the older clinical evidence in the economic evaluation. Nevertheless, in some cases, using clinical data from an older version of a technology could have a substantial impact on the results of the economic evaluation.

Organizational impacts have been observed but not examined quantitatively. For example, Tarricone and Drummond (2011) discussed the fact that adoption of TAVI in a given hospital would normally involve investments in infrastructure and training. Because of the complexity and high risk of patients, TAVI would ideally need to be performed in a hybrid setting with equipment and facilities including both operating rooms and catheterization laboratories (cathlabs). Alternatively, it could be delivered either in operating rooms with adequate visualization equipment or in sterile cathlabs. This remains a challenge today, since “pure” hybrid operating rooms or appropriately equipped operating rooms and cathlabs are the norm only in the most advanced hospitals and would need to be developed further if TAVI were to be more widely diffused. Logistics and multidisciplinary team approaches also represent an organizational issue for many hospitals and must be assessed against investment costs in training and equipment. Moreover, a minimum number of cases needs to be planned in order to secure the return on the investment and, more importantly, the sufficient level of performance.

The impact of price changes on the cost-effectiveness of MDs was illustrated in the technology appraisals conducted on drug-eluting stents (DES) by NICE in the United Kingdom. In the initial appraisal, conducted in 2004, NICE found that DES were cost-effective (according to its threshold of £20,000 per QALY gained) for patients with a high risk of restenosis. However, when NICE reappraised DES in 2008, the price of bare metal stents (the alternative technology) had fallen substantially, pushing the estimated incremental cost-effectiveness ratio (ICER) above NICE’s threshold (Drummond et al., 2009).

Taken together, these special characteristics can influence the planning and analysis of cost-effectiveness studies. For example, the potential existence of a learning curve suggests that the cost-effectiveness of MDs should be studied over a period of time, as the results when the device is first introduced may be misleading. The potential existence of incremental innovation is a challenge for the clinical component of the assessment, as (in principle) separate clinical studies may need to be conducted for each version of the device. In addition, if the changes in the benefits to patients from using the new version of a device are small, these may not be reflected in changes in the generic instruments used for estimating the QALYs gained. Therefore, analysts wanting to estimate the value of these changes may need to estimate individuals’ willingness to pay or conduct a discrete choice experiment (Wilkinson & Drummond, 2015).

The potential for organizational impacts of devices suggests that the perspective for costing needs to be broad, covering a wide range of hospital departments and possibly also primary care, if use of the device changes the pattern of care. In addition, implementation costs, such as training and the provision of additional support services, may also need to be considered. Finally, the possibility of dynamic pricing suggests that a sensitivity analysis should be performed on both the acquisition cost of the new device and the one that it is replacing.

A more fundamental point relating to the special characteristics of MDs is that, as compared with drugs, the real-world cost-effectiveness may differ substantially from that observed within the context of controlled clinical trials. This suggests that the conduct of economic evaluations might be better focused on the postmarketing rather than the premarketing phase, as in the case of pharmaceuticals.

On the other hand, it has been argued that, to the extent that these differences exist between MDs and other technologies (in particular pharmaceuticals), the differences are only a matter of degree (Taylor & Iglesias, 2009). For example, over time physicians may learn how to prescribe a drug more precisely in order to obtain the optimum balance between efficacy and adverse events, and new modes of administration of a given drug (e.g., oral instead of intravenous infusion) might be viewed as incremental innovations or may lead to shifts in care from an inpatient to an outpatient setting with the associated changes in infrastructure. Furthermore, tendering or price competition applies to pharmaceuticals in some jurisdictions (e.g., US managed care), and there are tendering processes for vaccines in many jurisdictions. Nevertheless, the special characteristics mentioned above are likely to have a greater impact in the economic evaluation of devices than other health technologies, so they need to be considered.

Another important characteristic is that many MDs, such as CT scanners, are diagnostic. The economic evaluation of these technologies presents three specific challenges. First, the major benefit arising from diagnostic technologies is their ability to improve treatment choices and hence final health outcomes. Therefore, an accurate diagnostic test is only as good as the treatment that follows it. This means that the economic evaluation of a diagnostic technology usually encompasses the evaluation of the associated treatment technologies, with all the inherent challenges of conducting the required clinical evaluations.

Second, although the major economic benefit of diagnostic technologies results from the improvement in treatment outcomes, there may be value in just knowing the accurate diagnosis, both to the patient and the physician, even if this does not affect the immediate treatment choices. In some cases there may be value from the reassurance offered by a diagnostic test. Although these aspects of value could, in principle, be detected by the standard measures of outcome used in economic evaluations, such as the QALY, on occasions these measures may lack the adequate sensitivity. Therefore, it might be necessary to use alternative measures of value, such as willingness to pay (Lin, Cangelosi, Lee, & Neumann, 2013).

Third, there are indivisibilities in the use of large diagnostic MDs such as scanners. This means that not only is it necessary to show individual indications for use of the scanner, but also it is necessary that the totality of the workload required to occupy the scanner results in a cost-effective use of resources. If there were only a few potentially cost-effective indications for the scanner, this would not justify the large capital expenditure.

Initiatives to Improve the Economic Evaluation of MDs

Managing the Diffusion of MDs

Understanding the factors that lead to the adoption of MDs is especially important to healthcare decision-makers because the regulatory barriers to market access are considerably lower for MDs than for pharmaceuticals. A number of studies have shown that the utilization of MDs varies substantially, both within and between countries (Valzania, Torbica, Tarricone, Leyva, & Boriani, 2015). In a recent study, Torbica, Banks, Valzania, Boriani, and Fattore (2017) used a panel data regression model to investigate the variation in implant rates for cardiac implantable electrical devices across 57 regions in 5 EU countries. By analyzing a total of 1,330,098 hospitalizations extracted from discharge databases in Austria, England, Germany, Italy, and Slovenia from 2008 to 2012, they found implant rates were positively associated with higher levels of tertiary education among the labor force and the percentage of population over 74 years old. Regional per capita GDP and the number of implanting centers appeared to have no significant effect. There were also significant country effects in some of the analyses, suggesting that institutional factors play a role.

In another recent study, Hatz, Schreyögg, Torbice, Oriani, and Blankart (2017) investigated the characteristics of organizations (e.g., hospitals) and individuals (e.g., clinicians) who are more inclined to adopt and utilize cardiovascular devices based on a comprehensive analysis of environmental, organizational, individual, and technological factors. Seven random intercept hurdle models were estimated using data obtained from 1,249 surveys completed by members of the European Society of Cardiology. They found that better (device) manufacturer support increased the adoption probability of new cardiac devices (defined by CE mark approval dates) and that budget pressure increased the adoption probability of older devices. They argue that the role of manufacturer support should be investigated in more detail in order to assess whether it functions as a substitute for medical evidence on new devices and to gain insights into its relationship with clinical and cost-effectiveness.

This goes to the heart of the matter. Namely, if wide variation is observed in the utilization of MDs across countries and regions, is the level of utilization in each jurisdiction “appropriate” (defined in terms of clinical and cost-effectiveness), or are devices being overused in some jurisdictions and inappropriately restricted in others? Ideally, one would investigate the link between the use of the technologies and the patient outcomes obtained, but this is not possible using routinely available data. Alternatively, it may be possible to investigate whether physicians’ practices are in line with accepted clinical guidelines, although with a few notable exceptions (e.g., NICE’s clinical guidelines in the United Kingdom) these do not consider cost-effectiveness.

Another approach would be to use economic evaluation to manage the diffusion of MDs by conducting studies not only at market entry but also in the postmarket phase. In recent years there has been a growth in managed entry agreements that are collectively known as performance-based risk-sharing arrangements (PBRSAs). These schemes are known by different names and take slightly different forms in different jurisdictions, such as “coverage with evidence development” in the United States, “field evaluations” in Canada, and “only with research” in the United Kingdom. The basic approach is that new technologies are reimbursed by the healthcare system on the condition that further research is conducted into the clinical and/or cost-effectiveness of the technology. This approach is particularly well suited to MDs, since they are less well studied in clinical trials in the premarket phase and, because of the potential for a learning curve and incremental innovation, it is likely that clinical and cost-effectiveness in regular clinical practice will differ substantially from that observed in an experimental setting such as a controlled clinical trial.

Garrison et al. (2013) reviewed PBRSAs existing in a number of countries. Under their working definition, a PBRSA exhibits the following key characteristics:

Examples of PBRSA for procedures involving devices initiated in the United States include those for positron emission tomography for cancer diagnosis, percutaneous transluminal angioplasty and stenting for secondary prevention of stroke, and spinal cord stimulation for failed back surgery syndrome (Garrison et al., 2013). In another example of a PBRSA in Ontario, the generalizability of existing RCTs of DES conducted in the United States was questioned. Therefore, a pragmatic registry of all patients receiving DES was established in order to conduct a “field evaluation.” Coverage was provided for the stents provided under the registry. It was found that DES were more effective only in patients at high risk of stenosis (e.g., those with diabetes or particularly long or narrow lesions). Since this represented about 30% of the whole patient population, it was argued that this policy saved between $35 million and $58 million, as compared with the potential uncontrolled adoption of DES (Goeree et al., 2009; Levin, Goeree, Sikich, & Jorgensen, 2007).

The key questions in using economic evaluation to manage the diffusion of MDs concern the timing of studies (e.g., before or after providing reimbursement) and the nature of the research (e.g., the parameters estimated and the methods used). Making a device available early can have benefits, as it allows patient access to potentially beneficial therapies. However, decisions about the use of the device when the evidence base is relatively immature can lead to clinical and financial risk. Also, once the device is generally available, the possibilities for further research to reduce uncertainty may be limited, either because manufacturers have fewer incentives to invest in further research or because patients and clinicians are unwilling to participate in studies, especially RCTs.

Health economists have explored the use of value-of-information analysis to determine the balance between the benefits of early access to a technology and the value of additional evidence to resolve uncertainty (Claxton et al., 2012). In a recent paper, Rothery et al. (2017) set out a framework for considering the issues relating to the uncertainty in the diffusion and use of MDs, considering the learning curve, incremental innovation, investment and irrecoverable costs, and dynamic pricing. They use this framework to illustrate the circumstances under which it might be preferable to restrict access to the device until further research establishes its value or to allow access as long as further research is conducted.

Rothery et al. (2017) also discuss the incentives for further research and how the value and cost of such research might be shared between the manufacturer and the healthcare sector. Drummond et al. (2016) argue that incentives to manufacturers to conduct research need particular attention, given the lack of data mentioned earlier. In a situation where a PBRSA has been established for the first device to market, this might provide an opportunity for different manufacturers to share the costs of the research, as opposed to most of the costs being borne by the first to market. For example, “fast follower” devices claiming substantial equivalence could be included in the scheme, thereby sharing the costs of the research and generating evidence on the comparative effectiveness of the different devices.

Concluding Remarks

This article outlines the key challenges in conducting economic evaluations of MDs and how they might be tackled. However, many methodological and policy challenges remain, and further research is required. First, approaches need to be developed to integrate regulatory and health technology assessment for MDs, so that future data needs can be anticipated when clinical studies are being planned. It is likely that the balance of data collection for the economic evaluation of MDs will be in the postmarketing phase. Therefore, effort needs to be made in the analysis of “real-world” costs and effects of devices in regular clinical use. Linked to this, attention needs to given to the design of clinical registries, so that these will also be useful for conducting economic evaluations as well as establishing the efficacy and safety of MDs.

Second, more study is required to explore the impact of the various distinctive characteristics of medical devices. In particular, it would also be useful to determine whether it is possible to categorize devices in terms of their likely learning curve, according to the types of technology (diagnostic or therapeutic; implantable or nonimplantable), the medical specialty, or the device risk class. In addition, there should be more quantitative assessments of the organizational impact of MDs.

Finally, further study is required of the various influences on the diffusion of devices, especially the motivational factors facing patients, clinicians, and the institutions in which they work. In turn, this could help improve the design and conduct of PBRSAs, which are likely to be central to the economic evaluation of MDs in the future.