Show Summary Details

Page of

date: 18 August 2019

# Considering Health-Systems Constraints in Economic Evaluation in Low- and Middle-Income Settings

## Abstract and Keywords

In order to secure effective service access, coverage, and impact, it is increasingly recognized that the introduction of novel health technologies such as diagnostics, drugs, and vaccines may require additional investment to address the constraints under which many health systems operate. Health-system constraints include a shortage of health workers, ineffective supply chains, or inadequate information systems, or organizational constraints such as weak incentives and poor service integration. Decision makers may be faced with the question of whether to invest in a new technology, including the specific health system strengthening needed to ensure effective implementation; or they may be seeking to optimize resource allocation across a range of interventions including investment in broad health system functions or platforms. Investment in measures to address health-system constraints therefore increasingly need to undergo economic evaluation, but this poses several methodological challenges for health economists, particularly in the context of low- and middle-income countries.

Designing the appropriate analysis to inform investment decisions concerning new technologies incorporating health systems investment can be broken down into several steps. First, the analysis needs to comprehensively outline the interface between the new intervention and the system through which it is to be delivered, in order to identify the relevant constraints and the measures needed to relax them. Second, the analysis needs to be rooted in a theoretical approach to appropriately characterize constraints and consider joint investment in the health system and technology. Third, the analysis needs to consider how the overarching priority- setting process influences the scope and output of the analysis informing the way in which complex evidence is used to support the decision, including how to represent and manage system wide trade-offs. Finally, there are several ways in which decision analytical models can be structured, and parameterized, in a context of data scarcity around constraints. This article draws together current approaches to health system thinking with the emerging literature on analytical approaches to integrating health-system constraints into economic evaluation to guide economists through these four issues. It aims to contribute to a more health-system-informed approach to both appraising the cost-effectiveness of new technologies and setting priorities across a range of program activities.

# Understanding Investment in New Health Technologies Against a Backdrop of Constrained Health Systems

Investment in health in low- and middle-income countries (LMICs) increasingly incorporates the formal assessment of the incremental cost-effectiveness of new health technologies.1 In recent years, there has been a recognition that the uptake and implementation of case detection, diagnostic, treatment, and care services may be constrained by underlying weaknesses in the health system (Govindasamy, Ford, & Kranzer, 2012; Govindasamy, Kranzer, & Ford, 2014). As a consequence, the effective delivery of new global health technologies may require additional measures to address health-system constraints. A number of the disease-focused global health funders, such as the Global Fund for Aids, Tuberculosis and Malaria and GAVI, the Vaccine Alliance have identified the need for such investments to enable the effective delivery of interventions and have opened new channels for financing health systems strengthening (HSS): HSS being defined as including actions and strategies that aim to improve the performance (access, coverage, quality, or efficiency) of delivery platforms, management systems, financing and governance of the health sector (Mikkelsen et al., 2017). Moreover, it is now technically feasible to assess optimal resource allocation across the delivery of multiple services and technologies simultaneously. For example, for diseases such as HIV and tuberculosis (TB), economic models that define optimal cost-effectiveness across packages of interventions are increasingly used to inform National Strategic Plans (Kelly et al., 2018; Kerr et al., 2015). This capability to consider the cost-effectiveness of multiple services to support national planning has also drawn attention to the need to simultaneously define optimal funding levels for underlying service delivery platforms and health systems.

In recent years analysts have explored methods to integrate assessments of HSS and/or health-systems constraints into either the economic evaluation of single technologies or broader packages of services. There has also been some exploration by economists as to whether health system considerations should be incorporated within the analysis or alongside economic evaluation in the process of priority setting. Considering health systems in both economic evaluation and priority setting processes presents several challenges. Fundamentally, once a health technology is jointly assessed with additional actions to address health-systems constraints, the framing of the analysis changes: it becomes a “strategy,” with both costs and impact rooted in the “real world” performance of the health sector (Vassall et al., 2016). Moreover, HSS and the constraints it addresses have a complex and often indirect relationship with multiple health outcomes simultaneously. The impact of some forms of system strengthening can be linked to specific health outcomes—for instance, supply-side pay-for-performance programs provide incentives for health workers to improve the delivery of specific health services (often related to maternal and child health services). In this case, it should be possible to measure the impact of HSS on specific health outcomes. However, others, such as investment in a pay-for-performance scheme that targets facilities broadly may improve a range of outcomes and also generate spillover effects (Borghi et al., 2015). Other HSS initiatives, such as improvements to pharmaceutical supply or health information systems, may have little direct effect on specific health outcomes but an indirect effect on a wide set of health conditions. Moreover, the impact of HSS (or the impact of health-systems constraints on new technology performance) may be highly context specific; evidence on the influence of health-systems constraints may also only become available after an initial decision to adopt a technology has been taken (Vassall et al., 2017). Thus, incorporating evidence of the effectiveness of investment in HSS in economic evaluation (or priority setting more broadly) can be complex, as the pathway of influence of health outcomes may be unclear (and difficult to measure), and/or apportioning investment to specific health outcomes can be empirically challenging. Finally, the economic analysis and broader priority setting process may need to involve and inform multiple payers involved in HSS, due to the institutional fragmentation of budgets within the health sector between disease programs and general health services, and between national governments and external funders.

This article aims to support those considering incorporating health-systems constraints or strengthening in economic evaluations used to set priorities in LMICs, in addressing these challenges. Economic evaluation methods for investment in HSS alongside new technologies and services have been explored in a range of conceptual, theoretical, methodological, and empirical work. However, this is a new field and is being approached from a number of different directions: by those interested in priority setting processes, those interested in understanding and measuring HSS outcomes, those interested in economic evaluation theory, and finally, those interested in applying economic evaluation and resource allocation models in LMICs both to inform disease-specific programs and health benefit packages. Our aim is to bring together and summarize this disparate corpus so as to structure and organize the empirical and methodological issues that analysts wishing to develop and apply economic evaluation to investments that include HSS need to consider.

# Constraints, Feasibility, and Health System Strengthening

## Understanding the Interface Between New Technologies and the Health System

A first step in integrating health-systems constraints with economic evaluation is to identify the dependencies between a new technology and the broader system that delivers it. It should be noted that we define “new technology” broadly as any change in the way a health service or good is delivered or a change in how a new health service or good is provided. One of the most commonly used frameworks for characterizing health systems is the “building blocks” framework, developed and applied by the World Health Organization (WHO) with the aim of promoting a common understanding of health systems (World Health Organization, 2007). The “building blocks” framework identifies six different functions of a health system: service delivery, health workforce, information, medical products and services, financing, and leadership and governance. All health systems must have the capacity to perform these functions. The “building blocks” conceptualization of a health system also recognizes that there is an interdependency among the six functions and a diversity of actors performing them, including suppliers, service providers, and the communities and households they serve.

Drawing on sociology and policy analysis, in addition to the “hardware” capacity of the health system outlined by WHO, others have highlighted the importance of understanding the “software” elements of the health system—the ideas, incentives, interests, values, norms, affinities, and power structures embodied within those structures (Sheikh et al., 2011). For some, the “software” of the health system is fundamentally embedded in the social and political context; and both health system organization and performance are therefore ultimately determined by culture and behavior. In a similar vein, economists have turned to principal-agent theory to analyze the health system, characterizing the actors and institutions that make up the system as being influenced by incentives, accountabilities, and information flows (Hanson, 2011, Pritchett, 2004). More recent conceptual developments emphasize the complex, dynamic nature of health systems in which the elements of the system interact in unpredictable ways, producing positive and negative spillover effects, and in which past behaviors influence current actions (de Savigny, 2009, Atun, 2012).

## Health-System Constraints and Feasibility

As the case for large-scale investment in essential health benefit packages in LMICs emerged in the early 2000s, health system analysts began to consider whether there were systemic constraints to scaling up services that would restrict a health sector’s ability to meet global health targets. Hanson et al. (2003) defined health-systems constraints as obstacles that “might affect the feasibility and returns from rapid expansion of health services” and classified these by the level at which they operated—from community and health service delivery level through to the public policies, such as civil service rules and regulations, that cut across sectors. Implicit was the idea that specific investments might be needed to relax these constraints to enable the delivery of priority health services. It was also recognized that some constraints might be related to the broader incentive and management environment and would need to be addressed through organizational change, rather than large-scale financial input.

This notion of constraints is closely related to work that emerged in the same period on priority setting that referred to the concept of “feasibility” (Baltussen et al., 2013; Gericke, Kurowski, Ranson, & Mills, 2005), as a consideration alongside other decision criteria such as cost-effectiveness. Feasibility is the term often used to justify investment (or lack of investment) decisions in specific health interventions and technologies, reflecting technological or external constraints affecting service delivery. As a term, feasibility is often not precisely defined in practice; however, depending on the circumstance, it is used to encompass a range of aspects of an intervention such as affordability, physical health-systems constraints that directly restrict access to services or technologies (human resources scarcity, barriers to use and uptake etc.), and possibly arbitrary beliefs held by policymakers and the wider environment that influences the priority-setting process (Guindo et al., 2012; Kim et al., 2006; Sendi & Briggs, 2001; van Baal, Thongkong, & Severens, 2016). Gericke et al. (2005) proposed that the “feasibility” of an intervention is determined by local institutional capacity (extrinsic to the intervention) and the degree of technical complexity of the intervention (intrinsic). Feasibility is thus the “match between technical complexity and capacity.” Several widely used classifications of decision criteria and constraints for priority setting incorporate both “feasibility” and “complexity.” Some allow these criteria to be distinct (Guindo et al., 2012; Tromp et al., 2018), while others combine them (Kapiriri & Martin, 2010).

As part of this literature, the concept of feasibility was specifically related to “health-system constraints” in the priority setting literature. Baltussen and colleagues define feasibility as the set of “constraints at the personal and health system level that may impede the implementation” of programs (Baltussen et al., 2013). Thus, the concepts of feasibility and constraints, emerging from two different schools of thought (health systems and priority setting), are intertwined in that “feasibility” as a criterion in priority setting is at least in part determined by the hardware and software of the health system.

## From Constraints to Health Systems Strengthening

While new technologies may be simply defined, identifying constraints to their performance and actions to address them may be complex. To assist analysts, several health system frameworks provide the starting point for the identification of actions and investment to improve access, coverage, quality or efficiency: the definition of health system strengthening (HSS) used by WHO. Depending on the health systems framework used and the constraint that is identified, investment in the “hardware” of human resources (ensuring adequate numbers and appropriate training of health workers) or logistics systems for procuring and distributing medicines may be enough to ensure the effective implementation of technologies and services. Other frameworks focus on “software”, for example the emphasis would be on investing in the correct incentives and working environment for nurses to enable them to provide quality care.

To help identify HSS investment to complement new health technologies, several tools, based on these frameworks, have been developed that can assist those working in economic evaluation to characterize health system and other implementation constraints around technology delivery and uptake. One framework developed by Vassall and colleagues illustrates the experience of a patient moving through a cascade of care (from case detection to eventual treatment outcome) and describes how the health system constrains that progression along the cascade and influences outcomes (see Figure 1). In this framework, constraints are characterized in two dimensions—the first that constrains the demand for health services and others that influence their supply. For both demand and supply there are proximal and distal constraints to patients accessing services along a cascade of care (Vassall et al., 2016). Proximal constraints on the supply side directly influence service availability and quality of care available: for example, they may result in a lack of resources or negatively influence the behavior of service providers. These, in turn, are determined by the distal health-systems constraints related to some of the “building blocks” and supply side incentives inherent in the prevailing organizational structure. For example, a lack of appropriate staff at the service delivery level may be driven by a lack of a sound human resource management system. On the demand side, patients are similarly resource and behaviorally constrained from accessing services appropriately, which in turn may be distally driven by underlying norms, values and education together with other aspects that influence demand (see Figure 1).

Similar frameworks have also been used to identify the HSS required to support disease programs. For example, UNAIDS encourages HIV programs to identify investment in new technologies and service delivery and in “critical enablers” that address both proximal and distal constraints to service scale-up and utilization (Schwartlander et al., 2011). In addition, there are also several management and program evaluation frameworks, which may not specifically incorporate a preconceived concept of the health system but reflect a broad understanding of interventions as complex and adaptative, such as program theory and realist evaluations (Mukumbang, Marchal, van Belle, & van Wyk, 2018; Mukumbang, van Belle, Marchal, & van Wyk, 2016; Mangham-Jefferies et al., 2014).

Click to view larger

Figure 1. Framework of Interventions including HSS.

Source: Vassall et al. (2016).

In some cases, the analyst may be presented with a clearly defined program of HSS to assess. However, in many cases, analysts may simply be required to consider the cost-effectiveness of reaching a specified coverage level of a new technology and therefore to reflect on whether this is feasible within the current health system infrastructure or whether investment in HSS might be needed. In both cases these frameworks can be used by economic evaluators to ensure that all incremental actions and resource use are identified and considered appropriately in their analysis.

# Theoretical Approaches to Incorporating HSS in Economic Evaluation

Once, the intervention or “strategy” has been defined, the usual starting point for economic evaluation is to frame the investment decision appropriately, and this will have consequences for how HSS is addressed in the evaluation. As outlined in the introduction there are two different types of investment decision for which the consideration of HSS is relevant. These are:

1. 1. For new technology, with complementary HSS:

Whether to invest in a new technology when HSS is required to ensure effective implementation—in other words, the investment decision incorporates both the technology and HSS

2. 2. For optimizing resource allocation:

How to optimally allocate the health sector budget/or program budget between service delivery (including new technologies) and health system (including HSS) investment?

It should be noted that, for decision type 1, in principle the extent to which HSS resource use is included depends not just on the intervention definition but also on the extent to which HSS is “incremental” between the intervention and comparator. As stated above, the analyst can use a framework to define both the intervention and comparator and to consider whether any action to address constraints (HSS) is incremental (Vassall et al., 2016). In practice, however, the scope of the decision and the intervention being evaluated is often determined by who is paying for it. Defining the scope of the evaluation may be more complex in a LMIC context where external funding may play a significant role; even in countries where donor funding is a small share of total health expenditure, it may be important for certain disease program. While many LMICs are setting up sector-wide HTA processes, in many cases analyses to inform decisions to adopt and invest in a new technology are still commissioned by vertical disease programs, with the objective of paying for a disease-specific outcome. This disease-focused payer perspective may determine which costs and outcomes are evaluated and which are considered to be spillovers or ignored entirely. This narrow payer perspective may lead to the exclusion of HSS costs thus biasing any assessment of cost-effectiveness viewed from a broad health sector perspective, that is more typical in high income countries.

Several theoretical approaches have been proposed that can be used as the basis for considering how to frame economic analyses to inform the different types of decisions above. Table 1 provides a summary of four theoretical approaches, each reflecting differing but not incompatible interpretations of the decision problem. Fenwick, Claxton, and Sculpher (2008) and van Baal, Morton, and Severens (2018) offer approaches for addressing decision type 1 and explore how implementation constraints and HSS can be incorporated in analyses of new technologies. Morton, Thomas, and Smith (2016) and Hauck et al. (2019) address decision type 2 and provide a framework for analyses to inform the optimal allocation between “service delivery” and HSS investment.

Each of the theories defines the health system/implementation constraints and their interaction with the service delivery in a slightly different way. None encapsulate the conceptual frameworks of the health system but approach the health system constraint as a singular entity.

Table 1. Summary of Different Theoretical Approaches

Approach

Decision problem

Interaction with other interventions

Fenwick (Fenwick et al., 2008)

Decision type 1 and 2.

Any

No explicit consideration of spill over outcomes of HSS (but also not ruled out)

Two stage decision: (1) approve technology; (2) approve costs to implement it at different coverage levels

Broad approach can be used for single or multiple intervention priority setting

van Baal (van Baal et al., 2018)

Decision type 1.

HS constraints leading to input constraints in the short run

NA

One stage decision: to invest in intervention or not, given current performance of the health system

Morton (Morton et al., 2016)

Decision type 2.

HSS to improve productivity (health benefits per investment) generically

Can be used for multiple interventions

One stage: to allocate investment across intervention and HSS

Hauck (Hauck et al., 2019)

Decision type 2.

HSS to improve productivity, but different models for: a) efficiency b) increasing capacity c) increasing reach

Can be used for multiple interventions

One stage: to allocate investment across intervention and HSS

Fenwick and colleagues address decision type 1 and the challenge of incorporating HSS into the decision to invest in a new technology, but in practice their approach may be used to address decision type 2 as well (Fenwick et al., 2008). They begin by conceptually distinguishing between the new technology and the constraints that limit its optimal implementation, dividing the decision to adopt new technologies into two stages. First, the decision maker decides whether the technology should receive guideline approval and so can be funded (in the case of the United Kingdom, this might mean eligibility for use in the National Health Service, or globally, that it is incorporated in WHO guidelines). Then there is a second decision to determine the appropriate investment to expand coverage of the new technology. Dividing the process in this way allows the “expected value of implementation” to be estimated using the simple method of valuing the health benefits achieved by moving from current to so-called perfect implementation (100% coverage at 100% of achievable efficacy). This value can then be compared with the costs of various strategies to achieve “perfect” implementation and thus the total net benefit of implementation can be estimated and compared between strategies. The broad approach can, in principle, also be applied to levels of implementation that are less than “perfect” and to reflect other definitions of “effective” implementation as defined by decision makers. Aside from its simplicity, an advantage of this approach is that it can be used in combination with analyses that estimate the value of additional information about the effectiveness of the technology during implementation. In doing so, this approach can also inform the decision of whether or not to invest in pilots, demonstration projects, and further evaluation before full scale-up.

Fenwick and colleagues’ approach in describing a two-stage decision also does not reflect a conceptual understanding of the intervention as a strategy that simultaneously includes both new technology and the actions to support its implementation. However, the approach can be applied to identify the maximum efficient level of expenditure on HSS to support effective implementation of specific new technologies. In addition, the value of implementation approach can be used to identify the maximum efficient level of expenditure for packages of services (decision type 2). For example, this approach was applied to define an essential health package for Malawi to estimate how the budget should be divided between service delivery and sector-wide HSS (Ochalek et al., 2018). Similar approaches have also been used to define the maximum threshold level of expenditure to support disease-specific services, for example, determining the level of expenditure on HSS required to improve adherence to HIV diagnosis and treatment (Working Group on Modelling of Antiretroviral Therapy Monitoring Strategies in Sub-Saharan et al., 2015).

Van Baal and colleagues also address the challenge posed by informing decision type 1, examining the specific case of incorporating non-financial constraints on inputs into the CEA of a new technology (van Baal et al., 2018). Here the decision problem is not divided into two stages but instead examined from the perspective of a decision maker who wishes to understand the value of implementing a new technology within a health system where some inputs are constrained—in the sense that the market will not supply the correct level of inputs at an allocatively efficient price in the short run. They illustrate their approach using the case where the availability of one input (human resources) is constrained in the short run due to the presence of both monopoly and monopsony in the health work labor supply. Van Baal and colleagues illustrate how, under set conditions, the cost-effectiveness of the investment may be assessed.

The authors begin by demonstrating that allocative efficiency will be achieved, in a two-intervention, two-input model, under a fixed health budget constraint when the allocation of resources is such that the marginal return on each intervention is equal. They then estimate the impact on optimal cost-effectiveness if one of the inputs required to produce both interventions is constrained and point out that, in effect, this situation is equivalent to having to two separate healthcare budgets—one for each input, each with its own threshold equivalent to its shadow price. For example, in a situation where human resources are constrained, investing in additional nurse time to provide an intervention will lead to higher health gains being foregone elsewhere than if the same amount is invested in other inputs that are unconstrained. Thus, constrained inputs have a higher shadow price than non-constrained inputs, and input constraints reduce the overall potential efficiency of the health sector.

Input constraints are empirically incorporated into the incremental cost-effectiveness ratio (ICER) by weighting the cost of constrained inputs to reflect their relative shadow price compared with all other inputs. The authors argue that if cost-effectiveness estimates do not incorporate this weighting, these will be biased, with the magnitude depending on the extent and the number of constraints.

Despite the theoretical clarity, this approach presents a substantial challenge given the difficulties in estimating opportunity cost–based thresholds generally. Here, van Baal and colleagues suggest a neat solution, albeit under the conditions of constant (linear) returns to scale for all inputs and an assumption that the current input mix used to deliver interventions reflects the efficient allocation of inputs. They posit that the ratio of the shadow price of the constrained input to non-constrained inputs can be identified as the ratio of the cost of the constrained input to the unconstrained input per healthcare unit produced observed during the delivery of the intervention. Their approach is applied to a stylized example of eye care in Zambia and used to estimate the extent to which the human resource constraint influences the potential health gains produced by the optimal intervention combination. This approach has also been applied to HIV treatment (Revill et al., 2018). While van Baal’s approach does not estimate the value of any specific HSS action per se, new technologies that rely less on constrained inputs will receive a higher cost-effectiveness ranking than those that rely heavily on constrained inputs. Thus, the approach can be used to correct for downward bias in ICERs of new technologies implemented in health systems with constraints on specific inputs.

Morton and colleagues examine optimal resource allocation across HSS and service delivery across health programs (i.e., decision type 2) (Morton et al., 2016). In this case, health-systems constraints are defined as anything that impedes optimal implementation of services and the decision maker can invest in HSS to relax these constraints. The starting point, as with Fenwick and colleagues is to therefore characterize HSS as an investment that enhances the effective implementation of services. The challenge Morton and colleagues address is how best to characterize the indirect influence of HSS, via effective service delivery, on multiple health outcomes.

Morton and colleagues consider the case where all HIV, TB, and Malaria services are delivered as part of vertical programs. Each of these vertical programs contains multiple services but share a common platform (defined as a horizontal program). Morton and colleagues propose a mathematical model with an objective function that maximizes health benefits, subject to a budget constraint, under a set of conditions: (a) that each service exhibits constant returns to scale; (b) that services are independent from one another; (c) that costs can be disaggregated into health system and service delivery costs; and (d) that all costs are incurred within a fixed period.

$Display mathematics$

$Display mathematics$

$Display mathematics$

$Display mathematics$

$Display mathematics$

Their model is captured in the above equation, which characterizes the production of health outcomes using a weighted (for each service) power term for the investment made in the platform. I is the index of projects, $vi$ terms are the benefits, $ci$ terms are the costs, $xi$ is the proportion of project i implemented and y represents expenditure on HSS. The effect of spending y is to scale the benefits by$yγ$, bounded by P and p. A value of $γ>1$ results in increasing returns to each additional HSS dollar spent, and of $γ<1$ decreasing returns to HSS.

The parameterization of the power term alters the extent to which the production function between HSS (strengthening the platform) and health outcomes is concave. As the parameter for the power term increases, so does the optimal proportion of HSS investment to the total investment. Characterizing the impact of HSS in this nonlinear objective function allows economies of scope to be reflected.

Finally, Hauck and colleagues also address decision type 2, expanding on the work by Morton to characterize the different ways in which HSS may indirectly influence health outcomes more generally (Hauck et al., 2019). As with Morton and colleagues the health system is seen as a platform where investment may impact multiple health outcomes. However, Hauck and colleagues expand on the work by the previous authors, highlighting the importance of economies of scope; and by positing that HSS investment can either: (1) improve the efficiency of the existing shared platform (in line with Morton and colleagues) (2) relax input capacity constraints (expanding on van Baal and colleagues) and thus improve allocative efficiency, or (3) improve coverage by expanding the platform to new populations and/or allowing for the addition of new services. As with Morton, they then proceed to outline the mathematical models that determine the optimal balance between investment in HSS and service delivery for each of these mechanisms for health outcome improvement and provide worked examples. They conclude that viewing interventions independently where they share platforms may result in misleading analyses, leading to in allocative inefficiency across the health sector.

# Placing Economic Evaluation of HSS Within Priority Setting Processes

In addition to deciding on which theoretical approach to take, economists must also consider how health-systems constraints are considered in the priority-setting process and procedures more broadly when framing analyses. In some cases, the HTA and/or priority setting process may include no explicit consideration of health-systems constraints or constraints may also considered implicitly in the use of considerations of feasibility or other similar criteria. Priority setting processes vary considerably by LMIC setting. In countries such as Thailand all new health technologies require a cost-effectiveness assessment (Teerawattananon et al., 2009). In other countries, HTA may be less developed, and CEA is applied differently by various programs or authorities within the health sector (Kalo et al., 2016). In addition, formal priority setting processes, including cost-effectiveness analyses are increasingly part of defining health (or disease program) packages, as part of National Strategic Planning process (Tantivess, Chalkidou, Tritasavit, & Teerawattananon, 2017). In principle, these processes can either formally or informally recommend that HSS (or elements of constraints or feasibility) is captured within any CEA (or any accompanying analysis, such as value of implementation), or that these elements need to be considered alongside CEA. Where priority setting or HTA institutions are less developed, the decision as to whether to include health-systems constraints or HSS can however be left to the analyst.

The simplest way for decision makers to ensure that health-systems constraints and feasibility are formally considered alongside other decision criteria, such as cost-effectiveness is to adopt checklists. These checklists may be used to encourage decision makers to think through and consult experts and evidence on different aspects of feasibility and health-systems constraints (Kapiriri & Martin, 2010). Criteria include the availability of resources, capacity, and political will to implement interventions within the health system.

The incorporation of health-systems constraints may be also considered in priority setting through multi-criteria decision analysis (MCDA). MCDA moves beyond checklists by formalizing the process for weighing each of different criteria that are being considered and provides a set of practical tools for guiding policymakers through a decision-making process that allows them to consider other criteria alongside (but external to) CEA (Baltussen, Youngkong, Paolucci, & Neissen, 2010; Peacock et al., 2009). According to Peacock and colleagues a key requirement of MCDA for deciding on resource allocation is that analysts cooperate with policymakers to define a set of locally relevant decision criteria through semi-structured discussion (Peacock et al., 2009). In doing so, it allows for the consideration of health-systems constraints that may not be easily quantified and should allow for discussion of overlap between different criteria. For example, in a literature review of decision criteria for resource allocation and healthcare decision making, Guindo and colleagues identified legislation, skills, and level of integration within the health system as determinants of intervention feasibility—all criteria that are not readily quantifiable for direct inclusion in CEA (Guindo et al., 2012).

Other established checklists have been used to measure the completeness and usefulness of the decision-support evidence generated through cost-effectiveness analysis; and some of these also refer to feasibility, albeit indirectly. Most recently, the International Decision Support Initiative published a reference case for economic evaluation that includes the principle that in a good economic evaluation study, “The impact of implementing the intervention on the health budget and on other constraints should be identified clearly and separately” (Principle 10) (Wilkinson et al., 2016). Here the recommendation is not to include implementation directly in ICERs but to suggest the analyst include information on health-systems constraints and any costs of HSS alongside the CEA. Analysts need to be careful to check that priority setting processes do not dictate that constraints are considered externally to CEA; but if they do, they should at minimum be transparent about any overlap.

## Measurement and Modeling HSS in CEA

Finally, once the analyst has considered the intervention, the appropriate theoretical approach and the scope of the analysis, an appropriate model structure and parameterization needs to be defined. Each of the theoretical approaches explores an aggregate relationship between HSS and health outcomes that needs to be empirically incorporated in decision analytic model or other analytical approaches. For example, the approach proposed by Fenwick and colleagues requires parameterizing the influence of investment in “implementation activities” on the effectiveness of specific services. Without this parameter the analysis can only provide a threshold level of investment in HSS. Likewise, the approaches proposed by Morton and colleagues and Hauck and colleagues require parameterizing the influence of HSS on health service performance, such as improved service quality and coverage and its consequential impact on multiple outcomes; these approaches also require an understanding of the extent of economies of scope in the cases where HSS supports multiple services. Van Baal and colleagues’ approach is less empirically demanding if good cost data are available, but it is applicable only under a condition where the health sector allocates resources such that input prices reflect opportunity cost.

Unfortunately, despite identifying the characteristics of the parameter required, in nearly all cases the aggregate causal relationship between the health system and the effectiveness of any new technology is most often not observable, let alone measurable. Instead, as described above, there is often a complex relationship between specific aspects of HSS investment and range of aspects of health system performance and multiple outcomes, all of which presents considerable empirical challenge.

To date, work addressing this challenge has taken two approaches. First, much of the empirical work in this area simply aims to highlight the importance to decision makers (particularly those with disease-specific responsibilities) of not considering HSS by examining the extent to which constraints impact the effective implementation and thus the ICERs of specific new technologies. Second, there is also a body of work that aims to examine the relationship between joint investment in HSS and new technology and outcomes in line with the theoretical approaches outlined by Fenwick, Claxton and Sculpher (2008), Morton, Thomas and Smith (2016), and Hauck and colleagues (2019). Given the complexity of the interface between health systems with health outcomes, most of the empirical methods used employ models that identify pathways between HSS and outcomes, and then parameterize each element of that model rather than trying to empirically measure the aggregate relationships characterized in the theoretical work. We do not aim to summarize all the work in this field here but rather point to a few recent examples of the use of these approaches in economic evaluation in LMICs.

### Estimating the Impact of Health-System Constraints on the ICERs of New Technologies

There are several examples of using mathematical modeling to explore the impact of constraints on ICERs, that also highlight the consequences of not considering HSS when conducting CEA of new technologies in LMICs. Mathematical programming has been the main method used to predict optimal cost-effectiveness given a set of predefined (financial and nonfinancial) constraints on resource allocation (Cleary, Mooney, & McIntyre, 2010; Epstein, Chalabi, Claxton, & Sculpher, 2007, Morton, Thomas, & Smith, 2016), and has been applied to transmission dynamic models in the case of infectious diseases (Hontelez et al., 2016). The empirical challenge is to identify and then quantify different constraints. Bozzani and colleagues developed a “proof of concept” method to identify and estimate different health-system constraints for inclusion in model-based economic evaluations, using the example of intensified case-finding (ICF) strategies for TB in South Africa. As part of a national strategic planning process, the authors estimated the financial and human resources needed to scale up different ICF strategies, then identified three constraints to employing these resources through discussions with local stakeholders. Constraints (e.g., feasible nursing supply) were then quantified using routine health system data and then applied to CEA, by attaching resource usage estimates to dynamic transmission model outputs. The model predicted costs and impact when resource usage was not allowed to exceed these constraints (Bozzani et al., 2018).

An alternative is to draw upon data on how similar interventions have been implemented in the past to predict potential costs and effectiveness during implementation. There is a wide-ranging globally applicable literature on methods to synthesize “real world” programmatic and observational data on areas such as technology uptake with data from clinical trial settings (Garrison et al., 2007). The data can then be incorporated in models to modify assumptions around the extent of effective implementation.

### Estimating the Costs and Impact of HSS for New Technologies

Where analysts want to implement the approaches suggested by Fenwick, Claxton and Sculpher (2008), Morton, Thomas and Smith (2016), and Hauck and colleagues (2019), they need to estimate the potential cost and impact of the HSS required to effectively implement a new technology or a portfolio of new services, in some cases considering economies of scope. Decision analytical models may also be linked with health system models to identify the HSS costs required to address capacity constraints, together with information gathered from decision makers and local stakeholders. For example, operational models can be used to quantify “bottlenecks” in the health system. Langley and colleagues use this approach to examine the case of new TB diagnostics, parameterizing an operational model of how patients and tests flow through health centers and laboratories in Tanzania (Langley et al., 2014; Lin et al., 2011). Working with local decision makers, they use this model to identify additional staffing and infrastructure requirements (akin to the “building blocks” conceptual framework used by WHO). These are then costed, and the systems model is linked to an infectious disease model to estimate cost-effectiveness of the new diagnostic including HSS.

Another approach that has been used extensively in the commercial sector and is beginning to be applied in the health sector in LMICs is group based system dynamics modeling. These models can be used to help identify and determine the cost of actions required to ensure perfect implementation. Group-based model building (GMB) is a collection of techniques to elicit and harmonize a group of experts’ understanding of complex problems (de Gooyert et al., 2017; Vennix, 1996). Experts are likely to be knowledgeable in only partially overlapping areas, so GMB can help them build a holistic view around the implementation of interventions, including identifying systemic constraints (Sterman, 2000). This approach can then be used to build a system dynamics model that characterizes the structure and behavior of complex adaptive systems focusing on the (relationships between) smaller parts of it one at a time (Sterman, 2000). System dynamics modeling methods are used to visually map networks of interlinked factors, feedback loops, and time delays. When these maps are parameterized, computer simulations can be run to investigate whether the model fits past behavior of the system, as well as to explore expected future behavior and the effects of proposed new investments in real-time (Vennix, 1996). As with operational models, these models can be linked with conventional decision analytic models to identify where in the system HSS is required and thus the costs of the actions to implement a new technology effectively.

In recent years, several costing tools have been developed that can be used to model the input use and costs of the required level of health systems “hardware” investment to support health benefit packages. For example, the One Health Tool developed by the WHO allows users to specify the inputs required for service delivery of health benefit packages (that in this case include all supplies and human resources), and then estimates the necessary investments in HSS such as human resource management and training across the system. The tool does not directly compare different allocations between systems and service delivery or across different health systems functions, nor does it establish the relative value of HSS investment but can facilitate discussion through examining the costs of different scenarios (Hanson et al., 2008; Borghi et al., 2005, 2015). These tools may also be used to help identify potential economies of scope by identifying potential shared costs. A common thread in the approaches described above is the role of combining data with both stakeholder and expert views to understand context and the costs of HSS.

There is considerably less readily available data on the observed costs of specific HSS interventions and even less that attempt to directly measure the cost-effectiveness of such interventions. Analysts however may have some access to useful cost data prior to implementation of the service/technology being evaluated. For example, clinical trials often invest substantially to ensure maximum fidelity to implementation guidelines (thus providing an example of the costs of doing so), or in some cases demonstration trials will be conducted before seeking substantial investment in new technologies. In recent years there has been a limited but growing body of economic evaluations directly measuring the “real world” costs and cost-effectiveness of new technologies during or post implementation. These real-world data have been used in some cases to refine estimates (or inform other countries using the experience of early “adopters” of new technologies). In other cases, the implementation of a new technology has been phased, enabling the collection of comparative cost data between the standard of care and the new technology (Vassall et al., 2017). Finally, there is also a body of empirical evidence that examines the health impact of HSS investment on general health outcomes (although may not include assessments of cost and cost-effectiveness), which can be referred to parameterize potential impact (Hatt et al., 2015).

# Summation and Way Forward

This article organizes the current work on concepts, theoretical frameworks, and the empirical assessment of HSS in CEA, in order to guide analysts through the methods and approaches available in this complex area. There are a range of health systems frameworks that analysts can use to define feasibility, constraints, and the HSS intervention (either standalone or as part of a new technology investment). Different theoretical approaches then provide guidance on how HSS should be considered in economic evaluation and priority setting, depending on the structure of the investment decision. Finally, there is a range of empirical methods and a growing body of evidence emerging, which can be used to incorporate HSS in ICERs or consider feasibility and constraints alongside estimates of cost-effectiveness in priority setting.

Despite the range of research in this area, more work is required to refine methods. Theoretically, while the “value” of implementation may assist decision makers to distinguish between the level of cost-effectiveness that can be achieved and that which occurs during implementation, the principle of opportunity cost requires that all incremental resources required to achieve health outcomes are included. Given that implementation may be hard to predict, more guidance is needed on how often and at what point decisions and cost-effectiveness estimates should be revisited during implementation, as new data become available that may substantially change ICERs; potentially influencing price negotiations around new technologies. Underestimating opportunity cost is a particular concern in LMICs with weak health technology assessment infrastructure and procedures, where there may be considerable risk associated with implementation. Moreover, van Baal and colleagues highlight that even if the incremental resources required to implement new technologies are correctly identified, the existence of constraints may underestimate opportunity costs. More work is required to understand the extent of the bias that this may have created in past estimates.

Given the availability of theoretical approaches, the main challenge preventing the application of economic evaluation to HSS is now empirical. While theoretically HSS has been characterized as a continuous single variable, in the “real world” HSS can address a wide variety of health system functions. Designing, costing, and understanding the impact of strategies requires the complex relationship between HSS investment and the effective implementation of new technologies is only beginning to be unpacked. Nevertheless, a range of developments, from the One Health Tool to evidence synthesis methods to the various complex models of the health system, offers some promise that the widespread incorporation of HSS will be feasible in the future. There remains a dearth of studies examining the causal link between investment in HSS (either standalone or in conjunction with new technologies) and health outcomes using trial-based or econometric approaches, and a lack of tested methods to generalize or transfer evidence across settings. Even where the impact of HSS is rigorously evaluated, evaluations rarely consider cost-effectiveness. This mismatch contributes to (and perhaps reflects) the tendency to consider HSS separately outside the field of economic evaluation and priority-setting processes.

Given that the evidence base remains weak and considering the complex nature of health-systems constraints and feasibility, it remain important to ensure stakeholder and expert involvement to consider these issues alongside cost-effectiveness. Because much HSS has a diffuse and indirect impact on health outcomes (e.g., pharmaceutical supply strengthening, information system improvements) it will always be more difficult to subject these to or include them in CEA, compared to HSS interventions that have a more direct health impact. If CEA increasingly considers HSS, it will thus still be important to design HTA and priority settings processes that ensure that these interventions are not neglected. Designing sound evidence-informed deliberative priority setting processes that include HSS and that can consider health systems will remain challenging in LMICs, where priority setting is not fully institutionalized (Baltussen et al., 2013; Barasa, Cleary, English, & Molyneux, 2016) but remains the foundation of any effort, even as more sophisticated measurement and modeling methods become available.

Atun, R. (2012). Health systems, systems thinking and innovation. Health Policy and Planning, 27 (Suppl. 4), iv, 4–8.Find this resource:

Baltussen, R., Mikkelsen, E., Tromp, N., Hurting, A., Byskov, J., Olsen, O., . . . Norheim, O. F. (2013). Balancing efficiency, equity and feasibility of HIV treatment in South Africa—development of programmatic guidance. Cost Effectiveness and Resource Allocation, 11(26).Find this resource:

Bozzani, F. M., Mudzengi, D., Sumner, T., Gomez, G. B., Hippner, P., Cardenas, V., . . . Vassall, A. (2018). Empirical estimation of resource constraints for use in model-based economic evaluation: an example of TB services in South Africa. Cost Effectiveness and Resource Allocation, 16, 27.Find this resource:

Fenwick, E., Claxton, K., & Sculpher, M. (2008). The value of implementation and the value of information: combined and uneven development. Medical Decisision Making, 28, 21–32.Find this resource:

Hatt, L., Johns, B., Connor, C., Meline, M., Kukla, M., & Moat, K. (2015). Impact of Health Systems Strengthening on Health. Bethesda, MD: Health Finance & Governance Project, Abt Associates.Find this resource:

Hanson, K., Ranson, M. K., Oliveira-Cruz, V., & Mills, A. (2003). Expanding access to priority health interventions: A framework for understanding the constraints to scaling-up. Journal of International Development, 15, 1–14.Find this resource:

Hauck, K., Morton, A., Chalkidou, K., Chi, Y. L., Culyer, A., Levin, C., . . . Smith, P. C. (2019). How can we evaluate the cost-effectiveness of health system strengthening? A typology and illustrations. Social Science and Medicine, 220, 141–149.Find this resource:

Hontelez, J. A. C., Change, A. Y., Ogbuoji, O., & De Vlas, S. J., Barnighausen, T., & Atun, R. (2016). Changing HIV treatment eligibility under health system constraints in sub-Saharan Africa: Investment needs, population health gains, and cost-effectiveness. AIDS, 30, 2341–2350.Find this resource:

Langley, I., Lin, H., Egwaga, S., Doulla, B., Ku, C., Murray, M., . . . Squire, S. B. (2014). Assessment of the patient, health system, and population effects of Xpert MTB/RIF and alternative diagnostics for tuberculosis in Tanzania: An integrated modelling approach. Lancet Global Health, 2, e581–591.Find this resource:

Morton, A., Thomas, R., & Smith, P. C. (2016). Decision rules for allocation of finances to health systems strengthening. Journal of Health Economics, 49, 97–108.Find this resource:

Revill, P., Walker, S., Cambino, V., Phillips, A., & Sculpher, M. J. (2018). Reflecting the real value of health care resources in modelling and cost-effectiveness studies: The example of viral load informed differentiated care. PLoS One, 13, e0190283.Find this resource:

Van Baal, P., Morton, A., & Severens, J. L. (2018). Health care input constraints and cost effectiveness analysis decision rules. Social Science and Medicine, 200, 59–64.Find this resource:

Van Baal, P., Thongkong, N., & Severens, J. L. (2016). Human resource constraints and the methods of economic evaluation of health care technologies. iDSI Methods Working Group report.Find this resource:

Vassall, A., Mangham-Jefferies, L., Gomez, G. B., Pitt, C., & Foster, N. (2016). Incorporating demand and supply constraints into economic evaluation in low-income and middle-income countries. Health Economics, 25, 95–115.Find this resource:

## References

Atun, R. (2012). Health systems, systems thinking and innovation. Health Policy and Planning, 27(Suppl. 4), iv, 4–8.Find this resource:

Baltussen, R., Mikkelsen, E., Tromp, N., Hurting, A., Byskov, J., Olsen, O., . . . Norheim, O. F. (2013). Balancing efficiency, equity and feasibility of HIV treatment in South Africa—development of programmatic guidance. Cost Effectiveness and Resource Allocation, 11(26).Find this resource:

Baltussen, R., Youngkong, S., Paolucci, F., & Niessen, L. (2010). Multi-criteria decision analysis to prioritize health interventions: Capitalizing on first experiences. Health Policy, 96, 262–264.Find this resource:

Barasa, E. W., Cleary, S., English, M., & Molyneux, S. (2016). The influence of power and actor relations on priority setting and resource allocation practices at the hospital level in Kenya: A case study. BMC Health Services Research, 16, 536.Find this resource:

Borghi, J., Gorter, A., Sandiford, P., & Segura, Z. (2005). The cost-effectiveness of a competitive voucher scheme to reduce sexually transmitted infections in high-risk groups in Nicaragua. Health Policy and Planning, 20, 222–231.Find this resource:

Borghi, J., Little, R., Binyaruka, P., Patouillard, E., & Kuwawenaruwa, A. (2015). In Tanzania, the many costs of pay-for-performance leave open to debate whether the strategy is cost-effective. Health Affairs (Millwood), 34, 406–414.Find this resource:

Bozzani, F. M., Mudzengi, D., Sumner, T., Gomez, G. B., Hippner, P., Cardenas, V., . . . Vassall, A. (2018). Empirical estimation of resource constraints for use in model-based economic evaluation: an example of TB services in South Africa. Cost Effectiveness and Resource Allocation, 16, 27.Find this resource:

Cleary, S., Mooney, G., & McIntyre, D. (2010). Equity and efficiency in HIV-treatment in South Africa: The contribution of mathematical programming to priority setting. Health Economics,19, 1166–1180.Find this resource:

De Gooyert, V., Rouwett, E., Van Kranenburg, H., & Freeman, E. (2017). Reviewing the role of stakeholders in operational research: A stakeholder theory perspective. European Journal of Operational Research, 262, 402–410.Find this resource:

De Savigny, D. A., T. (2009). Thinking for health systems. Geneva, Switzerland: Alliance for Health Policy and Systems Research.Find this resource:

Epstein, D., Chalabi, Z., Claxton, K., & Sculpher, M. (2007). Efficiency, equity and budgetary policies: Informing decisions using mathematical programming. Medical Decision Making, 27, 128–137.Find this resource:

Fenwick, E., Claxton, K., & Sculpher, M. (2008). The value of implementation and the value of information: combined and uneven development. Medical Decision Making, 28, 21–32.Find this resource:

Garrison, L. P., Jr., Neumann, P. J., Erickson, P., Marshall, D., & Mullins, C. D. (2007). Using real-world data for coverage and payment decisions: The ISPOR Real-World Data Task Force report. Value in Health,10, 326–335.Find this resource:

Gericke, C. A., Kurowski, C., Ranson, M. K., & Mills, A. (2005). Intervention complexity—a conceptual framework to inform priority-setting in health. Bulletin of the World Health Organization, 83, 285–293.Find this resource:

Govindasamy, D., Ford, N., & Kranzer, K. (2012). Risk factors, barriers and facilitators for linkage to antiretroviral therapy care: A systematic review. AIDS, 26, 2059–2067.Find this resource:

Govindasamy, D., Kranzer, K., & Ford, N. (2014). Strengthening the HIV cascade to ensure an effective future ART response in sub-Saharan Africa. Transactions of the Royal Society of Tropical Medicine and Hygiene, 108, 1–3.Find this resource:

Guindo, L. A., Wagner, M., Baltussen, R., Rindress, D., Van Til, J., Kind, P., & Goetghebeur, M. M. (2012). From efficacy to equity: Literature review of decision criteria for resource allocation and healthcare decisionmaking. Cost Effectiveness and Resource Allocation, 10(1), 9.Find this resource:

Hanson, K. (2011). Delivering health services: Incentives and information in supply-side innovations. In R. H. K. Smith (Ed.), Health systems in low- and middle-income countries: An economic and policy perspective. Oxford, U.K.: Oxford University Press.Find this resource:

Hanson, K., Nathan, R., Marchant, T., Mponda, H., Jones, C., Bruce, J., . . . Schellenberg, J. A. (2008). Vouchers for scaling up insecticide-treated nets in Tanzania: Methods for monitoring and evaluation of a national health system intervention. BMC Public Health, 8, 205.Find this resource:

Hanson, K., Ranson, M. K., Oliveira-Cruz, V., & Mills, A. (2003). Expanding access to priority health interventions: A framework for understanding the constraints to scaling-up. Journal of International Development, 15, 1–14.Find this resource:

Hatt, L., Connor, C., Meline, M., Kukla, M., & Moat, K. (2015). Impact of health systems on health. Bethesda, MD: Health Finance & Governance Project, ABT Associates.Find this resource:

Hauck, K., Morton, A., Chalkidou, K., Chi, Y. L., Culyer, A., Levin, C., . . . Smith, P. C. (2019). How can we evaluate the cost-effectiveness of health system strengthening? A typology and illustrations. Social Science and Medicine, 220, 141–149.Find this resource:

Hontelez, J. A. C., Change, A. Y., Ogbuoji, O., De Vlas, S. J., Barnighausen, T., & Atun, R. (2016). Changing HIV treatment eligibility under health system constraints in sub-Saharan Africa: Investment needs, population health gains, and cost-effectiveness. AIDS, 30, 2341–2350.Find this resource:

Kalo, Z., Gheorghe, A., Huic, M., Csanadi, M., & Kristensen, F. B. (2016). HTA implementation roadmap in central and eastern European countries. Health Economics,25(Suppl. 1), 179–192.Find this resource:

Kapiriri, L., & Martin, D. K. (2010). Successful priority setting in low and middle income countries: A framework for evaluation. Health Care Analysis, 18, 129–147.Find this resource:

Kelly, S. L., Martin-Hughes, R., Stuart, R. M., Yap, X. F., Kedziora, D. J., Grantham, K. L., . . . Wilson, D. P. (2018). The global Optima HIV allocative efficiency model: Targeting resources in efforts to end AIDS. Lancet HIV, 5, e190–e198.Find this resource:

Kerr, C. C., Stuart, R. M., Gray, R. T., Shattock, A. J., Fraser-Hurt, N., Benedikt, C., . . . Wilson, D. P. (2015). Optima: A model for HIV epidemic analysis, program prioritization, and resource optimization. Journal of Acquired Immune Deficiency Syndrome, 69, 365–376.Find this resource:

Kim, J. J., Salomon, J. A., Weinstein, M. C., & Goldie, S. J. (2006). Packaging health services when resources are limited: The example of a cervical cancer screening visit. PLoS Medicine, 3, e434.Find this resource:

Langley, I., Lin, H., Egwaga, S., Doulla, B., Ku, C., Murray, M., . . . Squire, S. B. (2014). Assessment of the patient, health system, and population effects of Xpert MTB/RIF and alternative diagnostics for tuberculosis in Tanzania: An integrated modelling approach. Lancet Global Health, 2, e581–591.Find this resource:

Lin, H. H., Langley, I., Mwenda, R., Doulla, B., Egwaga, S., Millington, K. A., . . . Cohen, T. (2011). A modelling framework to support the selection and implementation of new tuberculosis diagnostic tools. International Journal of Tuberculosis and Lung Disease, 15, 996–1004.Find this resource:

Mangham-Jefferies, L., Wiseman, V., Achonduh, O. A., Drake, T. L., Cundill, B., Onwujekwe, O., . . . Mbacha, W. (2014). Economic evaluation of a cluster randomized trial of interventions to improve health workers’ practice in diagnosing and treating uncomplicated malaria in Cameroon. Value in Health, 17, 783–791.Find this resource:

Mikkelsen, E., Hontelez, J. A. C., Jansen, M. P. M., Barnighausen, T., Hauck, K., Johansson, K. A., . . . Baltussen, R. M. P. M. (2017). Evidence for scaling up HIV treatment in sub-Saharan Africa: A call for incorporating health systems constraints. PLoS Medicine, 14, e1002240.Find this resource:

Morton, A., Thomas, R., & Smith, P. C. (2016). Decision rules for allocation of finances to health systems strengthening. Journal of Health Economics, 49, 97–108.Find this resource:

Mkumbang, F. C., Marchal, B., Van Belle, S., & Van Wyk, B. (2018). Patients are not following the [adherence] club rules anymore: A realist case study of the antiretroviral treatment adherence club, South Africa. Qualitative Health Research, 28(12), 1839–1857.Find this resource:

Mukumbang, F. C., Van Belle, S., Marchal, B., & Van Wyk, B. (2016). Realist evaluation of the antiretroviral treatment adherence club programme in selected primary healthcare facilities in the metropolitan area of Western Cape Province, South Africa: A study protocol. BMJ Open, 6, e009977.Find this resource:

Ochalek, J., Revill, P., Manthalu, G., McGuire, F., Nkhoma, D., Rollinger, A., . . . Claxton, K. (2018). Supporting the development of a health benefits package in Malawi. BMJ Global Health, 3, e000607.Find this resource:

Peacock, S., Mitton, C., Bate, A., McCoy, B., & Donaldson, C. (2009). Overcoming barriers to priority setting using interdisciplinary methods. Health Policy, 92, 124–132.Find this resource:

Pritchett, L. W. (2004). Solutions when the solution is the problem: Arraying the disarray in development. World Development, 32, 191–212.Find this resource:

Revill, P., Walker, S., Cambiano, V., Phillips, A., & Sculpher, M. J. (2018). Reflecting the real value of health care resources in modelling and cost-effectiveness studies: The example of viral load informed differentiated care. PLoS One, 13, e0190283.Find this resource:

Schwartlander, B., Stover, J., Hallett, T., Atun, R., Avila, C., Gouws, E., . . . Investment Framework Study. (2011). Towards an improved investment approach for an effective response to HIV/AIDS. Lancet, 377, 2031–2041.Find this resource:

Sendi, P. P., & Briggs, A. H. (2001). Affordability and cost-effectiveness: Decision-making on the cost-effectiveness plane. Health Economics, 10, 675–680.Find this resource:

Sheikh, K., Gilson, L., Agyepong, I. A., Hanson, K., Ssengooba, F., & Bennett, S. (2011). Building the field of health policy and systems research: framing the questions. PLoS Med, 8, e1001073.Find this resource:

Sterman, J. D. (2000). Systems thinking and modelling for a complex world. Boston, MA: Irwin McGraw-Hill.Find this resource:

Tantivess, S., Chalkidou, K., Tritasavit, N., & Teerawattananon, Y. (2017). Health technology assessment capacity development in low- and middle-income countries: Experiences from the international units of HITAP and NICE. F1000Res, 6, 2119.Find this resource:

Teerawattananon, Y., Tantivess, S., Yothasamut, J., Kinkaew, P., & Chaisiri, K. (2009). Historical development of health technology assessment in Thailand. International Journal of Technology Assessment in Health Care,25(Suppl. 1), 241–252.Find this resource:

Tromp, N., Prawiranegara, R., Siregar, A., Wisaksana, R., Pinxten, J., Putra, A. L., . . . Baltussen, R. (2018). Translating international HIV treatment guidelines into local priorities in Indonesia. Tropical Medicine and International Health, 23(3), 279–294.Find this resource:

Van Baal, P., Morton, A., & Severens, J. L. (2018). Health care input constraints and cost effectiveness analysis decision rules. Social Science and Medicine, 200, 59–64.Find this resource:

Van Baal, P., Thongkong, N., & Severens, J. L. (2016). Human resource constraints and the methods of economic evaluation of health care technologies. IDSI Methods Working Group Report.Find this resource:

Vassall, A., Mangham-Jefferies, L., Gomez, G. B., Pitt, C., & Foster, N. (2016). Incorporating demand and supply constraints into economic evaluation in low-income and middle-income countries. Health Economics,25, 95–115.Find this resource:

Vassall, A., Siapka, M., Foster, N., Cunnama, L., Ramma, L., Fielding, K., . . . Sinanovic, E. (2017). Cost-effectiveness of Xpert MTB/RIF for tuberculosis diagnosis in South Africa: A real-world cost analysis and economic evaluation. Lancet Global Health,5, e710–719.Find this resource:

Vennix, J. A. M. (1996). Group model building: Facilitating team learning using system dynamics. Chichester, U.K.: John Wiley.Find this resource:

Wilkinson, T., Sculpher, M. J., Claxton, K., Revill, P., Briggs, A., Cairns, J. A., . . . Walker, D. G. (2016). The international decision support initiative reference case for economic evaluation: An aid to thought. Value in Health, 19, 921–928.Find this resource:

Working Group on Modelling of Antiretroviral Therapy Monitoring Strategies in Sub-Saharan, Phillips, A., Shroufi, A., Vojnov, L., Cohn, J., Roberts, . . . Revill, P. (2015). Sustainable HIV treatment in Africa through viral-load-informed differentiated care. Nature, 528, S68–S76.Find this resource:

World Health Organization. Health systems strengthening glossary. Geneva, Switzerland: WHO.Find this resource:

World Health Organization. (2007). Every body’s business: Strengthening health systems to improve health outcomes. WHO’s Framework for Action. Geneva, Switzerland: WHO.Find this resource:

## Notes:

(1.) Here the technology is defined as a new way of delivering services (which might, but does not necessarily include a new biomedical device).