Water Resources Planning Under (Deep) Uncertainty
Water Resources Planning Under (Deep) Uncertainty
- Riddhi SinghRiddhi SinghIndian Institute of Technology Bombay
Summary
Public investments in water infrastructure continue to grow where developed countries prioritize investments in operation and maintenance while developing countries focus on infrastructure expansion. The returns from these investments are contingent on carefully assessed designs and operating strategies that consider the complexities inherent in water management problems. These complexities arise due to several factors, including, but not limited to, the presence of multiple stakeholders with potentially conflicting preferences, lack of knowledge about appropriate systems models or parameterizations, and large uncertainties regarding the evolution of future conditions that will confront these projects. The water resources planning literature has therefore developed a variety of approaches for a quantitative treatment of planning problems. Beginning in the mid-20th century, quantitative design evaluations were based on a stochastic treatment of uncertainty using probability distributions to determine expected costs or risk of failure. Several simulation–optimization frameworks were developed to identify optimal designs with techniques such as linear programming, dynamic programming, stochastic dynamic programming, and evolutionary algorithms. Uncertainty was incorporated within existing frameworks using probability theory, using fuzzy theory to represent ambiguity, or via scenario analysis to represent discrete possibilities for the future.
As the effects of climate change became palpable and rapid socioeconomic transformations emerged as the norm, it became evident that existing techniques were not likely to yield reliable designs. The conditions under which an optimal design is developed and tested may differ significantly from those that it will face during its lifetime. These uncertainties, wherein the analyst cannot identify the distributional forms of parameters or the models and forcing variables, are termed “deep uncertainties.” The concept of “robustness” was introduced around the 1980s to identify designs that trade off optimality with reduced sensitivity to such assumptions. However, it was not until the 21st century that robustness analysis became mainstream in water resource planning literature and robustness definitions were expanded to include preferences of multiple actors and sectors as well as their risk attitudes. Decision analytical frameworks that focused on robustness evaluations included robust decision-making, decision scaling, multi-objective robust decision-making, info-gap theory, and so forth. A complementary set of approaches focused on dynamic planning that allowed designs to respond to new information over time. Examples included adaptive policymaking, dynamic adaptive policy pathways, and engineering options analysis, among others. These novel frameworks provide a posteriori decision support to planners aiding in the design of water resources projects under deep uncertainties.
Keywords
Subjects
- Management and Planning
Water Resources Planning as a Complex Decision Problem
Water infrastructure aids the management of spatiotemporally variable water resources with the broad goals of socioeconomic development and ecological conservation. Water infrastructure projects fall within a wider portfolio of activities undertaken for water resources planning, which also includes the design of operational strategies, and soft measures such as institutional and policy interventions. Some typical water resource planning problems include the design of reservoirs, floodplain management, design of levees, evaluation of long-term water management plans for urban water supply, inter-basin water transfers, irrigation scheduling, sequencing of water supply infrastructure, management of evolving water markets, and so on. These projects affect economic, social, and environmental conditions in the regions they serve. As a consequence, water resources planning encompasses a range of decision problems whose various aspects have been explored by multiple disciplines including civil engineering, hydrology, climate science, economics, social sciences, and so forth.
Water resource planning problems are complex in nature as they deal with a resource base governed by both complicated natural processes and human interventions, and they involve multiple stakeholders with varying preferences distributed spatially. Water resource projects affect multiple sectors related to water, food, and energy supply. Stakeholders may be spread across political or river basin boundaries. They have a lifetime of several decades, which introduces considerable uncertainties in the systems models, parameters, and climatic conditions under which to evaluate alternative designs. For example, identifying optimal operating policies for a reservoir providing water supply and hydropower requires estimates of inflows, evaporation fluxes, water demands, and energy requirements over several decades. So, the presence of uncertainty lends considerable complexity to water resource planning. Thus, their proper treatment in the decision framework has been vital to the long-term planning of water resources.
Water resource planning problems are representative of other environmental decision-making problems such as land use planning, setting standards for air or water quality, evaluating the risk of using pesticides or other chemicals, and so on. These problems share the same underlying complexities related to poorly specified systems models that map actions to consequences, the presence of diverse stakeholders spread across multiple spatiotemporal scales, and the lack of commensurate measures to judge the desirability of various outcomes. Decision frameworks and concepts developed to aid water resources planning under uncertainty are thus likely to have applications in several allied areas of environmental science and vice versa. It is perhaps not surprising that nearly all major decision analytic advances in 21st century water resources planning literature have either adopted frameworks from allied disciplines or contributed frameworks applicable to other environmental science disciplines.
Part 1
Risk-Based Analysis in Water Resources Design
One of the earliest problems of water resources planning under uncertainty was the design of water storage reservoirs and understanding their behavior under an expected lifetime of a few decades. The observed variability in streamflow records necessitated the development of methodologies and concepts to quantify uncertainty. In this context, the concept of “risk of failure” was introduced by Hazen in 1914 (Klemeš, 1981). Put simply, risk of failure is the probability of a storage reservoir being empty in any given year. The risk of failure was estimated after quantifying uncertainty in annual or seasonal inflows using probability distributions (Moran, 1954). These methods were analytical in nature and employed the frequentist interpretation of probability. The risk of failure was defined as the ratio of time periods with the reservoir being empty to the total number of time periods, with the total number of time periods tending to infinity. A number of simplifications were needed, however, to derive analytical results. Many studies that evaluated risk of failure used the normal distribution to represent uncertainty in annual inflows. Often, to contain the dimensionality of the problem, a hypothetical reservoir was considered with discretized levels of storage, inflows, and demands (Askew, 1974). Several developments focused on estimating the risk value given the limited lifetime of the reservoir, again requiring considerable simplifications in the frequentist framework to obtain analytical results (Klemeš, 1967). An additional assumption was that of stationarity in the distribution of annual or seasonal inflows to the reservoir.
Concurrent to these developments, the 1960s saw the emergence of the field of synthetic hydrology. The work of Thomas and Fiering (1962) laid the foundation for synthetic hydrology that employed streamflow sequences to represent observed uncertainty. This technique gained momentum with the widespread availability of computing resources (Matalas, 1967). The main advantage of using synthetically generated hydrologic sequences was that they could be used to test system performance under futures that were not exactly like the historical observations but still shared the same statistical properties. In addition to generating sequences for one site, techniques were also developed to allow multivariate generation so that sequences could be generated across several sites while preserving the observed relationships between flows across them (Fiering, 1964). However, three main challenges arose in using synthetic sequences for planning purposes. First, the statistical parameters of the synthetic sequences had to be derived from a limited record of observations. Depending upon the length of the record available and the type of parameter being inferred, there could be significant bias in its estimate, irrespective of the methods used to quantify them. Second, if the planner was interested in testing system performance under some special conditions, such as by creating sequences of varying durations and magnitudes of flows below a prespecified threshold, the available approaches provided little help. Finally, the analyst had to choose the type of distribution as well as the statistical properties to be kept the same between the synthetic sequence and observations. Matalas (1967) suggested that more than one distribution should be tested to observe the consequence of planning decisions but typically found limited application as it was not yet clear how to identify the most appropriate distributional form.
The limitations related to unaccounted parameter or model uncertainties were partially addressed using Bayesian analysis in statistical decision theory (Davis et al., 1972; Vicens et al., 1975). A key difference between synthetic hydrology-based analysis and Bayesian decision analysis was in the way uncertainty in streamflow was treated in the latter. Within a Bayesian framework, streamflow descriptors such as mean discharge or total discharge could themselves be assumed as random variables following an underlying distribution whose parameters could be estimated from observations (Marin, 1986; Wood, 1978). On the other hand, synthetic hydrology generates streamflow realizations assuming that there is a true value of the mean discharge upon which the value of sample mean discharge depends. Obtaining estimators of the mean discharge required the analyst to makes assumptions regarding the loss functions. The most common quadratic loss function resulted in sample mean being equal to the expected value of the true mean, but the same did not hold true for higher-order descriptors such as standard deviation. Similarly, using other loss functions that represented decision makers’ risk attitudes could result in optimal estimates deviating significantly from sample statistics. Thus, maintaining historical statistics does not necessarily result in the best designs due to inherent assumptions regarding loss functions in parameter estimation procedures and the inability to obtain true statistics from limited streamflow data.
Using a didactic example of reservoir design, Wood (1978) showed that Bayesian decision theory led to more conservative reservoir designs when compared to designs based on parameters estimated from historical records. Another advantage of Bayesian decision analysis was that it could shed light on the value of additional data or the value of having perfect information. This shifted the focus from reduction of uncertainties to the decision process itself. The analyst could decide whether to proceed with a decision or collect more data to reduce uncertainties. Even so, the issue of identifying the “true” model that generated the observations remained a challenge, although the Bayesian approach did provide a formal framework to explore the consequences of various model related choices on the decision process (Bodo & Unny, 1976; Wood & Rodríguez‐Iturbe, 1975; Wood et al., 1974). Marin (1986) further stressed the need for developing regionalization methodologies to specify priors in Bayesian analysis as mis-specified priors may lead to suboptimal decisions.
In general, developments up to the 1980s were far more focused on the quantitative treatment of streamflow uncertainty in the optimization or analysis process, and relatively little effort was made to test whether the choice of the uncertainty quantification was realistic considering the hydro-climatological origin of streamflow (Klemes̆, 1993) or whether other sources of uncertainty could be accounted for in the planning stages. This was despite the widespread acceptance that uncertainty in water resources planning could be of different origins. For example, Bower (1965) discussed two types of uncertainties facing water-related decisions. The first arises from physio-climatic processes governing streamflow generation, and the second is due to changes in water demands conditional on future population and lifestyles, among other sociopolitical factors. The existing probability-based frameworks did little to address the varied sources of uncertainties within an integrated decision analysis framework.
Simulation–Optimization Approaches in Water Resources Planning
The Harvard Water program was one of the earliest efforts at an integrated understanding of water resources management considering economic, social, governance, and technological aspects of large water investments (Maass et al., 1962). It attempted to analyze the general water resources planning problem by addressing its interdisciplinary nature. It brought together hydrologists, operation research scientists, economists, and practitioners to study typical large-scale planning problems. The program and its outputs widened the scope of water resources planning literature, shifting the focus from purely mathematical issues related to optimization to the inclusion of legislative and socioeconomic contexts. It highlighted the value of broadening the objectives considered in planning and their evolution through a project’s lifetime, the latter remaining largely unaddressed even in the 21st century. These efforts culminated in the development of a large body of simulation–optimization approaches to identify optimal designs and operating policies.
Application of simulation–optimization to water resource planning required specifying four aspects of the decision problem: decisions to be made or decision variables, performance metrics that guided ranking of alternatives or objective functions, a systems model that translated decisions to performance metrics, and associated uncertainties. For example, a typical reservoir operation problem involved identification of reservoir release rules (decisions) that maximized a performance metric such as reliability of water supply. Observed or synthetically generated streamflow was used to force a water balance model to update storage states of the reservoir after accounting for releases and other fluxes. An optimization algorithm then searches the space of feasible release rules and identifies one that maximizes the performance metric.
The search for optimal operating strategies or designs was supported by advances in control theory in the 1950s that quickly found their application to water resource planning problems. Dynamic programming allowed the identification of an optimal operating rule curve while accounting for nonlinearity in the system and performance measures, stochasticity of hydrologic variables involved, and constraints on decision and state variables (Keckler & Larson, 1968; Tilmant et al., 1970; Yakowitz, 1982). Dynamic programming partitioned a decision problem into smaller sub-problems by use of the recursive Bellman equation. As long as the dimensionality of the problem was small, exact optimal solutions could be identified considering stochasticity in reservoir inflows. An initial criticism of this technique was the inability to deal with noneconomic measures such as the risk of failure, which could only be assessed for post optimization (Askew, 1974). The same issue existed with linear programming that optimized a linear approximation to the reservoir operation problem. Additionally, for linear programming to be applicable, the reservoir operating policy, objective function, and constraints should be a linear function of system states, most often the reservoir storage. Here too, earlier developments focused on deterministic inflows until the work by Revelle et al. (1969), who developed a methodology to include stochastic inflows in the linear decision rule. Nayak and Arora (1971) applied linear programming to a multireservoir system in the Minnesota River basin in the United States. The inclusion of a risk of failure constraint in linear programming was made possible by using chance-constrained programming that allowed some constraints to be violated given an acceptability threshold (Charnes & Cooper, 1963). However, linear programming and related methodologies fell short of the reality that indicated a nonlinear system with nonlinear performance measures and where the functional form of the policy function was hard to know a priori and may or may not be represented by a linear function. Dynamic programming, though overcoming several of these issues, was still faced with the curse of dimensionality, wherein computational requirements grow exponentially with the dimension of the state space (Bellman, 1961; Herman et al., 2020).
Keeney and Wood (1977) highlighted the complexity of water resources planning projects stemming from potentially conflicting objectives, and uncertainties regarding how actions translate to consequences. It was evident that the assumption of a single rational agent optimizing a system-level performance measure encapsulated in a utility function or expected costs is a limited way to represent real-world water planning problems. The outputs of such analysis, while providing useful starting points to understand the decision problem, did little to facilitate a dialogue between stakeholders or address the complexity of the decision problem. Two main approaches emerged to address the need to incorporate multiple preferences in an optimization. The first was that of multicriteria decision-making wherein multiple objective functions are aggregated to convert the optimization to a single objective problem. One of the earliest examples of combining multiple objectives was by using a weighted linear function, which can then be treated using linear programming or any other single objective optimization (Cohon & Marks, 1973; Major, 1969; Raj, 1995; Stewart & Scott, 1995). However, associating weights with each objective presented both technical and normative issues. On the technical side, the range of variation of each objective may differ and a smaller set of objectives may end up dominating the search process. On the normative front, fixed weights assume that decision-makers’ preferences regarding the level of trade-offs are constant irrespective of the objective value attained. This is hardly the case in practical applications, where decision-makers may be more willing to trade-off after a threshold of performance in a given objective is reached. Multicriteria methods such as ELECTRE (ÉLimination Et Choix Traduisant la REalité) ranked predefined alternatives using an elicitation procedure; however, there was no explicit accounting of different uncertainties involved. Keeney and Wood (1977) presented the application of multiattribute utility theory to overcome these issues in long-term water resources planning of the Tisza River basin in Europe but did not include a formal uncertainty analysis. The fundamental problem remained: decision-makers had to prescribe their preferences, in terms of utility functions and scaling constants, before seeing the outcome of the analysis. Utility units carry little practical inference for decision-makers, who may find it easier to interpret trade-offs in relevant units of individual objectives. To summarize, the optimization of an aggregated objective function or a multiattribute utility function is an example of prescriptive decision aiding where stakeholders prescribe the weights or utility functions before understanding the trade-offs implicit in their choices (Tsoukias, 2008).
The second approach to consider multiple objectives in the optimization of water resources problems was built on the concept of Pareto optimality. Solutions are said to be Pareto optimal when it is not possible to improve one objective without degrading the other(s). Many objective optimization using evolutionary algorithms allowed the identification of multi-objective trade-offs and found wide application in water resource literature (Afshar et al., 2009; Chang & Chang, 2009; Reddy & Kumar, 2009; Reed et al., 2013). These constituted an example of a posteriori decision aiding as stakeholders could identify the level of trade-offs they were willing to accept after the optimization. This presented a clear practical advantage as it also allowed the use of noncommensurable measures of performance in the problem formulation. Thus social, economic, and structural risk-related objectives could potentially be optimized simultaneously. Another important advantage of using multi-objective optimization was that it provided stakeholders with a large set of potential designs, which could be further tested for their institutional or sociopolitical acceptability, criteria not easily quantifiable within an optimization framework. Finally, within a simulation framework, many objective optimization allows the inclusion of stochastic uncertainties in inflows as well as scenario-based uncertainties in other variables such as demands. The main caveat of this methodology was that analytical derivation of the Pareto optimal set was infeasible for real-world water resource problems given their complex systems models and forcing. Therefore, heuristics-based algorithms employed to solve such problems required a careful assessment to ensure that the algorithm converged to the solution. Multi-objective evolutionary algorithms also tend to be sensitive to their specific choice of operators and their hyper-parameter settings, resulting in a lack of consistency in their use across different studies. Although formal mechanisms to account for these issues have been developed, their application in water resource planning literature remains uncommon.
Summary of Uncertainty Treatment in Simulation–Optimization Approaches
The treatment of uncertainty in optimization frameworks for water resources planning has been carried out using the following ways:
Use of expected values: The expected values of net profits, benefit–cost ratios, or other objective functions can be estimated based on assumed probability distributions of unknown parameters. Expected value-based optimization assumes risk neutrality and relies on probabilistic treatment of uncertainty.
Risk theoretic frameworks: To better capture the risk attitudes of decision-makers, a utility function is used and optimal designs are those that maximize the expected net present value of utility. Other alternatives include analyzing the variance of net benefits and comparing them against the expected benefits. Designs with a higher variance of benefits are less likely to be preferred.
Resilience, reliability, and vulnerability metrics: These metrics can be used to summarize time-varying system performance across a time horizon and across various possible realizations of the future (Hashimoto, 1980; Kundzewicz & Kindler, 1995). Reliability is the fraction of design periods when system performance requirements are met. Resilience is the ability of a water resource system to adjust to new situations without creating unacceptable performance deterioration; often quantified as the time taken to recover from a failure state. Vulnerability is used to quantify the magnitude of failure. The metrics have operational relevance and have been extensively analyzed using both single and multi-objective optimization (Asefa et al., 2014; Raje & Mujumdar, 2010; Shin et al., 2018). The effects of uncertainty on system performance can be quantified by aggregating these metrics across multiple possible future conditions. The metrics are estimated by driving a water balance using a large number of synthetic sequences of reservoir inflows and demands. Then, summary statistics of metric values across all realizations using median or other quantiles is used as the objective function.
Scenario analysis: When probabilistic representation of uncertainty is not supported on theoretical or observational grounds, scenario analysis can be used to represent uncertainty about the various components of the problem. Scenarios represent possible future states of the world and initially decision-makers may not assign a probability to them. A limited number of discrete scenarios may be generated to analyze the variation of system performance. Traditional applications of this approach relied on a limited number of scenarios to represent uncertainties related to institutional or socioeconomic contexts. For example, Jenkins and Lund (2000) used scenario analysis to identify the impact of institutional uncertainty on system performance. However, advanced forms of this approach were later developed to enable deep uncertainty characterization in long-term planning as discussed in the section Planning for deep uncertainty: robust and adaptive planning.
Sensitivity analysis: Here, the change in system performance is measured against possible changes in uncertain variables. Designs that are highly sensitive to underlying parameter distributions may be avoided. Only limited application of sensitivity analysis is found in the planning literature with earlier efforts using partial derivatives to approximate sensitivity (Dessai & Hulmes, 2007; O’Laoghaire & Himmelblau, 1972; Quinn et al., 2019). As an example, Dessai and Hulme (2007) explored the sensitivity of water resources planning decisions to uncertainties in environmental predictions. Using a local (one-at-a-time) method of sensitivity analysis, they showed that adaptation decisions were most sensitive to regional climate response, for water resources in East Suffolk and Essex in East England.
Using Fuzzy theory: Water resources planning may require incorporation of soft information from stakeholders. For example, nonquantitative goals related to social and environmental well-being exist but cannot be formally embedded in optimization frameworks without further simplification (Esogbue & Ahipo, 1982; Ragade et al., 1976). These are not amenable to treatment by probabilistic methods as they represent ambiguity in translating qualitative information to quantitative measures, not the inherently random nature of a variable like inflow. Such cases of uncertainties were dealt with using fuzzy theory (Bender & Simonovic, 2000; Golmohammadi et al., 2021; Li et al., 2010; Wang & Huang, 2011; Zadeh et al., 2014).
It is evident that a large number of computationally sophisticated methodologies were developed by the early 21st century to analyze the impact of uncertainty on the performance of water resource systems. Even so, significant challenges remained in representing real-world contexts in water resources planning. As early as the 1970s, Kaynor (1978) summarized the main drawbacks of available approaches for uncertainty treatment in water resources planning and noted that “principles or prescriptions suggested may not be behavioural axioms valid at all times in all social contexts.” He demonstrated this through several examples of navigation, flood control, and inter-basin water transfers in the Connecticut River basin, United States. Haimes and Hall (1977) raised the issue of understanding risk in a multi-objective context. They argued that goals of planning such as risk minimization and maximization of economic output are often conflicting in nature. Additionally, though a tremendous amount of literature paid attention to the issue of hydrologic uncertainty, fewer efforts were made to evaluate the impact of demand uncertainties, economic uncertainties, or value judgments on the decision process (Krzysztofowicz & Duckstein, 1979). Szidarovszky et al. (1976) highlighted the impact of economic uncertainty embedded in relationships of flood losses and construction costs with levee height for flood protection while also accounting for hydrologic uncertainty for a flood levee along the left bank of the Tisza River in Hungary. Krzysztofowicz and Duckstein (1979) argued that value judgments form the basis of all decisions, but this subjective aspect has received less attention in water resources systems planning despite being embedded in the systems model via objective functions weights or prespecified target levels for reservoir. Haimes (1984) claimed that uncertainties associated with demand projections that are a function of demographic changes, technological growth, and other factors are likely to be larger when compared to those associated with supply. A number of failures of large-scale water projects were observed that occurred due to unforeseen changes in demands or drastic changes in sociopolitical landscapes (Kaynor, 1978; Klemeš, 1993). Some customized decision frameworks that modeled the “surprise” event as a stochastic process were developed to address this challenge (Erlenkotter et al., 1989), but the issue remained largely unaddressed.
Part 2
Deep Uncertainty
Much of the work up to the 1990s assumed that stochasticity in hydroclimatic variables could be quantified via stationary distributions, that is, those in which the parameters of the distributions did not change with time. Examples of relevant variables included those that represent water supply potential at a location such as annual inflows or total monthly inflows. Annual inflows at a reservoir site were modeled using assumed probability distributions, whose parameters were obtained from historical observations. As anthropogenic climate change became established as a major driver of climate in the 1990s, it also became evident that stationary assumptions may not hold in the future. The long-term means, seasonality, as well as other characteristics of hydroclimatic variables such as intensity and duration of extremes, were likely to exhibit departures from historically inferred values. Hydroclimatic variables could exhibit a long-term trend, sudden shifts in statistical properties, or cyclicity, all resulting in changing statistical properties over time or nonstationary behavior (Milly et al., 2008). New approaches were needed to address this issue while evaluating alternative designs.
Another important source of uncertainty in water resources planning arises due to the complex interactions between coupled human-water systems, also referred to as endogenous uncertainties. There has been an increased focus on understanding such interactions in the 21st century with the rise of socio-hydrological analysis (Sivapalan et al., 2012). These studies aim to disentangle the feedbacks between hydrologic changes and human responses to them to better understand the co-evolution of the two systems. Some important mechanisms through which endogenous uncertainties may affect decision-making include indirect feedbacks, institutional uncertainties, alteration in system behavior triggered by policy actions, and through environmental dynamics. For example, changes in agricultural yields under warmer climates could trigger unforeseen land use changes, resulting in altered water demands. Similarly, an action such as expansion of reservoir capacity or increasing water supply through inter-basin transfers may trigger increase in water demands and increase susceptibility to droughts. The dynamics and long-term co-evolution of social and economic behavior remains largely indeterminate and often lends greater uncertainty to long-term planning problems than exogenous climate forcing (Herman et al., 2020).
Thus, as early as 1980, the United States Water Resources Council differentiated between the concept of risk and uncertainty, which closely followed a similar distinction introduced by Knight (1921) in business economics. They maintained that risk based analysis was suitable in decision situations where probability distributions can reasonably well represent underlying uncertainties in outcomes. On the other hand, instances where probability distributions are poorly specified were termed uncertain (Goicoechea et al., 1982; Haimes, 1984). In general, any long-term policy analysis constitutes a problem of planning under deep uncertainties where the system’s model, distribution of model parameters, or the valuation of different outcomes cannot be agreed upon by the parties to the decision. This definition by Lempert et al. (2003) is aligned with earlier developments in water resources literature but explicitly mentions the uncertainties involved in valuation of different outcomes.
Consider the problem of estimating flood damages from alternative flood protection interventions. Even under assumptions of a stationary climate, deeply uncertain factors involved in the calculations of flood risk arise from lack of knowledge about how humans respond to the presence of flood protection infrastructure thereby altering estimates of future damages (White, 1964). The presence of protective infrastructure creates a false sense of security and triggers infrastructure investments in protected but flood prone regions, which in turn results in increasing risk of catastrophic damages from extreme floods. On similar lines, capacity expansion of water supply infrastructure needs to be based on an accurate understanding of changes in future demands, again a deeply uncertain factor governed by political, social, legal, and institutional changes. Actions such as the introduction of water metering or changing water pricing can have a substantial impact on water demands. Thus, long-term planning of water infrastructure is a problem of planning under deep uncertainty.
Planning for Deep Uncertainty: Robust and Adaptive Planning
Broadly, there are two emerging methodologies for water resources planning under deep uncertainty (Herman et al., 2020). The first suite of methods are termed as robust planning frameworks that aim to iteratively test designs across a wide array of possible futures. The second set of methods is based on the concepts of adaptive management, proposed as early as the 1970s by Holling (1978). Adaptive planning is motivated by the idea that plans will have to respond to new information as and when it arrives. It, therefore, enables the design of alternative plans that aim to achieve robust performance, while robust decision frameworks focusing on stress testing these alternatives to identify a suitable plan.
One of the first uses of the term “robustness” in the context of water resources literature was made by Fiering (1976), who merited its origin to statistical theory. A hypothesis was said to be robust if its outcome did not change across a wide range of sample data or other evidence used to test it. In the context of water resources systems, robust designs referred to those designs that displayed minimal deviations in performance when subject to conditions substantially different than those used in the original optimization. Near optimal solutions that performed within an acceptable range of the optimal, but presented different institutional and public acceptability, were considered worth exploring. In this sense, the concept of robustness warranted a departure from focus on the “optimal” and exploration of alternatives that were in the vicinity of the optimal in their performance. On similar lines, Haimes (1977) discussed the value of quantifying the sensitivity of the optimized solution to underlying assumptions about system model and parameterizations. Sensitivity can be quantified analytically as the first (or higher) order derivative of the optimized solution with respect to any underlying choices in problem framing such as model parameterization, exogenous variables values, or constraints. From a risk-minimization perspective, a solution that is less sensitive to underlying choices should be preferred to another, which may have a better performance, but also higher sensitivity.
Hashimoto et al. (1982) provided a definition motivated by the concept of economic flexibility; robustness was the ability of a project design to adapt to a wide range of changing conditions with minimal additional investments. Mathematically, he formulated an expected regret criterion to quantify robustness, regret being the difference between the performance of an alternative in a given future and the optimal design for that future. A design alternative that maximized robustness would then minimize the expected value of regret across all possible futures. Other criteria such as the minimax criterion from game theory could also be used, where a design alternative is selected that minimizes the worst-case regret. These attempts to deal with (deep) uncertainty were generally limited to exploring the impact of different probability distributions of uncertain factors on benefit-cost ratios and by eliciting best- and worst-case scenarios by interviewing experts (Goicoechea et al., 1982).
A number of decision analysis frameworks were developed in the 21st century that prioritized identification of robust designs. Examples include robust decision-making, decision scaling, info-gap, and multi-objective robust decision-making (MORDM; Bhave et al., 2016; Hadka et al., 2015; Hipel & Ben-Haim, 1999; Kasprzyk et al., 2013). The application of these methods to a long-term planning problem broadly is composed of four stages, with variations in the specific procedures or order of execution of different stages. The first stage was structuring the decision problem. This included identifying the decisions to be made, the performance metrics to compare outcomes, uncertainties involved, and the system model mapping decisions to outcomes. In the second stage, decision alternatives were identified via multi-objective optimization (in MORDM) or by prior specification in other methods. A distinct advantage of MORDM over others is that multi-objective optimization generated a suite of possible compromise solutions, which can be further evaluated in possible future states-of-the-world.
In the third stage, a metric of robustness was used to quantify the performance of a candidate design in possible future states-of-the-world. Typically, a large number of future states were constructed, say tens or hundreds of thousands. Future states-of-the-world were generated by combining information on future climate projections, possible demands changes, and other scenarios that may be identified by stakeholders. Following this, a robustness metric summarized the performance of a design alternative across all states-of-the-world. A large number of robustness metrics were developed to represent varying risk attitudes and diversity of stakeholder interests (Herman et al., 2015; McPhail et al., 2018). A metric commonly employed to water resources planning is the Starr’s domain criterion that defined robustness as the percentage of states-of-the-world in which a design alternative maintains performance above predefined thresholds (Trindade et al., 2017). The thresholds could be specified for multiple system-related performance measures or state variables, resulting in a multivariate robustness metric. Robustness metrics then allowed a comparison of performance of optimized or prespecified design alternatives under deeply uncertain futures.
The fourth and final stage of robustness analysis was termed “scenario discovery,” which enabled the identification of main drivers of failure or success of design alternatives. Visualization or data mining of the large amount of information produced during the robustness assessment helped analysts locate regions of vulnerability in multidimensional space of unknown factors. This allowed analysts to identify which combinations of climate and demand changes are likely to cause unacceptable performance deterioration in a reservoir operation. Algorithms such as the patient rule induction method (PRIM) and classification and regression trees have been used in this context (Lempert et al., 2008). This information can then be used to identify modifications to existing plans or new plans that performed favorably across the sampled scenarios. An important component of robust decision-making was the development of interactive visuals to enable decision-makers to iteratively explore the consequences of choosing different policies or identify new scenarios to test these policies on. A related technique of robust optimization attempts to include robustness considerations during the search phase of MORDM. Although conceptually advantageous, robust optimization considering multiple objective and uncertainties is faced with the challenge of increasing computational complexity in the optimization resulting from the need to evaluate robustness of each design against a large number of possible futures (Bartholomew & Kwakkel, 2020).
It is natural to expect that the inferences drawn from robustness analysis would depend on the choice of robustness metrics, which in turn depend upon how stakeholders perceive vulnerabilities in their systems. As water infrastructure projects generally involve multiple actors spatially distributed within one or more river basins and concern multiple water related sectors, identification of a robustness metric considering this diversity has been a major challenge (Herman et al., 2015). Coproduction of knowledge with stakeholders to guide robustness analysis is perhaps the only practically useful way to define robustness (Borgomeo et al., 2018). However, sociopolitical barriers may prohibit such an engagement. The existing literature has yet to provide an adequate resolution of this issue. It follows that defining robustness metrics for large-scale water infrastructure projects remains a challenge. Identifying robust designs is an iterative process between the stakeholders and analysts and requires stakeholder inputs at several stages, which are in turn guided by the outputs of the analysis.
Bottom-up approaches such as decision scaling are also based on similar principles and aim to identify critical thresholds of climate or socioeconomic changes beyond which the performance of a strategy degrades to unacceptable levels (Brown et al., 2012; Culley et al., 2016; Poff et al., 2016). Bottom-up approaches provide more relevant information to decision-makers as opposed to the traditional top-down frameworks for climate impact assessment as they explore a larger range of future conditions than those projected by climate model outputs. On the one hand, top-down frameworks begin with processing of climate model outputs via downscaling techniques. Downscaling climate data is then used to force a water resources systems model to quantify performance of a design under projected climates. A suitable design alternative can then be selected based on the acceptability of its performance across climate futures. On the other hand, bottom-up approaches begin with the identification of relevant indicators of interest to stakeholders and performance thresholds for those indicators. Thus, stakeholder engagement is the first step in a bottom-up analysis while it may occur in the later stages in the top-down decision framework. Top-down frameworks also suffer from the issue of uncertainty cascading through each stage, resulting in potentially large uncertainties in resultant indicators of stakeholder interest. These large uncertainties offer little information to planners. By exploring performance across a range of climate futures and focusing the analysis on identification of critical thresholds, stakeholders can better understand the vulnerability of design alternatives in bottom-up approaches. It is worth noting that such frameworks are a subset of the scenario discovery process that is the final stage of MORDM.
An important step in robustness analysis is the generation of alternative states-of-the-world that represent possible future climates and socioeconomic conditions. In earlier developments by Hashimoto et al. (1982) and others, formal mathematical techniques used to quantify robustness largely relied on probability theory to assign probabilities to possible future hydroclimatic states. More recent efforts have focused on synthetic scenario generation using stochastic generators that can produce a large number of realizations of future streamflow by perturbing parameters of a stochastic generator (Herman et al., 2020). These allow the exploration of changes in long-term mean of streamflow, its seasonality and inter-annual variance, drought severity, as well as monsoonal dynamics. Such methods employ climate model projections either to perturb a historically based stationary stochastic process or to directly infer a nonstationary stochastic process. Advanced generators also consider additional factors such as preservation of multisite and multivariable correlations. As an example, the correlation between temperature and precipitation at a site or between streamflow realizations across different sites in a region could be preserved.
A common way to sample future socioeconomic conditions is to generate discrete scenarios of demand or population changes that can either be based on stakeholder inputs or projections of future socioeconomic conditions from state-of-the-art integrated assessment models. Alternatively, an ensemble of future changes in demands can be generated by sampling a multiplier within a prespecified range. An alternative to sophisticated scenario generation methodologies is the derivation of storylines for regional changes, coproduced with stakeholders and climate scientists (Shepherd, 2019). Storylines combine advanced knowledge of regional climate change with existing projection information to identify possible trajectories that future climate may take in a region. This information can be combined with socioeconomic scenarios resulting in several states-of-the-world to explore. Though likely resulting in a fewer number of scenarios than more sophisticated approaches, scenarios developed this way are more likely to be received by planners given their involvement in the scenario-generation process.
Decision frameworks discussed so far primarily focus on the evaluation of alternative designs against a large ensemble of possible climate and socioeconomic conditions. The use of robustness metrics provides a means to compare alternative actions but does not explicitly suggest how stakeholders should adopt designs as new information is obtained. Adaptive planning frameworks recognize that long-term planning requires planners to explicitly quantify when and how new information should be used. It allows consideration of time evolution of system models, parameterizations, forcing, or the preference structure guiding the choice of actions. Examples of such frameworks include adaptive policymaking, adaptation pathways, dynamic adaptive policy pathways, and engineering options analysis (Haasnoot et al., 2013; Walker et al., 2013). Broadly, dynamic plans are based on the premise that as the future unfolds, more information would be obtainable upon which future actions could be based. This allows planners to identify a plan that has adaptation built into its design and avoids ad hoc changes to actions in response to future events.
There are three important characteristics of plans designed to adapt to changing conditions. First, they differentiate between near-term and long-term actions by providing a categorization of action types. Adaptive plans classify actions as near term that are time urgent with preferably low regret, and others that are taken either to reduce known or expected vulnerabilities or those that are taken to update a policy when certain conditions are met. Mitigating actions aim to reduce known vulnerabilities of a policy, and hedging actions reduce vulnerabilities against possible failures. Similarly, corrective actions are those that can be taken once a threshold of system state variable or performance metric is exceeded. Second, the policy design explicitly identifies signposts that may be an event or a system-related threshold, which when exceeded necessitates corrective or defensive actions. The critical values of relevant state variables or performance metrics that should enable an adaptive response are termed as triggers. Third, dynamic plans incorporate learning explicitly in the design process and provide formal mechanisms to trade-off present benefits for more information if deemed necessary.
As dynamic plans develop alternative pathways that sequence different action types based on predetermined triggers under deeply uncertain futures, these frameworks are far more sensitive to uncertainty characterization process. In summary, dynamic plans are incremental in nature, focusing on near-term actions along with a sequencing of future actions conditional on predetermined triggers. Applications in water resource planning have used dynamic adaptive policy pathways (DAPP) or other optimization-based dynamic frameworks such as direct policy search or stochastic dynamic programming (Haasnoot et al., 2013; Kim et al., 2021; Lawrence & Haasnoot, 2017; Ranger et al., 2013; Salazar et al., 2017). DAPP developed by Haasnoot et al. (2013) has emerged as a prominent method for adaptive planning considering deep uncertainties. It focuses on development of transient scenarios that specify how relevant uncertainties develop over time. It specifies different types of actions and their sequencing into adaptation pathways and a monitoring procedure. A comparison of the flow of analysis in robust and adaptive planning approaches is presented in Figure 1.

Figure 1. The flow of analysis for multi-objective robust decision-making (MORDM) and dynamic adaptive policy pathways (DAPP) approaches for robust and adaptive planning, respectively. MORDM’s strength lies in enabling a detailed assessment of performance of strategies under a wide range of deeply uncertain scenarios using a multi-variate robustness metric. DAPP focuses on the development of adaptive plans that aid in achieving robustness under conditions of deep uncertainty. The figure emphasizes the complementarity of these approaches. Well-characterized (deep) uncertainties refer to those for which a probability distribution can (cannot) be specified.
Example Application: The Red River Basin, Vietnam
As an example of quantitative decision analytic approaches employed in the water resources literature, the operational planning of the water infrastructure in the Red River basin is discussed. The Red River basin is Vietnam’s second-largest river basin, spanning 25 provinces with a population of 32 million people as of 2014 (Nguyen et al., 2021). The river supports agriculture, fisheries, and aquaculture. But monsoonal flows often cause flood damages downstream of the river. The capital city of Hanoi located in the Red River delta is particularly vulnerable to flooding events. To help manage floods as well as secure water, food, and energy-related services, several reservoirs have been constructed on various tributaries of the river. The simultaneous operation of these reservoirs presents a multi-objective multireservoir control problem. The nonlinear high-dimensional nature of the control problem and the time-inseparable nature of engineering objectives and constraints necessitated the use of evolutionary algorithms to identify Pareto-optimal reservoir operation strategies (Giuliani et al., 2016; Nguyen et al., 2021; Quinn et al., 2017). Evolutionary multi-objective direct policy search (EMODPS) has been used to identify trade-offs between flood protection for Hanoi city, water supply, and hydropower production by optimizing operations of the system’s multiple reservoirs (Giuliani et al., 2016; Quinn et al., 2017). EMODPS combines the search capacity of multi-objective evolutionary algorithms with flexibility of direct policy search to formulate adaptive state-aware reservoir operating rules. Giuliani et al. (2016) report a clear conflict between hydropower production and flood protection–related objectives. This occurs as minimizing flood damages requires the reservoirs to be maintained at low levels while maximizing hydropower production favors keeping them at their maximum level. Quinn et al. (2017) further explored multiple formulations of the control problem, which varied in the quantitative representation of the problem objectives. A major finding was that strategies aimed at minimizing expected flood damages, estimated using a damage function, may result in increasing the risk of catastrophic flood events. This was mainly attributed to the unbounded nonlinear nature of penalty function that quantify flood damages as a function of overtopping height.
Giuliani et al. (2016) carried out a scenario analysis to identify potential vulnerabilities of reservoir operating policies to climate change. They report a general deterioration of system performance for all objectives under a changing climate, based on five climate scenarios constructed from a downscaled five-member perturbed physics ensemble of HaCM3 general circulation model. Quinn et al. (2018) also evaluated control policies optimized under assumptions of stationarity against a range of scenarios constructed to represent nonstationary monsoonal dynamics and socioeconomic conditions. Following the literature on robust planning, they quantified the robustness of a policy using a satisficing metric that estimates the percent of scenarios within which a policy satisfies given performance benchmarks. Scenario discovery was carried out using logistic regression to identify the combinations of hydrologic and socioeconomic factors that result in policy failure under deeply uncertain conditions. Using this approach, they discovered that several combinations of hydrologic and socioeconomic changes may result in increasing flood risks for Hanoi as well as negatively affecting water supply for agriculture. Together these analyses highlight the complex effects of problem formulations on the resultant perception of policy success as well as the potential for changes in monsoonal dynamics to increase flood and water deficit-related risks in the Red River system.
Conclusion
Long-term planning of water resources constitutes the suite of most challenging decision problems due to the complex nature of the underlying coupled human-nature systems and large uncertainties in future hydroclimatic and socioeconomic conditions. The presence of multiple stakeholders with potentially conflicting preferences, which evolve with time, also complicates the decision process. In addition to consideration of deep uncertainties regarding the choice of system model, parameterization, and forcing scenario specifications, planners need to give due attention to risk attitudes and ethical assumptions embedded in the decision analysis. Responding to these challenges, the literature on water management has seen tremendous progress over the 21st century. Advancements have been made in the methods to characterize hydroclimatic uncertainty, using multiple problem formulations to explore the consequences of choices involved in structuring water management problems, defining and using the concept of robustness, and developing dynamic adaptive plans that respond to new information. Simultaneously, there has been a growth in available optimization approaches considering stochastic uncertainties and multiple preferences exploiting the increasing computational power through parallel computing. Other aspects that have witnessed increased attention include the improvement of underlying systems models that explicitly consider feedback between the coupled human-water systems, quantifying the value of regional cooperation in water management, and simultaneous consideration of long-term and short-term planning actions within the decision analysis. Finally, increased attention has been given to systematically incorporating stakeholder-elicited information in the decision process for defining critical performance thresholds, robustness measures, and relevant scenarios. Water resources planning under deep uncertainty has evolved into an interdisciplinary problem with experts from policy analysis, hydrology, climate science, human geography, economics, ecology, and other allied areas coming together to provide solutions.
Despite these advances, there continues to be growing conflicts around water, highlighting the need to synthesize information across the vast body of existing literature. Water resources planning continues to be plagued by the challenge of making the complex decision frameworks accessible to water planners, who have historically relied on simpler analytical procedures, especially in developing countries. Bringing the scientific insights gained from advanced decision analytical frameworks into day-to-day planning and operation of water infrastructure requires efforts to improve and update existing water resource–related curricula at undergraduate and graduate levels, development of open source software and associated toolkits, and tailoring the decision frameworks to variable legislative and political contexts.
Further Reading
- Haasnoot, M., Kwakkel, J. H., Walker, W. E., & Ter Maat, J. (2013). Dynamic adaptive policy pathways: A method for crafting robust decisions for a deeply uncertain world. Global Environmental Change, 23(2), 485–498.
- Hashimoto, T. (1980). Robustness, reliability, resilience and vulnerability criteria for planning water resources systems. Cornell University.
- Herman, J. D., Quinn, J. D., Steinschneider, S., Giuliani, M., & Fletcher, S. (2020). Climate adaptation as a control problem: Review and perspectives on dynamic water resources planning under uncertainty. Water Resources Research, 56(2), e24389.
- Herman, J. D., Reed, P. M., Zeff, H. B., & Characklis, G. W. (2015). How should robustness be defined for water systems planning under change? Journal of Water Resources Planning and Management, 141(10), 04015012.
- Kasprzyk, J. R., Nataraj, S., Reed, P. M., & Lempert, R. J. (2013). Many objective robust decision making for complex environmental systems undergoing change. Environmental Modelling & Software, 42, 55–71.
- Loucks, D. P., & Van Beek, E. (2017). Water resource systems planning and management: An introduction to methods, models, and applications. Springer.
- Marchau, V. A., Walker, W. E., Bloemen, P. J., & Popper, S. W. (2019). Decision making under deep uncertainty: From theory to practice. Springer Nature.
- Reed, P. M., Hadka, D., Herman, J. D., Kasprzyk, J. R., & Kollat, J. B. (2013). Evolutionary multiobjective optimization in water resources: The past, present, and future. Advances in Water Resources, 51, 438–456.
- Simonović, S. P. (2012). Managing water resources: Methods and tools for a systems approach. Routledge.
- Walker, W. E., Haasnoot, M., & Kwakkel, J. H. (2013). Adapt or perish: A review of planning approaches for adaptation under deep uncertainty. Sustainability, 5(3), 955–979.
References
- Afshar, A., Sharifi, F., & Jalali, M. R. (2009). Non-dominated archiving multi-colony ant algorithm for multi-objective optimization: Application to multi-purpose reservoir operation. Engineering Optimization, 41(4), 313–325.
- Asefa, T., Clayton, J., Adams, A., & Anderson, D. (2014). Performance evaluation of a water resources system under varying climatic conditions: Reliability, resilience, vulnerability and beyond. Journal of Hydrology, 508, 53–65.
- Askew, A. J. (1974). Optimum reservoir operating policies and the imposition of a reliability constraint. Water Resources Research, 10(1), 51–56.
- Bartholomew, E., & Kwakkel, J. H. (2020). On considering robustness in the search phase of robust decision making: A comparison of many-objective robust decision making, multi-scenario many-objective robust decision making, and many objective robust optimization. Environmental Modelling & Software, 127, 104699.
- Bellman, R. (1961). Curse of dimensionality. Adaptive control processes: A guided tour. Princeton University Press.
- Bender, M. J., & Simonovic, S. P. (2000). A fuzzy compromise approach to water resource systems planning under uncertainty. Fuzzy Sets and Systems, 115(1), 35–44.
- Bhave, A. G., Conway, D., Dessai, S., & Stainforth, D. A. (2016). Barriers and opportunities for robust decision making approaches to support climate change adaptation in the developing world. Climate Risk Management, 14, 1–10.
- Bodo, B., & Unny, T. E. (1976). Model uncertainty in flood frequency analysis and frequency‐based design. Water Resources Research, 12(6), 1109–1117.
- Borgomeo, E., Mortazavi‐Naeini, M., Hall, J. W., & Guillod, B. P. (2018). Risk, robustness and water resources planning under uncertainty. Earth’s Future, 6(3), 468–487.
- Bower, B. T. (1965). Some important research problems in the water resources field. Natural Resources Journal, 5, 286.
- Brown, C., Ghile, Y., Laverty, M., & Li, K. (2012). Decision scaling: Linking bottom‐up vulnerability analysis with climate projections in the water sector. Water Resources Research, 48(9), W09537.
- Chang, L. C., & Chang, F. J. (2009). Multi-objective evolutionary algorithm for operating parallel reservoir system. Journal of Hydrology, 377(1–2), 12–20.
- Charnes, A., & Cooper, W. W. (1963). Deterministic equivalents for optimizing and satisficing under chance constraints. Operations Research, 11(1), 18–39.
- Cohon, J. L., & Marks, D. H. (1973). Multiobjective screening models and water resource investment. Water Resources Research, 9(4), 826–836.
- Culley, S., Noble, S., Yates, A., Timbs, M., Westra, S., Maier, H. R., Giuliani, M., & Castelletti, A. (2016). A bottom‐up approach to identifying the maximum operational adaptive capacity of water resource systems to a changing climate. Water Resources Research, 52(9), 6751–6768.
- Davis, D. R., Kisiel, C. C., & Duckstein, L. (1972). Bayesian decision theory applied to design in hydrology. Water Resources Research, 8(1), 33–41.
- Dessai, S., & Hulme, M. (2007). Assessing the robustness of adaptation decisions to climate change uncertainties: A case study on water resources management in the East of England. Global Environmental Change, 17(1), 59–72.
- Erlenkotter, D., Sethi, S., & Okada, N. (1989). Planning for surprise: Water resources development under demand and supply uncertainty I. The general model. Management Science, 35(2), 149–163.
- Esogbue, A. O., & Ahipo, Z. M. (1982). A fuzzy sets model for measuring the effectiveness of public participation in water resources planning. Journal of the American Water Resources Association, 18(3), 451–456.
- Fiering, M. B. (1964). Multivariate technique for synthetic hydrology. Journal of the Hydraulics Division, 90(5), 43–60.
- Fiering, M. B. (1976). The role of systems analysis in water program development. Natural Resources Journal, 16(4), 759–771.
- Giuliani, M., Anghileri, D., Castelletti, A., Vu, P. N., & Soncini-Sessa, R. (2016). Large storage operations under climate change: Expanding uncertainties and evolving tradeoffs. Environmental Research Letters, 11(3), 035009.
- Goicoechea, A., Krouse, M. R., & Antle, L. G. (1982). An approach to risk and uncertainty in benefit‐cost analysis of water resources projects. Water Resources Research, 18(4), 791–799.
- Golmohammadi, M. H., Safavi, H. R., Sandoval-Solis, S., & Fooladi, M. (2021). Improving performance criteria in the water resource systems based on fuzzy approach. Water Resources Management, 35(2), 593–611.
- Haasnoot, M., Kwakkel, J. H., Walker, W. E., & Ter Maat, J. (2013). Dynamic adaptive policy pathways: A method for crafting robust decisions for a deeply uncertain world. Global Environmental Change, 23(2), 485–498.
- Hadka, D., Herman, J., Reed, P., & Keller, K. (2015). An open source framework for many-objective robust decision making. Environmental Modelling & Software, 74, 114–129.
- Haimes, Y. Y. (1977). Hierarchical analyses of water resources systems: Modeling and optimization of large scale systems McGraw-Hill Higher Education, 512 pp.
- Haimes, Y. Y. (1984). Integrated risk and uncertainty assessment in water resources within a multiobjective framework. Journal of Hydrology, 68(1–4), 405–417.
- Haimes, Y. Y., & Hall, W. A. (1977). Sensitivity, responsivity, stability and irreversibility as multiple objectives in civil systems. Advances in Water Resources, 1(2), 71–81.
- Hashimoto, T. (1980). Robustness, reliability, resilience and vulnerability criteria for planning water resources systems. Cornell University.
- Hashimoto, T., Loucks, D. P., & Stedinger, J. R. (1982). Robustness of water resources systems. Water Resources Research, 18(1), 21–26.
- Herman, J. D., Quinn, J. D., Steinschneider, S., Giuliani, M., & Fletcher, S. (2020). Climate adaptation as a control problem: Review and perspectives on dynamic water resources planning under uncertainty. Water Resources Research, 56(2), e24389.
- Herman, J. D., Reed, P. M., Zeff, H. B., & Characklis, G. W. (2015). How should robustness be defined for water systems planning under change? Journal of Water Resources Planning and Management, 141(10), 04015012.
- Hipel, K. W., & Ben-Haim, Y. (1999). Decision making in an uncertain world: Information-gap modeling in water resources management. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 29(4), 506–517.
- Holling, C. S. (1978). Adaptive environmental assessment and management. John Wiley & Sons.
- Jenkins, M. W., & Lund, J. R. (2000). Integrating yield and shortage management under multiple uncertainties. Journal of Water Resources Planning and Management, 126(5), 288–297.
- Kaynor, E. R. (1978). Uncertainty in water resource planning in the Connecticut river basin. Journal of the American Water Resources Association, 14(6), 1304–1313.
- Keckler, W. G., & Larson, R. E. (1968). Dynamic programming applications to water resource system operation and planning. Journal of Mathematical Analysis and Applications, 24(1), 80–109.
- Keeney, R. L., & Wood, E. F. (1977). An illustrative example of the use of multiattribute utility theory for water resource planning. Water Resources Research, 13(4), 705–712.
- Kim, G. J., Kim, Y. O., & Reed, P. M. (2021). Improving the robustness of reservoir operations with stochastic dynamic programming. Journal of Water Resources Planning and Management, 147(7), 04021030.
- Klemeš, V. (1967). Reliability of water supply performed by means of a storage reservoir within a limited period of time. Journal of Hydrology, 5, 70–92.
- Klemeš, V. (1981). Applied stochastic theory of storage in evolution. In V. T. Chow (Ed.), Advances in hydroscience (Vol. 12, pp. 79–141). Elsevier.
- Klemes̆, V. (1993). Water storage: Source of inspiration and desperation. [Chester C. Kisiel Twelfth memorial lecture]. Department of Hydrology and Water Resources, University of Arizona.
- Knight, F. H. (1921). Risk, uncertainty and profit. Houghton Mifflin Harcourt.
- Krzysztofowicz, R., & Duckstein, L. (1979). Preference criterion for flood control under uncertainty. Water Resources Research, 15(3), 513–520.
- Kundzewicz, Z. W., & Kindler, J. (1995). Multiple criteria for evaluation of reliability aspects of water resource systems. Publications-Series of Proceedings and Reports-International Association Hydrological Sciences, 231, 217–224.
- Lawrence, J., & Haasnoot, M. (2017). What it took to catalyse uptake of dynamic adaptive pathways planning to address climate change uncertainty. Environmental Science & Policy, 68, 47–57.
- Lempert, R. J., Popper, S. W., & Bankes, S. C. (2003). Shaping the next one hundred years: New methods for quantitative long-term policy analysis. The RAND Pardee Centre, Santa Monica.
- Lempert, R. J., Bryant, B. P., & Bankes, S. C. (2008). Comparing algorithms for scenario discovery. RAND.
- Li, Y. P., Huang, G. H., & Nie, S. L. (2010). Planning water resources management systems using a fuzzy-boundary interval-stochastic programming method. Advances in Water Resources, 33(9), 1105–1117.
- Maass, A., Hufschmidt, M. M., Dorfman, R., Thomas, H. A., Jr., Marglin, S. A., & Fair, G. M. (1962). Design of water resource system. Soil Science, 94(2), 135.
- Major, D. C. (1969). Benefit‐cost ratios for projects in multiple objective investment programs. Water Resources Research, 5(6), 1174–1178.
- Marin, C. M. (1986). Parameter estimation in water resources planning and management: Optimal actions or optimal parameters? Water Resources Research, 22(3), 353–360.
- Matalas, N. C. (1967). Mathematical assessment of synthetic hydrology. Water Resources Research, 3(4), 937–945.
- McPhail, C., Maier, H. R., Kwakkel, J. H., Giuliani, M., Castelletti, A., & Westra, S. (2018). Robustness metrics: How are they calculated, when should they be used and why do they give different results? Earth’s Future, 6(2), 169–191.
- Milly, P. C., Betancourt, J., Falkenmark, M., Hirsch, R. M., Kundzewicz, Z. W., Lettenmaier, D. P., & Stouffer, R. J. (2008). Stationarity is dead: Whither water management? Science, 319(5863), 573–574.
- Moran, P. A. P. (1954). A probability theory of dams and storage systems. Australian Journal of Applied Science, 5, 116–124.
- Nayak, S. C., & Arora, S. R. (1971). Optimal capacities for a multireservoir system using the linear decision rule. Water Resources Research, 7(3), 485–498.
- Nguyen, L. X., Tran, T. D., Hoang, S. T., & Nguyen, P. T. (2021). Optimized operation of red-river reservoirs system in the context of drought and water conflicts. In Water security in Asia (pp. 715–732). Springer, Cham.
- O’Laoghaire, D. T., & Himmelblau, D. M. (1972). Modeling and sensitivity analysis for planning decisions in water resources expansion. Journal of the American Water Resources Association, 8(4), 653–668.
- Poff, N. L., Brown, C. M., Grantham, T. E., Matthews, J. H., Palmer, M. A., Spence, C. M., Wilby, R. L., Haasnoot, M., Mendoza, G. F., Dominique, K. C., & Baeza, A. (2016). Sustainable water management under future uncertainty with eco-engineering decision scaling. Nature Climate Change, 6(1), 25–34.
- Quinn, J. D., Reed, P. M., Giuliani, M., & Castelletti, A. (2017). Rival framings: A framework for discovering how problem formulation uncertainties shape risk management trade‐offs in water resources systems. Water Resources Research, 53(8), 7208–7233.
- Quinn, J. D., Reed, P. M., Giuliani, M., & Castelletti, A. (2019). What is controlling our control rules? Opening the black box of multireservoir operating policies using time‐varying sensitivity analysis. Water Resources Research, 55(7), 5962–5984.
- Quinn, J. D., Reed, P. M., Giuliani, M., Castelletti, A., Oyler, J. W., & Nicholas, R. E. (2018). Exploring how changing monsoonal dynamics and human pressures challenge multireservoir management for flood protection, hydropower production, and agricultural water supply. Water Resources Research, 54(7), 4638–4662.
- Ragade, R. K., Unny, T. E., & Hipel, K. W. (1976). Nonquantitative methods in water resources management. Journal of the Water Resources Planning and Management Division, 102(2), 297–309.
- Raj, P. A. (1995). Multicriteria methods in river basin planning—A case study. Water Science and Technology, 31(8), 261–272.
- Raje, D., & Mujumdar, P. P. (2010). Reservoir performance under uncertainty in hydrologic impacts of climate change. Advances in Water Resources, 33(3), 312–326.
- Ranger, N., Reeder, T., & Lowe, J. (2013). Addressing “deep” uncertainty over long-term climate in major infrastructure projects: Four innovations of the Thames Estuary 2100 project. EURO Journal on Decision Processes, 1(3–4), 233–262.
- Reddy, M. J., & Kumar, D. N. (2009). Performance evaluation of elitist-mutated multi-objective particle swarm optimization for integrated water resources management. Journal of Hydroinformatics, 11(1), 79–88.
- Reed, P. M., Hadka, D., Herman, J. D., Kasprzyk, J. R., & Kollat, J. B. (2013). Evolutionary multiobjective optimization in water resources: The past, present, and future. Advances in Water Resources, 51, 438–456.
- Revelle, C., Joeres, E., & Kirby, W. (1969). The linear decision rule in reservoir management and design: 1. Development of the stochastic model. Water Resources Research, 5(4), 767–777.
- Salazar, J. Z., Reed, P. M., Quinn, J. D., Giuliani, M., & Castelletti, A. (2017). Balancing exploration, uncertainty and computational demands in many objective reservoir optimization. Advances in Water Resources, 109, 196–210.
- Shepherd, T. G. (2019). Storyline approach to the construction of regional climate change information. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 475(2225), 20190013.
- Shin, S., Lee, S., Judi, D. R., Parvania, M., Goharian, E., McPherson, T., & Burian, S. J. (2018). A systematic review of quantitative resilience measures for water infrastructure systems. Water, 10(2), 164.
- Sivapalan, M., Savenije, H. H., & Blöschl, G. (2012). Socio-hydrology: A new science of people and water. Hydrological Processes, 26(8), 1270–1276.
- Stewart, T. J., & Scott, L. (1995). A scenario‐based framework for multicriteria decision analysis in water resources planning. Water Resources Research, 31(11), 2835–2843.
- Szidarovszky, F., Bogárdi, I., Duckstein, L., & Davis, D. (1976). Economic uncertainties in water resources project design. Water Resources Research, 12(4), 573–580.
- Thomas, H. A., & Fiering, M. ?. (1962). Mathematical synthesis of streamflow sequences for the analysis of river basins by simulation. In A. Maass, M. M. Hufschmidt, R. Dorfman, H. A. Thomas Jr., S. A. Marglin, & G. M. Fair (Eds.), Design of Water-Resource Systems (pp. 459–493). Harvard University Press.
- Tilmant, A., Persoons, E., & Vanclooster, M. (1970). Deriving efficient reservoir operating rules using flexible stochastic dynamic programming. WIT Transactions on Ecology and the Environment, 48, 353–364.
- Trindade, B. C., Reed, P. M., Herman, J. D., Zeff, H. B., & Characklis, G. W. (2017). Reducing regional drought vulnerabilities and multi-city robustness conflicts using many-objective optimization under deep uncertainty. Advances in Water Resources, 104, 195–209.
- Tsoukiàs, A. (2008). From decision theory to decision aiding methodology. European Journal of Operational Research, 187(1), 138–161.
- Vicens, G. J., Rodriguez‐Iturbe, I., & Schaake, J. C., Jr. (1975). A Bayesian framework for the use of regional information in hydrology. Water Resources Research, 11(3), 405–414.
- Walker, W. E., Haasnoot, M., & Kwakkel, J. H. (2013). Adapt or perish: A review of planning approaches for adaptation under deep uncertainty. Sustainability, 5(3), 955–979.
- Wang, S., & Huang, G. H. (2011). Interactive two-stage stochastic fuzzy programming for water resources management. Journal of Environmental Management, 92(8), 1986–1995.
- White, G. F. (1964). Choice of adjustment to floods. University of Chicago.
- Wood, E. F. (1978). Analyzing hydrologic uncertainty and its impact upon decision making in water resources. Advances in Water Resources, 1(5), 299–305.
- Wood, E. F., & Rodríguez‐Iturbe, I. (1975). A Bayesian approach to analyzing uncertainty among flood frequency models. Water Resources Research, 11(6), 839–843.
- Wood, E. F., Rodriguez-Iturbe, I., & Schaake, J. C., Jr. (1974). The methodology of Bayesian inference and decision making applied to extreme hydrologic events MIT Department of Civil Engineering, Ralph M. Parsons laboratory for water resources and hydrodynamics Rep. 178. p. 297.
- Yakowitz, S. (1982). Dynamic programming applications in water resources. Water Resources Research, 18(4), 673–696.
- Zadeh, L. A., Fu, K. S., Tanaka, K., & Shimura, M. (Eds.). (2014). Fuzzy sets and their applications to cognitive and decision processes. Proceedings of the US-Japan seminar on fuzzy sets and their applications (pp. 1–496, held at the University of California, July 1–4, 1974). Academic Press.