Catchment Change Network.net

The morning talks gave a perspective on handling uncertainties from the insurance industry perspective. It is clear that they know only too well that their predictions of expected losses from extreme natural events are often based on rather uncertain input data and model components (and exposure to losses not currently included in models) but that they are already looking forward to being able to take account of some of the relevant uncertainties.   One of the issues in doing so however was that some of the current models will take a week or two to run a single deterministic loss calculation. There was some hope that a new generation of computer technology, such as the use of graphics processing units (GPUs), would reduce model run-times sufficiently to allow some assessment of uncertainty (they clearly have not tried programming a GPU yet, though this is getting easier!). One presentation suggested that being able to make more and more runs would allow uncertainties to be reduced.  Over lunch I asked what he really meant by this… it seemed that it was only that the estimation of probabilities for a given set of assumptions could be made more precise given more runs.

There was a demonstration of this in the afternoon in an interesting study to estimate the uncertainty in losses due to hurricanes in Florida.  5 insurance modelling companies had been given the same data and asked to estimate both the expected loss for given return periods of events (up to 1000 years) and a 90% confidence range.Two of the companies had run multiple long term realisations of a given sample distribution of events based on the prior distributions of event parameters.  Their confidence limits became smaller as the number of realisations increased and improved the integration over the possible distribution of events allowed by the fixed prior distributions. Two other companies had taken a different strategy, running realisations of a length consistent with historical data periods and resulting in much wider uncertainty limits.  Uncertainty estimations, particularly when not conditioned on historical data, will always depend directly on the assumptions on which they are based!  An analysis of the Florida hurricane study had suggested that the uncertainty in the estimated hazard was more important than uncertainty in the estimated vulnerability. I am not sure that this would necessarily be the case in estimating flood risk.

There was some discussion of how to convey these assumptions to the people who actually take the risk for insurance companies in committing to contracts, and whether they should be allowed to play with dials that would allow sensitivities of estimated losses to vary with different parameters.  Given long model run times and short decision times in the real world this was generally not considered feasible (although more flexibility to explore model sensitivities rather than the ‘black box’ results provided currently, was suggested). There was also a suggestion that it was as important to “understand what is not in the models” as to understand sensitivities to what was in the models and that “adding more science” would not necessarily be considered advantageous in an industry with a 300 year old tradition.

One thought that came to me during the meeting was inspired by a passing mention of the verification of uncertainty estimates.  It seems to me that this would (a) be very difficult with any form of extreme event and (b) would never happen anyway because data from a new extreme event will be used to revise estimates of prior probabilities that might have been used in estimating uncertainties.  We know that this happens in flood risk estimation when every new extreme flood is used to revise the estimates of the probabilities of exceedence at a site.  Enough for now, it was an early start this morning!!

Farming and land management in Wales can deliver huge benefits for society and the agriculture industry, according to the Head of Agriculture, Forestry and Soils at the Directorate General for the Environment at the European Commission.

More than 90 per cent of Wales is either forestry or farmland. How we manage this land in the future, will be vital to combating climate change and improving our environment.

Different land management techniques could help to store more carbon in the soil, maintain healthier river levels for people and wildlife and keep more rainwater out of rivers and reduce the risk of flooding to rural and urban communities.

It could also contribute to cleaning up the water in Welsh rivers to meet new tougher standards and make a difference to the bathing water quality at beaches throughout Wales.

These changes have recently been recognised in the new agri-environment scheme, Glastir, announced by the Welsh Assembly Government, earlier this year.

Further reading.

This has some implications about predicting the effects of change since we clearly cannot easily test hypotheses (or sets of modelling assumptions) about what might happen in a particular catchment of interest in the future, we more usually rely on testing hypotheses under current conditions and, given a degree of belief that we are getting the right results for the right reasons, explore the consequences for scenarios of future change. Increasing that degree of belief is the purpose of testing but there are two difficulties involved in this process.  The first is that, as with classical statistical hypotheses testing, there is a possibility of making Type I (false positives, or incorrectly accepting a poor model) or Type II (false negatives, or incorrectly rejecting a good model), particular when there are observational errors in the data being used in testing.  The second is that this process of predicting future change relies on a form of uniformitarianism principle; i.e. that a model that has survived current tests has the functionality required to predict the potentially different future conditions. In both cases, classical hypothesis testing will be limited by epistemic errors (see the previous entry of 27^th September) in the observations and in our expectations about future processes.

That does not mean, however, that we should not try to test models as hypotheses, only that new ways of doing so might be required. We could, for example, explore the possibility of using real-world analogues for different scenarios of future conditions with (approximately) the right combinations of expected temperatures, land use and rainfalls to show that, if there are significant differences in processes the predictive model can represent them acceptably.  The analogues would not, of course, be perfect (uniqueness of place suggests that calibrated parameter values would also necessarily reflect other factors) but this might increase the degree of belief in model predictions of future change rather more than relying on a model that has only been shown to reproduce historical conditions at the site of interest.  As far as I know, no such study has been reported (although analogues have been used in other ways)…does any reader know of such a study?

It will not be surprising that many of the issues overlap with those that arose in the sessions at Hyderabad (see last entry). The discussions touched on the definition of risk, the assessment of model adequacy, the quantification of hazard and risk, and techniques for the visualisation and communication of uncertainties. There were interesting presentations from David Spiegelhalter on methods used in the medical sciences and Roger Cooke on methods used in the elicitation of expert opinions.

John Rees, the NERC Theme Leader for Natural Hazards, raised the following questions that he felt were important for this scoping study to address:

• If model uncertainty is needed to better inform policy decisions how is it best quantified?
• How should alternative conceptual models and evidence contradictions be used in policy and decision making?
• Is the mean value the appropriate safety metric to inform decisions?
• What is best way to represent scientific consensus?
• What are useful mechanisms for integrating risk and uncertainty science into policy development?
• What should be addressed by the research councils (there is a provisional budget of £1.5m available to support the research programme)?

There was a general recognition amongst the participants, who covered a range of different natural hazards, that the proper evaluation of hazard and risk is often difficult, in that we often have only sparse or no data with which to try and quantify sources of uncertainty and that there may be many different alternative predictive models of varying degrees of approximation. These are the epistemic uncertainties but there was not much discussion about how these might be reflected in the quantification of risk. Many participants seemed to accept that the only way to attempt such a quantification was using statistical methods. I am not so sure.

It is true that any assessment of uncertainty will be conditional on the implicit or explicit assumptions made in the assessment (which might involve treating all sources of uncertainty as if they can be treated statistically). It is also true that those assumptions should be checked for validity in any study (though this is not always evident in publications). But if the fact that the uncertainties are epistemic means that the errors are likely to strong structure and non-stationarity that will depend on a particular model implementation, then it is possible that alternative non-statistical methods of uncertainty estimation might be appropriate.

I have been trying to think about this in the context of testing models as hypotheses given limited uncertain data (something that frequently arises in the focus areas of CCN). Hypothesis testing means considering both Type I and Type 2 errors (accepting false positives and rejecting false negatives). An important areas of CCN is how to avoid both types of errors in model hypothesis testing so that in prediction we are more likely to be getting the right results for the right reasons. So an interesting question is what constitutes an adequate hypothesis test, adequate in the sense of being fit for purpose. This question was addressed, at least indirectly, by Britt Hill of the US Nuclear Regulatory Commission in a talk about the performance assessment process for the safety case for the Yucca Mountain repository site.

In that study, Monte Carlo simulation was used to explore a wide range of potential outcomes (in terms of future dose of radioactivity to a local population over a period of the next 1 million years or so. A cascade of model components from infiltration to waste leaching was involved in these calculations, each depending on multiple (uncertain) model components. The Monte Carlo experiments spanned a range of alternative conceptual models and possible model parameters. Decisions about which models to run appeared to have been produced by scientific consensus, something that Roger Cooke had earlier suggested was not necessarily the best way of extracting information from experts.

There is no explicit hypotheses testing in this type of approach, only some qualitative assessment of whether performance is “reasonably supported” in terms of predictions in past studies, history matching, scientific credibility etc. But it is sometimes the case that, for whatever, reasons, even the best models do not provide acceptably predictions for all times and all places. This could be because of errors in the forcing data, it could be because of model structural error, it could be because of error in the data with which a model is evaluated. It remains impossible to really separate out these different sources of error, and it therefore means that it is difficult to do rigorous hypothesis testing for this type of environmental model.

This seems to be an area where further research is needed. It is surely important in developing guidance for model applications within each of the three CCN focus areas…

Further reading.