Forecast Verification

33 %
67 %
Information about Forecast Verification

Published on October 5, 2007

Author: Spencer


Forecast Verification:  Forecast Verification Presenter: Neil Plummer National Climate Centre Lead Author: Scott Power Bureau of Meteorology Research Centre Acknowledgements A. Watkins, D. Jones, P. Reid, NCC Introduction:  Introduction Verification - what it is and why it is important? Terminology Potential problems Comparing various measures Assisting users of climate information What is verification?:  What is verification? “check truth or correctness of” “process of determining the quality of forecasts” “objective analysis of degree to which a series of forecasts compares and contrasts with the equivalent observations of a given period” Why bother with verification?:  Why bother with verification? Scientific admin support is a new system better? assist with consensus forecasts Application of forecasts “how good are your forecasts?” “should I use them?” can be used to help estimate value Terminology can be confusing:  Terminology can be confusing Verification is made a little tricky by the fact that everyday words are used to describe quantities with a precise statistical meaning. Common words include: accuracy skill reliability bias value hit rates, percent consistent, false alarm rate, ... all have special meanings in statistics Accuracy:  Accuracy Average correspondence between forecasts and observations Measures mean absolute error, root mean square error Bias:  Bias Correspondence between average forecast with average observation e.g. average forecast - average value of observation Skill:  Skill Accuracy of forecasts relative to accuracy of forecasts using a reference method (e.g. guessing, persistence, climatology, damped persistence, …) Measures numerous! Reliability:  Reliability Degree of correspondence between the average observation, given a particular forecast, and that forecast taken over all forecasts e.g. suppose forecasts of : “10% or 30% or , …, or 70% or … chance of rain tomorrow” are routinely issued for many years if we go back through all of the forecasts issued a forecast of looking for occasions when forecast probability of 70% was issued, then we would expect to find rainfall on 70% of occasions if the forecast system is “reliable” this is often not the case Slide10:  Reliability Graph Value:  Value Impact that prudent use of a given forecast scheme has on the user’s profits, COMPARED WITH profits made using a reference strategy Measures $, lives saved, disease spread reduced, … Slide12:  Contingency Table OBSERVED HIT RATE = Hits/(Hits + Misses) FALSE ALARM RATE = False Alarms/(False Alarms + Correct Rejections) PERCENT CONSISTENT = 100*(Hits+Correct Rejections)/Total Accuracy measures:  Accuracy measures Hit rates Proportion of observed events correctly forecast False alarm rates Proportion of observed non-events forecasted as events Percent Correct 100x (proportion of all forecasts that are correct) 1. Forecast performance 2x2 contingency table:  1. Forecast performance 2x2 contingency table Is this a good scheme?:  Is this a good scheme? 1. Original Scheme: Percent correct = 100(28 + 2680)/2803 = 96.6% so it is a very accurate scheme! or is it? 2. Performance of 2nd (reference) forecast method: never predict a tornado = a “lazy” forecast scheme!:  2. Performance of 2nd (reference) forecast method: never predict a tornado = a “lazy” forecast scheme! Performance measures:  Performance measures 1. Original Scheme: Percent correct = 100(28 + 2680)/2803 = 96.6% 2. Reference Lazy Scheme: Percent correct = 100(0 + 2752)/2803 = 98.2% !! Percent Correct: Performance measures:  Performance measures Hit rates: ) 28/51 … so over half the tornadoes predicted ) reference scheme: 0/51 … no tornadoes predicted Value:  Value Suppose an unexpected (unpredicted) tornado causes $500 million damage and that an expected (predicted) tornado results in only $100 million damage So forecast scheme (1) saves 28 x 400 million compared to forecast scheme (2) a huge saving - highly valuable!! Categorical versus probabilistic:  Categorical versus probabilistic Categorical “The temperature will be 26ºC tomorrow” Probabilistic “There is a 30% chance of rain tomorrow” “There is a 90% chance that wet season rainfall will be above median” Artificial Skill :  Artificial Skill danger of too many inputs danger of trying too many inputs independent data cross-validation importance of supporting evidence simple plausible hypothesis climate models process studies How do users verify predictions? :  How do users verify predictions? No single answer, however: some switch from probabilistic to categorical media prefer categorical forecasts assessments made on a single season extrapolation How can we assist users in verification:  How can we assist users in verification Increase access to verification information Simplify information Build partnerships media users & user groups other government departments Education (booklets, web, …) Summary:  Summary Verification is crucial but care is needed! Familiarise with terminology used skill, accuracy, value, … No single measure tells the whole story Importance of using independent data in verification Keep it simple Communicating verification results is challenging Users sometimes do their own verification - sobering Most people like to think categorically - challenging Dialogue with end-users is very important

Add a comment

Related presentations

Related pages

forecast verification - CAWCR

Issues What is forecast verification? If we take the term forecast to mean a prediction of the future state (of the weather, stock market prices, or ...
Read more

Forecast Verification | NOAA / NWS Space Weather ...

Introduction. Warnings and forecasts of geophysical activity are some of the services provided by the SWPC Space Weather Forecast Office and effective use ...
Read more

Forecast Verification - Weather News - Met Éireann - The ...

Weather News. Forecast Verification 11 November 2015. Forecast Verification “Forecasts possess no intrinsic value. They acquire value through their ...
Read more

Wiley: Forecast Verification: A Practitioner's Guide in ...

Forecast Verification: A Practioner's Guide in Atmospheric Science, 2nd Edition provides an indispensible guide to this area of active research by ...
Read more

Forecast_Verification - World Meteorological Organization

Special Issue of Meteorological Applications on Forecast Verification; Verification guidance and support for WWRP forecast and research demonstration ...
Read more

Forecast Verification: ebook jetzt bei

eBook Shop: Forecast Verification als Download. Jetzt eBook sicher bei Weltbild runterladen & bequem mit Ihrem Tablet oder eBook Reader lesen.
Read more

Forecast verification - Wikipedia

Forecast verification is a subfield of the climate, atmospheric and ocean sciences dealing with validating, verifying as well as determining the predictive ...
Read more

Forecast Verification eBook von - 9781119961079 | Kobo

Lesen Sie Forecast Verification A Practitioner's Guide in Atmospheric Science von mit Kobo. Forecast Verification: A Practioner's Guide in Atmospheric ...
Read more

Forecast verification - Met Office

The Met Office has been producing and verifying short-period (up to five to seven days ahead) forecasts for every tropical cyclone since 1988. Verification ...
Read more

Verification Home - National Weather Service

Developed to enhance capabilities for the verification of gridded aviation weather forecasts made available in the National Digital Forecast Database (NDFD).
Read more