ERRORS.- (ERROR PAGE)     

CONTENTS:  1. TIME DISADVANTAGE,  2. FORECAST AVERAGING,  3. INTERPOLATION,  4. MISSING FORECASTS,  5. DIFFERENT DATA SOURCES 

1.  TIME DISADVANTAGE:  There was a time disadvantage in the EURO comparisons due to its being based on 12 UTC data and  issued 12 hours prior to the other models which are based on 00 UTC data and hence issued 12 hour later.  

The First Half data, presented below in TABLE 1,  includes the summarized totals for the EURO model.  These totals are actually for forecasts verifying at 12 UTC on days 3, 4, 5 and 6 ( 72 hour, 96 hour, 120 hour and 144 hour forecasts).    Figure 1 shows the plotted curves for Forecast Correct, (FC),  and Forecast Error, (FE) derived from those totals.  The FC and FE scores are presented immediately below the forecast hour for all four time periods.  The other models were all based on 00 UTC data so their 60 hour forecast verified at the same time as the EURO's 72 hour forecast, the 84 the same as the EURO's 96 hour and so on.  To offset the difference in forecast projections, a time decay curve was constructed for the EURO model and used to estimate probable corrections to the FC and FE scores for 60 hours, 84 hours, 108 hours and 132 hours.  The FC and FE curves were extended to the left border which represents the 60 hour forecast time.  FC was .04 higher than at 72 hours, and FE was .04 lower.  The intersection of the curves at 84, 108 and 132 hours gave the corrections to be used for the values computed for the EURO's 96, 120 and 144 hour forecasts.  The corrections are given under the hours just below the chart.  

The Second Half portion of the study used full 3, 4, 5 and 6 day forecasts verifying at 00 UTC for all models except the EURO. The EURO forecasts at day 3 and day 4, verifying at 12 UTC, were averaged to produce a forecast verifying at the same time as the other forecasts.  Again there was a time disadvantage and so the EUOR's FC and FE were corrected according to the results in Figure1. 

                                         60 HR.                                     84 HR                                   108 HR                                132 HR

FC CORRECTION          + .04                                       + .04                                       + .04                                    + .03

FE CORRECTION           - .066                                     - . 066                                     - .083                                   - .057

GO TO  1. TIME DISADVANTAGE,  2. FORECAST AVERAGING,  3. INTERPOLATION,  4. MISSING FORECASTS,  5. DIFFERENT DATA SOURCES 

=====================================================================================================================

2.  FORECAST AVERAGING:   Some forecast verification times were not those needed for the comparative study.  For example the GLOBAL model does not have a 132 hour 500 mb output which necessitated averaging the 120 hour and 144 hour forecasts during the First Half of the study.  During the Second Half, the 72 hour output of the ENS12 was missing so the 60 hour and 84 hour forecasts were averaged.  Also, during the Second Half, all of the EURO forecasts were averaged to achieve a 00 UTC verification time.

The averaging process was simply to collect grid point heights from each of the two charts straddling the desired time period, add the values at each point and divide by 2.  The result was used as the forecast by that model for that forecast time.  To quantify the error this would produce, the MRF forecasts for days 3, 4, 5 and 6 were averaged over 15 days using forecasts for days 2.5, 3.5, 4.5, 5.5 and 6.5.  The Forecast Correct (FC) scores are shown in Figure 2.  There is no apparent reason for one forecast time to have more or less error than another forecast time so that combining and averaging the four sets of data produces an average FC error of .0133 for the 60 total forecasts. This error was considered too small to correct the affected scores but they are all notated by an asterisk.

Figure 2. Forecast Correct scores for 15 forecast days, comparing MRF forecasts with averaged MRF forecasts.  

GO TO   1. TIME DISADVANTAGE,  2. FORECAST AVERAGING,  3. INTERPOLATION,  4. MISSING FORECASTS,  5. DIFFERENT DATA SOURCES   

=====================================================================================================================

3.  INTERPOLATION:  Interpolating data from charts using 6 decameter contours produced some error.  Errors tended to be larger when charts lacked latitude and longitude lines.  The list of charts in MODEL ADDRESSES has preferred numbers after each address under each forecast model.  It was considered that values could be more accurately read from the preferred charts.  For example, the Unisys address under the MRF model, was used only as a last resort as the charts lack latitude and longitude lines.   The Edwards AFB addresses were preferred as they  were analyzed in 3 decameter contours.  The maximum error from reading point values from the charts was probably 1 decameter for U. S. points and up to 2 decameters for the Pacific point.  

Assume the maximum error is 1 decameter (+1 and -1), is random and all grid points have the same error. The erroneous gradient introduced from grid point A to grid point B would be zero for the cases when both were +1, and -1, and when there was no error. The gradient error for both +1and -1 at A would be 1 when B is correct. The error would be 2 for cases of opposite signs. The error for those that were correct would be 1 for the number of cases of +1 and -1.

For 100 cases when the maximum error is 1 decameter and occurs 20% percent of the time, there will be 10 cases of +1, 10 cases of -1 and 80 cases when the gradient is correct. The 10 cases of +1 at A will be distributed such that one case will be compared to B's +1, 8 cases with B's zero error, and 1 case with B's -1.  The total error for A's +1 is 10. A's -1 total is also 10. The Zero error cases at A will be 8 for both B's +1 and -1. The total fictitious gradient error for the 100 cases is 36 decameters. Repeating the calculation for a 1 decameter error 40% of the time produces a fictitious error of 64 decameters.  If the error at PAC  was 2 decameter 10% of the time and 1 decameters 20% while at RNO the error was 1 decameter 20% of the time the fictitious error would be 88 decameters  for 100 cases; if the error at RNO was 1 decameter 40% of the time, the error would be 96 for 100 cases.

There are approximately 50 cases of each model in the study with four grid point intervals in each forecast period. There are about 200 cases for each forecast period, which would produce a fictitious gradient of 72 decameters for a 1 decameter error 20% of the time and a 128 decameter error for an error 40% of the time.  Substituting the RNO to PAC figures in the above paragraph for 1 of the 4 grid intervals gives a fictitious error of 98 and 102 for 1 decameter error at RNO 20% of the time and 142 and 146 decameter error for a 1 decameter error at RNO 40% of the time; the error at PAC remaining at 2 decameters 10% and 1 decameter 20% of the time.  

The effect of an error in interpolating values between 60-decameter contours, since no skill is involved, is to add that error to all gradients involved. The MRF 3-day forecast, in the Second Half portion, had 52 cases. The observed gradient was 1654 decameters, the forecast gradient was 1640 decameters and the gradient forecast correctly was 1282.  To simplify calculations, only extreme values in the above paragraph will be used to obtain a range of possible errors. The estimated fictitious forecast gradient errors range from 100 to 150 decameters when applied to the MRF figures. The FC score uses the observed and forecast correct gradients which are reduced to ranges of 1554 and 1504 and 1182 and 1132 respectively.  The Forecast Correct score of .775 would be reduced to a range of .761 to .753.  This suggests a maximum error of only .022 which is less than 3%.  The impression of the author is that the error would probably be less than the least end of the error range for all models except the ENS17 which uses a map scale suitable for ant meteorolgists..

GO TO  1. TIME DISADVANTAGE,  2. FORECAST AVERAGING,   3. INTERPOLATION,  4. MISSING FORECASTS,  5. DIFFERENT DATA SOURCES 

==================================================================================

 4.  MISSING FORECASTS:  A  few forecasts were missing during the study. The most that were missing were 4 in the fourth forecast period for two of the models.  Four missing forecast accounted for about 120 decameters of observed gradient and since the missing forecasts were mainly in the last period when scores were low, probably did not have any affect on the comparison.  The change in the First Half of substituting the  ENS model for the UK model reduced the number of ENS cases from approximately 50 to 38 and may have made some difference comparatively. 

 1. TIME DISADVANTAGE,  2. FORECAST AVERAGING,  3. INTERPOLATION,    5. DIFFERENT DATA SOURCES               

=========================================================================================

5.  DIFFERENT  DATA SOURCES.   The ENS forecasts in the First Half were gathered from different internet addresses.  It was discovered that each source used different numbers of initial situations.  At the time of this study, the MRF ENS forecasts produced by the NCEP, Environmental Modeling Center of the NWS used 17 different initial situations, 12 based on 00 UTC data and 5 based on 12 UTC data.  This ENS model is the ENS17 in the Second Half of the study. The Edwards AFB internet address used only the 12 cases based on 00 UTC data.  The Utah address was used 6 times when the Edwards AFB source was not available and it was based on 4 to 6 of the initial analyses from the 00 UTC data.  

BACK TO START OR GO TO 1. TIME DISADVANTAGE,  2. FORECAST AVERAGING,  3. INTERPOLATION,  4. MISSING FORECASTS,