Satellite-derived irradiance data, as an alternative to ground-based measurements, offer a unique opportunity to verify gridded solar forecasts generated by a numerical weather prediction model. Previously, it has been shown that the mean square errors (MSE) evaluated against ground-based measurements and satellite-derived solar irradiance are comparable, which might warrant the use of satellite-based products for regional forecast verification. In this paper, the 24-h-ahead hourly forecasts issued by the North American Mesoscale forecast system are verified against both ground-based (Surface Radiation Budget Network, or SURFRAD) and satellite-based (National Solar Radiation Data Base, or NSRDB) measurements, at all 7 SURFRAD stations over 2015–2016. Three different MSE decomposition methods are used to characterize—e.g., through association, calibration, refinement, resolution, or likelihood—how well the two types of measurements can gauge the forecasts. However, despite their comparable MSEs, NSRDB is found suboptimal in its ability to verify forecasts as compared to SURFRAD. Nonetheless, if a new forecasting model produces significantly better forecasts than the benchmarking model, satellite-derived data are able to detect such improvements and make conclusions. This article comes with supplementary material (data and code) for reproducibility.

1.
R.
Urraca
,
T.
Huld
,
A.
Gracia-Amillo
,
F. J. M.
de Pison
,
F.
Kaspar
, and
A.
Sanz-Garcia
, “
Evaluation of global horizontal irradiance estimates from ERA5 and COSMO-REA6 reanalyses using ground and satellite-based data
,”
Sol. Energy
164
,
339
354
(
2018
).
3.
R.
Perez
,
J.
Schlemmer
,
A.
Kankiewicz
,
J.
Dise
,
A.
Tadese
, and
T.
Hoff
, “
Detecting calibration drift at ground truth stations: A demonstration of satellite irradiance models' accuracy
,”
in Proceedings of the 2017 IEEE 44th Photovoltaic Specialist Conference
(
2017
), pp.
1104
1109
.
4.
R.
Perez
,
P.
Ineichen
,
K.
Moore
,
M.
Kmiecik
,
C.
Chain
,
R.
George
, and
F.
Vignola
, “
A new operational model for satellite derived irradiances: Description and validation
,”
Sol. Energy
73
,
307
317
(
2002
).
5.
M.
André
,
R.
Perez
,
T.
Soubdhan
,
J.
Schlemmer
,
R.
Calif
, and
S.
Monjoly
, “
Preliminary assessment of two spatio-temporal forecasting technics for hourly satellite-derived irradiance in a complex meteorological context
,”
Sol. Energy
177
,
703
712
(
2019
).
6.
R.
Perez
,
J.
Schlemmer
,
K.
Hemker
,
S.
Kivalov
,
A.
Kankiewicz
, and
J.
Dise
, “
Solar energy forecast validation for extended areas and economic impact of forecast accuracy
,”
in Proceedings of the 2016 IEEE 43rd Photovoltaic Specialists Conference
(
2016
), pp.
1119
1124
.
7.
D.
Yang
,
J.
Kleissl
,
C. A.
Gueymard
,
H. T.
Pedro
, and
C. F.
Coimbra
, “
History and trends in solar irradiance and PV power forecasting: A preliminary assessment and review using text mining
,”
Sol. Energy
168
,
60
101
(
2018
).
8.

MBE=1Nn=1N(forecastnreferencen), RMSE=1Nn=1N(forecastnreferencen)2, where N is the number of forecasts being verified.

9.

nRMSE=n=1N(forecastnreferencen)2/n=1Nreferencen2.

10.

The NAM forecast is run four times a day at 0:00, 6:00, 12:00, and 18:00 UTC. The forecasts issued by the 0:00 run are used herein. In addition, all validations are performed for daylight hours only.

11.
D.
Yang
, “
A correct validation of the National Solar Radiation Data Base (NSRDB)
,”
Renewable Sustainable Energy Rev.
97
,
152
155
(
2018
).
12.
D.
Yang
, “
Kriging for NSRDB PSM version 3 satellite-derived solar irradiance
,”
Sol. Energy
171
,
876
883
(
2018
).
13.
D.
Yang
, “
SolarData: An R package for easy access of publicly available solar datasets
,”
Sol. Energy
171
,
A3
A12
(
2018
).
14.
SolarAnywhere is a paid product. Its adoption in research might be limited as compared to other publicly available alternatives, such as NSRDB. Currently, site adaptation is still considered to be an important preprocessing step when dealing with satellite-based irradiance data;
J.
Polo
,
S.
Wilbert
,
J.
Ruiz-Arias
,
R.
Meyer
,
C.
Gueymard
,
M.
Súri
,
L.
Martín
,
T.
Mieslinger
,
P.
Blanc
,
I.
Grant
,
J.
Boland
,
P.
Ineichen
,
J.
Remund
,
R.
Escobar
,
A.
Troccoli
,
M.
Sengupta
,
K.
Nielsen
,
D.
Renne
,
N.
Geuder
, and
T.
Cebecauer
, “
Preliminary survey on site-adaptation techniques for satellite-derived and reanalysis solar radiation datasets
,”
Sol. Energy
132
,
25
37
(
2016
).
15.
K. E.
Taylor
, “
Summarizing multiple aspects of model performance in a single diagram
,”
J. Geophys. Res.: Atmos.
106
,
7183
7192
, (
2001
).
16.
J.
Zhang
,
A.
Florita
,
B.-M.
Hodge
,
S.
Lu
,
H. F.
Hamann
,
V.
Banunarayanan
, and
A. M.
Brockway
, “
A suite of metrics for assessing the performance of solar power forecasting
,”
Sol. Energy
111
,
157
175
(
2015
).
17.

A set of 5 error metrics were considered, namely, bias, MSE, correlation coefficient, coefficient of determination, and MSE skill score.

18.
Y.
Tian
,
G. S.
Nearing
,
C. D.
Peters-Lidard
,
K. W.
Harrison
, and
L.
Tang
, “
Performance metrics, error modeling, and uncertainty quantification
,”
Mon. Weather Rev.
144
,
607
613
(
2016
).
19.
A. H.
Murphy
,
B. G.
Brown
, and
Y.-S.
Chen
, “
Diagnostic verification of temperature forecasts
,”
Weather Forecasting
4
,
485
501
(
1989
).
20.
H. V.
Gupta
,
H.
Kling
,
K. K.
Yilmaz
, and
G. F.
Martinez
, “
Decomposition of the mean squared error and NSE performance criteria: Implications for improving hydrological modelling
,”
J. Hydrol.
377
,
80
91
(
2009
).
21.
T. R.
Stewart
, “
A decomposition of the correlation coefficient and its use in analyzing forecasting skill
,”
Weather Forecasting
5
,
661
666
(
1990
).
22.
A. H.
Murphy
and
R. L.
Winkler
, “
A general framework for forecast verification
,”
Mon. Weather Rev.
115
,
1330
1338
(
1987
).
23.
A. H.
Murphy
, “
General decompositions of MSE-based skill scores: Measures of some basic aspects of forecast quality
,”
Mon. Weather Rev.
124
,
2353
2369
(
1996
).
24.

By using the bias–variance decomposition, MBE becomes redundant.

25.
L.
Wasserman
,
All of Nonparametric Statistics
(
Springer Science and Business Media
,
New York
,
2006
).
26.
R. J.
Hyndman
,
D. M.
Bashtannyk
, and
G. K.
Grunwald
, “
Estimating and visualizing conditional densities
,”
J. Comput. Graph. Stat.
5
,
315
336
(
1996
).
27.
The automatic bandwidth-selection algorithm [
D.
Ruppert
,
S. J.
Sheather
, and
M. P.
Wand
, “
An effective bandwidth selector for local least squares regression
,”
J. Am. Stat. Assoc.
90
,
1257
1270
(
1995
)] produced an overly smoothed fit, which makes the reconstructed MSE bigger than it should be.
28.
D.
Yang
, “
Ultra-fast preselection in lasso-type spatio-temporal solar forecasting problems
,”
Sol. Energy
176
,
788
796
(
2018
).
29.
C. A.
Gueymard
and
D. R.
Myers
, “
Evaluation of conventional and high-performance routine solar radiation measurements for improved solar resource, climatological trends, and radiative modeling
,”
Sol. Energy
83
,
171
185
(
2009
).
30.
C. A.
Gueymard
, “
Direct and indirect uncertainties in the prediction of tilted irradiance for solar engineering applications
,”
Sol. Energy
83
,
432
444
(
2009
).
31.
E.
Lorenz
,
J.
Hurka
,
D.
Heinemann
, and
H. G.
Beyer
, “
Irradiance forecasting for the power prediction of grid-connected photovoltaic systems
,”
IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.
2
,
2
10
(
2009
).

Supplementary Material

You do not currently have access to this content.