Cloud movement makes short-term forecasting of solar photovoltaic (PV) panel output challenging. A better PV forecast can realize value for both grid operators and commercial or industrial customers with solar assets. In this study, we build convolutional neural network (CNN) based models to forecast power output from PV panels 15 min into the future. Model inputs are the PV power output history and ground-based sky images for the past 15 min. The key challenge is ensuring that due importance is given to each type of input. We systematically explore 28 methods of “fusing” these heterogeneous inputs in our CNN. These methods of fusion (MoF) belong to 4 families. We also systematically explore the many hyperparameters related to model training and tuning. Limited resources preclude an exhaustive search. We apply a three-stage “funnel” approach instead, wherein we narrow our search to the most promising one of these 28 MoF. We find that a two-step autoregression-CNN MoF has the best performance followed closely by a “mix-in” MoF that performs feature expansion and reduction to give appropriate importance to the two types of inputs. The two-step autoregression-CNN model has a forecast skill (FS) of 17.1% relative to smart persistence on the test set comprising 20 complete days (9 sunny, FS = 22%; 11 cloudy, FS = 16.9%). This optimization results in the improvement of FS from 14.1% for a previously published nonoptimized “baseline” model, a CNN wherein the PV history was simply concatenated to the end of the image-sourced vector obtained after convolution, pooling, and flattening operations.

1.
Y.
Chu
,
B.
Urquhart
,
S. M. I.
Gohari
,
H. T. C.
Pedro
,
J.
Kleissl
, and
C. F. M.
Coimbra
, “
Short-term reforecasting of power output from a 48 MWe solar PV plant
,”
Sol. Energy
112
,
68
77
(
2015
).
2.
V. P. A.
Lonij
,
A. E.
Brooks
,
A. D.
Cronin
,
M.
Leuthold
, and
K.
Koch
, “
Intra-hour forecasts of solar power production using measurements from a network of irradiance sensors
,”
Sol. Energy
97
,
58
66
(
2013
).
3.
R.
Dambreville
,
P.
Blanc
,
J.
Chanussot
, and
D.
Boldo
, “
Very short term forecasting of the Global Horizontal Irradiance using a spatio-temporal autoregressive model
,”
Renewable Energy
72
,
291
300
(
2014
).
4.
J.
Zhang
,
R.
Verschae
,
S.
Nobuhara
, and
J.-F.
Lalonde
, “
Deep photovoltaic nowcasting
,”
Sol. Energy
176
,
267
276
(
2018
).
5.
M.
Rana
,
I.
Koprinska
, and
V. G.
Agelidis
, “
Univariate and multivariate methods for very short-term solar photovoltaic power forecasting
,”
Energy Convers. Manage.
121
,
380
390
(
2016
).
6.
G.
Reikard
, “
Predicting solar radiation at high resolutions: A comparison of time series forecasts
,”
Sol. Energy
83
(
3
),
342
349
(
2009
).
7.
A.
Moreno-Munoz
,
J. J. G.
de la Rosa
,
R.
Posadillo
, and
F.
Bellido
, “
Very short term forecasting of solar radiation
,” in
2008 33rd IEEE Photovoltaic Specialists Conference
(
2008
), pp.
1
5
.
8.
Y.
Sun
,
G.
Szucs
, and
A. R.
Brandt
, “
Solar PV output prediction from video streams using convolutional neural networks
,”
Energy Environ. Sci.
11
(
7
),
1811
1818
(
2018
).
9.
Y.
Sun
,
V.
Venugopal
, and
A. R.
Brandt
, “
Short-term solar power forecast with deep learning: Exploring optimal input and output configuration
,”
Sol. Energy
188
,
730
741
(
2019
).
10.
D.
Xu
,
D.
Anguelov
, and
A.
Jain
, “
PointFusion: Deep sensor fusion for 3D bounding box estimation
,” in
IEEE Conference on Computer Vision and Pattern Recognition
(
2018
), pp.
244
253
.
11.
E.
Ayrey
,
D. J.
Hayes
,
J. B.
Kilbride
, and
A. R.
Weiskittel
, “
Synthesizing disparate LiDAR and satellite datasets through deep learning to generate wall-to-wall forest inventories of New England
,”
580514
(
2019
).
12.
Y.
Zhou
and
K.
Hauser
, “
Incorporating side-channel information into convolutional neural networks for robotic tasks
,” in
2017 IEEE International Conference on Robotics and Automation (ICRA)
(
2017
), pp.
2177
2183
.
13.
Y.
Zhou
and
K.
Hauser
, “
6DOF grasp planning by optimizing a deep learning scoring function
,” in
Robotics: Science and Systems (RSS) Workshop on Revisiting Contact – Turning a Problem into a Solution
(
Springer
,
2017
)
14.
I.
Lenz
,
H.
Lee
, and
A.
Saxena
, “
Deep learning for detecting robotic grasps
,”
Int. J. Rob. Res.
34
(
4–5
),
705
724
(
2015
).
15.
J.
Ngiam
and
A. Y.
Ng
, “
Multimodal deep learning
,” in Proceedings of the 28th International Conference on Machine Learning, Bellevue, WA,
2011
.
16.
N.
Srivastava
, “
Multimodal learning with deep Boltzmann machines
,” in
Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
(
2013
), pp.
329
336
.
17.
D. P.
Kingma
and
J.
Ba
, “
Adam: A method for stochastic optimization
,” preprint arXiv:1412.6980 (
2014
).
18.
N.
Shrivastava
,
G.
Hinton
,
A.
Krizhevsky
,
I.
Sutskever
, and
R.
Salakhutdinov
, “
Dropout: A simple way to prevent neural networks from overfitting
,”
J. Mach. Learn. Res.
15
(
1
),
1929
1958
(
2014
).
19.
S.
Ioffe
and
C.
Szegedy
, “
Batch normalization: Accelerating deep network training by reducing internal covariate shift
,”
Proceedings of the 32nd International Conference on Machine Learning
, (PMLR, 2015), Vol. 37, pp.
448
456
.
20.
J. L.
Ba
,
J. R.
Kiros
, and
G. E.
Hinton
, “
Layer normalization
,” preprint arXiv:1607.06450.
21.
D.
Ulyanov
and
A.
Vedaldi
, “
Instance normalization: The missing ingredient for fast stylization
,” preprint arXiv:1607.08022.
22.
V.
Nair
and
G. E.
Hinton
, “
Rectified linear units improve restricted Boltzmann machines
,” in the Proceedings of the 27th International Conference on Machine Learning, Haifa, Israel, 2010).
23.
See http://cs231n.github.io/neural-networks-1/ for “CS-231N, Course Notes, Stanford University.”
24.
D.
Tran
,
L.
Bourdev
,
R.
Fergus
,
L.
Torresani
, and
M.
Paluri
, “
Learning spatiotemporal features with 3D convolutional networks
,” (IEEE, 2015). pp.
4489
4497
.
25.
T.
Hastie
,
R.
Tibshirani
,
J.
Gareth
, and
W.
Daniela
,
An Introduction to Statistical Learning with Applications in R
(
Springer
,
2017
).
26.
J.
Bergstra
and
Y.
Bengio
, “
Random search for hyper-parameter optimization
,”
J. Mach. Learn. Res.
13
,
281
305
(
2012
).
27.
F. A. C.
Viana
, “
A tutorial on Latin hypercube design of experiments
,”
Qual. Reliab. Eng. Int.
(
2016
).
28.
M.
Van der Merwe
,
Q.
Lu
,
B.
Sundaralingam
,
M.
Matak
, and
T.
Hermans
, “
Learning Continuous 3D Reconstructions for Geometrically Aware Grasping
,” eprint arXiv:1910.00983.
29.
W.
Zhang
and
K.
Hauser
, “
Single-image footstep prediction for versatile legged locomotion
,” in
2018 IEEE International Conference on Robotics and Automation (ICRA)
(
2018
), pp.
4407
4413
.
30.
Y.
Zhu
and
T.
Erez
, “
Reinforcement and imitation learning for diverse visuomotor skills
,” preprint arXiv:1802.09564.
31.
S.
Levine
,
C.
Finn
,
T.
Darrell
, and
P.
Abbeel
, “
End-to-end training of deep visuomotor policies
,”
J. Mach. Learn. Res.
17
,
1
40
(
2015
).
32.
S.
Koo
,
G.
Ficht
,
G. M.
García
,
D.
Pavlichenko
,
M.
Raak
, and
S.
Behnke
, “
Robolink feeder: Reconfigurable bin-picking and feeding with a lightweight cable-driven manipulator
,” in
2017 13th IEEE Conference on Automation Science and Engineering (CASE)
(
2017
), pp.
41
48
.
33.
A.
Yahya
,
A.
Li
,
M.
Kalakrishnan
,
Y.
Chebotar
, and
S.
Levine
, “
Collective robot reinforcement learning with distributed asynchronous guided policy search
,” in
2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
(
2017
), pp.
79
86
.
34.
J.
Mahler
,
J.
Liang
,
S.
Niyaz
,
M.
Laskey
,
R.
Doan
,
X.
Liu
,
J. A.
Ojea
, and
K.
Goldberg
, “
Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics
,” preprint arXiv:1703.09312.
35.
A.
Dosovitskiy
and
V.
Koltun
, “
Learning to act by predicting the future
,” International Conference on Learning Representations,
1
14
(
2017
).
36.
K.
Fang
,
Y.
Zhu
,
A.
Garg
,
A.
Kurenkov
,
V.
Mehta
,
L.
Fei-Fei
, and
S.
Savarese
, “
Learning task-oriented grasping for tool manipulation from simulated self-supervision
,” arXiv:1806.09266.
37.
R.
Calandra
,
A.
Owens
,
D.
Jayaraman
,
J.
Lin
,
W.
Yuan
,
J.
Malik
,
E.
Adelson
, and
S.
Levine
, “
More than a feeling: Learning to grasp and regrasp using vision and touch
,”
IEEE Robot. and Autom. Lett.
3
(
4
),
3300
3307
(
2018
).
38.
D.
Quillen
,
E.
Jang
,
O.
Nachum
,
C.
Finn
,
J.
Ibarz
, and
S.
Levine
, “
Deep reinforcement learning for vision-based robotic grasping: A simulated comparative evaluation of off-policy methods
,” in
2018 IEEE International Conference on Robotics and Automation (ICRA)
(
2018
), pp.
6284
6291
.
39.
K.
Fang
,
Y.
Bai
,
S.
Hinterstoisser
,
S.
Savarese
, and
M.
Kalakrishnan
, “
Multi-task domain adaptation for deep learning of instance grasping from simulation
,” in
2018 IEEE International Conference on Robotics and Automation (ICRA)
(
2018
), pp.
3516
3523
.
40.
S.
Levine
,
P.
Pastor
,
A.
Krizhevsky
,
J.
Ibarz
, and
D.
Quillen
, “
Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection
,”
Int. J. Rob. Res.
37
(
4-5
),
421
436
(
2018
).
41.
M.
Jaśkowski
 et al, “
Improved GQ-CNN: Deep learning model for planning robust grasps
,” arXiv:1802.05992.
42.
O.
Vinyals
,
A. S.
Vezhnevets
, and
D.
Silver
, “
StarCraft II: A new challenge for reinforcement learning
,” arXiv:1708.04782.
43.
F.
Sadeghi
,
A.
Toshev
, and
E.
Jang
, “
Sim2Real view invariant visual servoing by recurrent control
,” arXiv:1712.07642.
44.
T.
Elsken
,
J. H.
Metzen
, and
F.
Hutter
, “
Neural architecture search: A survey
,”
J. Mach. Learn. Res.
20
,
1
21
(
2019
).
45.
F.
Hutter
,
L.
Kotthoff
, and
J.
Vanschoren
,
Automated Machine Learning: Methods, Systems, Challenges
(
Springer
,
2019
).
46.
B.
Zoph
and
Q. V.
Le
, “
Neural architecture search with reinforcement learning
,” preprint arXiv:1611.01578 (
2016
).
47.
E.
Real
,
A.
Aggarwal
,
Y.
Huang
, and
Q. V.
Le
, “
Regularized evolution for image classifier architecture search
,”
Proc. AAAI Conf. Artif. Intell.
33
,
4780
4789
(
2019
).
48.
H.
Mendoza
,
A.
Klein
,
M.
Feurer
,
J. T.
Springenberg
, and
F.
Hutter
, “
Towards automatically-tuned neural networks
,” in Workshop on Automatic Machine Learning (
2016
), pp.
58
65
.
49.
H.
Liu
,
K.
Simonyan
, and
Y.
Yang
, “
DARTS: Differentiable architecture search
,” preprint arXiv:1806.09055 (
2018
).
50.
B.
Baker
,
O.
Gupta
,
R.
Raskar
, and
N.
Naik
, “
Accelerating neural architecture search using performance prediction
,” preprint arXiv:1705.10823 (
2017
).
51.
C.
Liu
,
B.
Zoph
,
M.
Neumann
,
J.
Shlens
,
W.
Huai
,
L.
Jia
,
L.
Fei-Fei
,
A.
Yuille
,
J.
Huang
, and
K.
Murphy
, “
Progressive neural architecture search
,” in
European Conference on Computer Vision
(
LNCS
,
2018
), Vol.
11205
, pp.
19
35
.
52.
M.
Suganuma
,
M.
Ozay
, and
T.
Okatani
, “
Exploiting the potential of standard convolutional autoencoders for image restoration by evolutionary search
,” in
35th International Conference on Machine Learning, ICML
(
2018
), Vol.
11
, pp.
7592
7601
.
53.
D. R.
So
,
C.
Liang
, and
Q. V.
Le
, “
The evolved transformer
,” arXiv:1901.11117.
54.
See https://edu.google.com/programs/credits/faqs/?modal_active=none#research-credits for “
Google Cloud Platform Research Credits Program
.”
55.
A. V.
Da Rosa
,
Fundamentals of Renewable Energy Processes
(
Academic Press
,
2012
).
56.
E.
Meyers
,
B.
Tabone
, and
M.
Kara
, “
Statistical clear sky fitting algorithm
,” in
45th IEEE Photovoltaic Specialist Conference (PVSC)
(
2018
), pp.
1
6
.
You do not currently have access to this content.