Drone detection is an important yet challenging task in the context of object detection. The development of robust and reliable drone detection systems requires large amounts of labeled data, especially when using deep learning (DL) models. Unfortunately, acquiring real data is expensive, time-consuming, and often limited by external factors. This makes synthetic data a promising approach to addressing data deficiencies. In this paper, we present a data generation pipeline based on Unreal Engine 4.25 and Microsoft AirSim, designed to create synthetic labeled data for drone detection using three-dimensional environments. As part of an ablation study, we investigate the potential use of synthetic data in drone detection by analyzing different training strategies, influencing factors, and data generation parameters, specifically related to the visual appearance of a drone.

1.
F.
Mahdavi
and
R.
Rajabi
, “
Drone Detection Using Convolutional Neural Networks
,”
2020 6th Iranian Conference on Signal Processing and Intelligent Systems (ICSPIS) (2020
), .
2.
A.
Bernardini
,
F.
Mangiatordi
,
E.
Pallotti
, and
L.
Capodiferro
, “
Drone detection by acoustic signature identification
,”
electronic imaging
2017
,
60
64
(
2017
).
3.
U.
Seidaliyeva
,
D.
Akhmetov
,
L.
Ilipbayeva
, and
E. T.
Matson
, “
Real-Time and Accurate Drone Detection in a Video with a Static Back-ground
,”
Sensors
20
(
2020
), .
4.
P.
Nguyen
,
H.
Truong
,
M.
Ravindranathan
,
A.
Nguyen
,
R. O.
Han
, and
T. N.
Vu
, “
Cost-Effective and Passive RF-Based Drone Presence Detection and Characterization
,”
GetMobile Mob. Comput. Commun.
21
,
30
34
(
2018
).
5.
B.
Nuss
,
L.
Sit
,
M.
Fennel
,
J.
Mayer
,
T.
Mahler
, and
T.
Zwick
, “
MIMO OFDM radar system for drone detection
,” in
2017 18th International Radar Symposium (IRS)
(
2017
) pp.
1
9
.
6.
F.
Svanström
,
C.
Englund
, and
F.
Alonso-Fernandez
, “
Real-Time Drone Detection and Tracking With Visible, Thermal and Acoustic Sensors
,” in
2020 25th International Conference on Pattern Recognition (ICPR)
(
2021
) pp.
7265
7272
.
7.
F.-L.
Chiper
,
A.
Martian
,
C.
Vladeanu
,
I.
Marghescu
,
R.
Craciunescu
, and
O.
Fratu
, “
Drone Detection and Defense Systems: Survey and a Software-Defined Radio-Based Solution
,”
Sensors
22
(
2022
), .
8.
F.
Yang
,
F.
Xu
,
F.
Fioranelli
,
J. L.
Kernec
,
S.
Chang
, and
T.
Long
, “
Practical Investigation of a MIMO radar system capabilities for small drones detection
,”
IET Radar, Sonar & Navigation
15
,
760
774
(
2021
).
9.
S.
Grác
,
P.
Beňo
,
F.
Duchoň
,
M.
Dekan
, and
M.
Tölgyessy
, “
Automated Detection of Multi-Rotor UAVs Using a Machine-Learning Approach
,”
Applied System Innovation
3
(
2020
), .
10.
T.
Kashiyama
,
H.
Sobue
, and
Y.
Sekimoto
, “
Sky Monitoring System for Flying Object Detection Using 4K Resolution Camera
,”
Sensors
20
(
2020
), .
11.
S.
Srigrarom
,
S. M.
Lee
,
M.
Lee
,
F.
Shaohui
, and
P.
Ratsamee
, “
An Integrated Vision-based Detection-tracking-estimation System for Dynamic Localization of Small Aerial Vehicles
,” in
2020 5th International Conference on Control and Robotics Engineering (ICCRE)
(
2020
) pp.
152
158
.
12.
A.
Rozantsev
,
V.
Lepetit
, and
P.
Fua
, “
Detecting Flying Objects Using a Single Moving Camera
,”
IEEE Transactions on Pattern Analysis and Machine Intelligence
39
,
879
892
(
2017
).
13.
J.
Peng
,
C.
Zheng
,
P.
Lv
,
T.
Cui
,
Y.
Cheng
, and
S.
Lingyu
, “
Using images rendered by PBRT to train faster R-CNN for UAV detection
,” (
2018
).
14.
R.
Girshick
,
J.
Donahue
,
T.
Darrell
, and
J.
Malik
, “
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
,” in
2014 IEEE Conference on Computer Vision and Pattern Recognition
(
2014
) pp.
580
587
.
15.
R.
Girshick
, “
Fast R-CNN
,” in
Proceedings of the IEEE International Conference on Computer Vision (ICCV)
(
2015
).
16.
S.
Ren
,
K.
He
,
R.
Girshick
, and
J.
Sun
, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,” in
Advances in Neural Information Processing Systems
, Vol.
28
, edited by
C.
Cortes
,
N.
Lawrence
,
D.
Lee
,
M.
Sugiyama
, and
R.
Garnett
(
Curran Associates, Inc
.,
2015
).
17.
J.
Redmon
,
S.
Divvala
,
R.
Girshick
, and
A.
Farhadi
, “
You Only Look Once: Unified, Real-Time Object Detection
,” in
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(
2016
) pp.
779
788
.
18.
W.
Liu
,
D.
Anguelov
,
D.
Erhan
,
C.
Szegedy
,
S.
Reed
,
C.-Y.
Fu
, and
A. C.
Berg
, “SSD: Single Shot MultiBox Detector,” in
Computer Vision – ECCV 2016
, edited by
B.
Leibe
,
J.
Matas
,
N.
Sebe
, and
M.
Welling
(
Springer International Publishing
,
Cham
,
2016
) pp.
21
37
.
19.
S. I.
Nikolenko
,
Synthetic Data for Deep Learning
(
Springer Optimization and Its Applications
,
2021
).
20.
K. E.
Emam
,
Accelerating AI with Synthetic Data
(
O'Reilly Media, Inc
.,
2020
).
21.
A.
Tsirikoglou
,
G.
Eilertsen
, and
J.
Unger
, “
A Survey of Image Synthesis Methods for Visual Machine Learning
,”
Computer Graphics Forum
39
(
2020
).
22.
F. C.
Akyon
,
O.
Eryuksel
,
K. A.
Özfuttu
, and
S. O.
Altinuc
, “
Track Boosting and Synthetic Data Aided Drone Detection
,” in
2021 17th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)
(
2021
) pp.
1
5
.
23.
D.
Marez
,
S.
Borden
, and
L.
Nans
, “
UAV detection with a dataset augmented by domain randomization
,” in
Defense+Commercial Sensing
(
2020
).
24.
K. A.
Özfuttu
,
Generating Synthetic Data with Game Engines dor Deep Learning Applications
, Master's thesis,
Hacettepe University
(
2022
).
25.
J.
Tremblay
,
A.
Prakash
,
D.
Acuna
,
M.
Brophy
,
V.
Jampani
,
C.
Anil
,
T.
To
,
E.
Cameracci
,
S.
Boochoon
, and
S.
Birchfield
, “
Training Deep Networks With Synthetic Data: Bridging the Reality Gap by Domain Randomization
,” in
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops
(
2018
).
26.
J.
Tremblay
,
T.
To
, and
S.
Birchfield
, “
Falling Things: A Synthetic Dataset for 3D Object Detection and Pose Estimation
,” in
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops
(
2018
).
27.
M.
Johnson-Roberson
,
C.
Barto
,
R.
Mehta
,
S. N.
Sridhar
,
K.
Rosaen
, and
R.
Vasudevan
, “
Driving in the Matrix: Can Virtual Worlds Replace Human-Generated Annotations for Real World Tasks?
” in
2017 IEEE International Conference on Robotics and Automation (ICRA)
(
IEEE Press
,
2017
) p.
746
753
.
28.
S. R.
Richter
,
V.
Vineet
,
S.
Roth
, and
V.
Koltun
, “Playing for Data: Ground Truth from Computer Games,” in
European Conference on Computer Vision (ECCV), LNCS
, Vol.
9906
, edited by
B.
Leibe
,
J.
Matas
,
N.
Sebe
, and
M.
Welling
(
Springer International Publishing
,
2016
) pp.
102
118
.
29.
E. A.
Lopez-Rojas
and
S.
Axelsson
, “
A review of computer simulation for fraud detection research in financial datasets
,” in
2016 Future Technologies Conference (FTC)
(
2016
) pp.
932
935
.
30.
M. F.
Zeager
,
A.
Sridhar
,
N.
Fogal
,
S.
Adams
,
D. E.
Brown
, and
P. A.
Beling
, “
Adversarial learning in credit card fraud detection
,” in
2017 Systems and Information Engineering Design Symposium (SIEDS)
(
2017
) pp.
112
116
.
31.
Y. R. Allan
Tucker
,
Zhenchen
Wang
and
P.
Myles
, “
Generating high-fidelity synthetic patient data for assessing machine learning healthcare software
,”
npj Digital Medicine
3
(
2020
), .
32.
R. J.
Chen
,
M. Y.
Lu
,
T. Y.
Chen
,
D. F. K.
Williamson
, and
F.
Mahmood
, “
Synthetic data in machine learning for medicine and healthcare
,”
Nature Biomedical Engineering
5
,
493
497
(
2021
).
33.
C.
Baur
,
S.
Albarqouni
, and
N.
Navab
, “Generating Highly Realistic Images of Skin Lesions with GANs,” in
OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis
, edited by
D.
Stoyanov
,
Z.
Taylor
,
D.
Sarikaya
,
J.
McLeod
,
M. A. González
Ballester
,
N. C.
Codella
,
A.
Martel
,
L.
Maier-Hein
,
A.
Malpani
,
M. A.
Zenati
,
S.
De Ribaupierre
,
L.
Xiongbiao
,
T.
Collins
,
T.
Reichl
,
K.
Drechsler
,
M.
Erdt
,
M. G.
Linguraru
,
C. Oyarzun
Laura
,
R.
Shekhar
,
S.
Wesarg
,
M. E.
Celebi
,
K.
Dana
, and
A.
Halpern
(
Springer International Publishing
,
Cham
,
2018
) pp.
260
267
.
34.
H.-C.
Shin
,
N. A.
Tenenholtz
,
J. K.
Rogers
,
C. G.
Schwarz
,
M. L.
Senjem
,
J. L.
Gunter
,
K. P.
Andriole
, and
M.
Michalski
, “Medical Image Synthesis for Data Augmentation and Anonymization Using Generative Adversarial Networks,” in
Simulation and Synthesis in Medical Imaging
, edited by
A.
Gooya
,
O.
Goksel
,
I.
Oguz
, and
N.
Burgos
(
Springer International Publishing
,
Cham
,
2018
) pp.
1
11
.
35.
H.
Su
,
C. R.
Qi
,
Y.
Li
, and
L. J.
Guibas
, “
Render for CNN: Viewpoint Estimation in Images Using CNNs Trained With Rendered 3D Model Views
,” in
Proceedings of the IEEE International Conference on Computer Vision (ICCV)
(
2015
).
36.
A.
Gaidon
,
Q.
Wang
,
Y.
Cabon
, and
E.
Vig
, “
Virtual Worlds as Proxy for Multi-Object Tracking Analysis
,” in
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(
2016
).
37.
G.
Varol
,
J.
Romero
,
X.
Martin
,
N.
Mahmood
,
M. J.
Black
,
I.
Laptev
, and
C.
Schmid
, “
Learning from Synthetic Humans
,” in
Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017
(
IEEE
,
Piscataway, NJ, USA
,
2017
) pp.
4627
4635
.
38.
S. R.
Richter
,
Z.
Hayder
, and
V.
Koltun
, “
Playing for Benchmarks
,” in
2017 IEEE International Conference on Computer Vision (ICCV)
(
2017
) pp.
2232
2241
.
39.
B.
Osiński
,
A.
Jakubowski
,
P.
Zięcina
,
P.
Miłoś
,
C.
Galias
,
S.
Homoceanu
, and
H.
Michalewski
, “
Simulation-Based Reinforcement Learning for Real-World Autonomous Driving
,” in
2020 IEEE International Conference on Robotics and Automation (ICRA)
(
2020
) pp.
6411
6418
.
40.
S.
Minhas
,
Z.
Khanam
,
S.
Ehsan
,
K.
McDonald-Maier
, and
A.
Hernández-Sabaté
, “
Weather Classification by Utilizing Synthetic Data
,”
Sensors
22
(
2022
), .
41.
B.
Hurl
,
K.
Czarnecki
, and
S. L.
Waslander
, “
Precise Synthetic Image and LiDAR (PreSIL) Dataset for Autonomous Vehicle Perception
,”
2019 IEEE Intelligent Vehicles Symposium (IV
),
2522
2529
(
2019
).
42.
A.
Handa
,
V.
Pătrăucean
,
S.
Stent
, and
R.
Cipolla
, “
SceneNet: An annotated model generator for indoor scene understanding
,” in
2016 IEEE International Conference on Robotics and Automation (ICRA)
(
2016
) pp.
5737
5743
.
43.
A.
Handa
,
V.
Patraucean
,
V.
Badrinarayanan
,
S.
Stent
, and
R.
Cipolla
, “
Understanding Real World Indoor Scenes With Synthetic Data
,” in
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(
2016
).
44.
S. Baressi
Šegota
,
N.
And¯elić
,
M.
Šercer
, and
H.
Meštrić
, “
Dynamics Modeling of Industrial Robotic Manipulators: A Machine Learning Approach Based on Synthetic Data
,”
Mathematics
10
(
2022
), .
45.
P.
Martinez-Gonzalez
,
S.
Oprea
,
A.
Garcia-Garcia
,
A.
Jover-Alvarez
,
S.
Orts-Escolano
, and
J.
Garcia-Rodriguez
, “
UnrealROX: an extremely photorealistic virtual reality environment for robotics simulations and synthetic data generation
,”
Virtual Reality
24
,
271
288
(
2020
).
46.
J. C.
Martínez-Franco
and
D.
Álvarez-Martínez
, “Machine Vision for Collaborative Robotics Using Synthetic Data-Driven Learning,” in
Service Oriented, Holonic and Multi-Agent Manufacturing Systems for Industry of the Future
, edited by
D.
Trentesaux
,
T.
Borangiu
,
P.
Leitão
,
J.-F.
Jimenez
, and
J. R.
Montoya-Torres
(
Springer International Publishing
,
2021
) pp.
69
81
.
47.
G.
Curnis
,
S.
Fontana
, and
D. G.
Sorrenti
, “
GTASynth: 3D synthetic data of outdoor non-urban environments
.”
Data in Brief
43
,
108412
(
2022
).
48.
A.
Prakash
,
S.
Boochoon
,
M.
Brophy
,
D.
Acuna
,
E.
Cameracci
,
G.
State
,
O.
Shapira
, and
S.
Birchfield
, “
Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data
,”
2019 International Conference on Robotics and Automation (ICRA
),
7249
7255
(
2019
).
49.
H.
Huang
,
z.
li
,
R.
He
,
Z.
Sun
, and
T.
Tan
, “IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesis,” in
Advances in Neural Information Processing Systems
, Vol.
31
, edited by
S.
Bengio
,
H.
Wallach
,
H.
Larochelle
,
K.
Grauman
,
N.
Cesa-Bianchi
, and
R.
Garnett
(
Curran Associates, Inc
.,
2018
).
50.
T.
Pollok
,
L.
Junglas
,
B.
Ruf
, and
A.
Schumann
, “UnrealGT: Using Unreal Engine to Generate Ground Truth Datasets,” in
Advances in Visual Computing
, edited by
G.
Bebis
,
R.
Boyle
,
B.
Parvin
,
D.
Koracin
,
D.
Ushizima
,
S.
Chai
,
S.
Sueda
,
X.
Lin
,
A.
Lu
,
D.
Thalmann
,
C.
Wang
, and
P.
Xu
(
Springer International Publishing
,
Cham
,
2019
) pp.
670
682
.
51.
S.
Borkman
,
A.
Crespi
,
S.
Dhakad
,
S.
Ganguly
,
J.
Hogins
,
Y. C.
Jhang
,
M.
Kamalzadeh
,
B.
Li
,
S.
Leal
,
P.
Parisi
,
C.
Romero
,
W.
Smith
,
A.
Thaman
,
S.
Warren
, and
N.
Yadav
, “
Unity Perception: Generate Synthetic Data for Computer Vision
,”
ArXiv
abs/2107.04259 (
2021
).
52.
T.
Hodaň
,
V.
Vineet
,
R.
Gal
,
E.
Shalev
,
J.
Hanzelka
,
T.
Connell
,
P.
Urbina
,
S. N.
Sinha
, and
B.
Guenter
, “
Photorealistic Image Synthesis for Object Instance Detection
,” in
2019 IEEE International Conference on Image Processing (ICIP)
(
2019
) pp.
66
70
.
53.
B.
Kiefer
,
D.
Ott
, and
A.
Zell
, “
Leveraging Synthetic Data in Object Detection on Unmanned Aerial Vehicles
,”
CoRR
abs/2112.12252 (
2021
).
54.
L.
Zherdeva
,
D.
Zherdev
, and
A.
Nikonorov
, “
Prediction of human behavior with synthetic data
,” in
2021 International Conference on Information Technology and Nanotechnology (ITNT)
(
2021
) pp.
1
6
.
55.
M.
Lyssenko
,
C.
Gladisch
,
C.
Heinzemann
,
M.
Woehrle
, and
R.
Triebel
, “
Instance Segmentation in CARLA: Methodology and Analysis for Pedestrian-oriented Synthetic Data Generation in Crowded Scenes
,” in
2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)
(
2021
) pp.
988
996
.
56.
Cityscapes Team
, “
The cityscapes dataset
,” https://www.cityscapes-dataset.com/ (Last visit: 10.07.2022).
57.
G. J.
Brostow
,
J.
Shotton
,
J.
Fauqueur
, and
R.
Cipolla
, “
Segmentation and Recognition Using Structure from Motion Point Clouds
,” in
ECCV
(
1
) (
2008
) pp.
44
57
.
58.
G. J.
Brostow
,
J.
Fauqueur
, and
R.
Cipolla
, “
Semantic Object Classes in Video: A High-Definition Ground Truth Database
,”
Pattern Recogni-tion Letters
30
,
88
97
(
2009
).
59.
Epic
Games
, “
Unreal Engine
,” https://www.unrealengine.com/en-US/ (Last visit: 21.06.2022).
60.
Unity Technologies
, “
Unity
,” https://unity.com/de (Last visit: 11.07.2022).
61.
CARLA Team
, “
CARLA - Open-source simulator for autonomous driving research.
https://carla.org/ (Last visit: 11.07.2022).
62.
Microsoft Research
, “
Welcome to AirSim
,” https://microsoft.github.io/AirSim/ (Last visit: 21.06.2022).
63.
Y.
Chen
,
P.
Aggarwal
,
J.
Choi
, and
C.-C. J.
Kuo
, “
A deep learning approach to drone monitoring
,” in
2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) (2017)
pp.
686
691
.
64.
S. R.
Yellapantula
,
Synthesizing Realistic Data for Vision Based Drone-to-Drone Detection
, Master's thesis,
Virginia Polytechnic Institute and State University
(
2019
).
65.
A.
Rozantsev
,
V.
Lepetit
, and
P.
Fua
, “
On rendering synthetic images for training an object detector
,”
Computer Vision and Image Under-standing
137
,
24
37
(
2015
).
66.
Urban City Environment, Unreal Engine Marketplace
, https://www.unrealengine.com/marketplace/en-US/product/urban-city.
67.
City Park Environment Collection, Unreal Engine Marketplace
, https://www.unrealengine.com/marketplace/en-US/product/city-park-environment-collection.
68.
Quadcopter pack - drones, Unreal Engine Marketplace
, https://www.unrealengine.com/marketplace/en-US/product/quadcopter-pack-drones.
69.
Ultralytics
, “
YOLOv5: The friendliest AI architecture you'll ever use
,” https://ultralytics.com/yolov5 (Last visit: 01.07.2022).
70.
A.
Bochkovskiy
,
C.
Wang
, and
H. M.
Liao
, “
YOLOv4: Optimal Speed and Accuracy of Object Detection
,”
CoRR
abs/2004.10934 (
2020
), 2004.10934.
71.
T.-Y.
Lin
,
M.
Maire
,
S. J.
Belongie
,
J.
Hays
,
P.
Perona
,
D.
Ramanan
,
P.
Dollár
, and
C. L.
Zitnick
, “
Microsoft COCO: Common Objects in Context
,” in
ECCV
(
2014
).
72.
R.
Padilla
,
S. L.
Netto
, and
E. A. B.
da Silva
, “
A Survey on Performance Metrics for Object-Detection Algorithms
,” in
2020 International Conference on Systems, Signals and Image Processing (IWSSIP)
(
2020
) pp.
237
242
.
73.
A.
Krizhevsky
, “
The CIFAR-10 dataset
,” https://www.cs.toronto.edu/kriz/cifar.html (Last visit: 10.07.2022).
74.
I.
Zafar
,
G.
Tzanidou
,
R.
Burton
,
N.
Patel
, and
L.
Araujo
,
Hands-On Convolutional Neural Networks with TensorFlow
(
Packt Publishing
,
2018
).
This content is only available via PDF.
You do not currently have access to this content.