Development and analysis of computer vision solution concepts using game engine based simulations in visually realistic environments

Abstract

Robotic platforms such as drones, (ROVs), and semi-autonomous vehicles are becoming more prevalent. The need for testing and analyzing techniques for algorithms and experimentation increases accordingly. For the goal of fully autonomous vehicles data gathering, annotating is a major obstacle. It requires resources for collecting data such as vehicles, cameras and sensors in addition to the manual effort of annotating the data for some of the applications. For that reason, most of the publicly available datasets and research work is carried out in common domains such as urban city space. A possible solution to this problem can be using visually realistic simulation. In this research, we will analyze the contribution of game engines as a close to realistic simulation medium. Our purposed methods involve using an existing game engine with existing assets originally developed for gaming to create a simulated visual environment for the purpose of conducting scientific experiments taking advantage of the visualisation power of the game engine. The current state of the art and latest technologies in game engines and computer graphics will also be considered. Using our approach will give researchers in the computer vision domain a valuable tool for testing their algorithms in high fidelity manner. New research tools will be presented such as frameworks for conducting experiments inside a game engine. Our tools will also provide capabilities for manipulating the simulated scene such as variable wind strength and lighting conditions. Two previously highly challenging unexplored simulation domains will be used as the main environments for conducting our experiments. One will be the drone simulation domain in a natural environment and the second will be the underwater ROV simulation domain. Both domains will be thoroughly explored. New visual simulation models and assets, as well as dynamic vehicle models that were developed for this research, will be presented as part of the simulated environments and frameworks. Existing computer vision algorithms and new algorithms developed for the purpose of this research will be tested in our simulation and in a real-world environment for comparison. The simulated experiments will be “end to end” experiments including fully simulated hardware, physics, inertial sensors and cameras. The experiments done in the simulation will be used to develop and fine-tune new methods for computer vision based navigation in those challenging domains. By showing the similar performance of a successful vehicle manoeuvre in simulation compared to a real environment using a newly developed computer vision algorithm we will validate the usefulness of game engines based simulation for computer vision research. The results show that our approach provides computer vision and autonomous vehicle researchers a new valuable tool for testing and analyzing algorithms by utilizing the visualization power of state of the art game engines in new highly challenging environments.