Testing Artificial Intelligence: from Games to Industrial Apps

Publications on developments in machine learning usually describe not only operating processes of a neural network or an algorithm but also the basis of AI training.

Generally, publicly available datasets recognized by the community are used for training purposes. However, it would be interesting to assess how effectively a system can be trained to the required quality level in another target system that a reader, for example, would like to develop.

By this, I refer to industrial developments with a specific set of classified objects. Many developers realized that a simple replication of the LeNet5 neural network that showed great results in a large-scale database of images known as ImageNet normally didn’t provide good quality with regard to other data. Video scene recognition applying the RNN and GAN technologies proved this fact.

The entry threshold in the machine learning technologies continues to increase. For example, the Reinforcement Learning technology requires building a sophisticated environment to train and test the developed system. It should reproduce the external environment as well as its reaction to the actions of the test system.

To train autopilots, large video sets and sensor data flows from real vehicles (for example, KITTI) have been created for this purpose. However, such data can’t be applied as widely as desired because it doesn’t support feedback mechanisms.

To train systems, artificial external environment is used as it is applicable for testing AI developments and, thus, it can be an effective proving ground.

Today the use of testing grounds in the form of special programming environments is a promising direction for AI training in real time. It can be applied in video analytics solutions, drone and autopilots management systems, etc. These testing grounds allow simulating the external environment as well as reaction of the outside world to the actions of the system.

Such simulation systems have been known for a long time and are usually applied for training pilots, military officers, traffic controllers as well as for entertainment (computer games).

A few days ago Microsoft shared an open source system for training drones – AirSim (https://github.com/Microsoft/AirSim).

AirSim is a simulator for drones (and soon other vehicles) built on Unreal Engine. It is open-source, cross platform and supports hardware-in-loop with popular flight controllers such as Pixhawk for physically and visually realistic simulations. It is developed as an Unreal plugin that can simply be dropped in to any Unreal environment you want. The AirSim exposes easy to use APIs to retrieve data from the drones that include ground truth, sensor data as well as various images. It also exposes APIs to control the drones in a platform independent way.

This allows you to use your code to control different drone platforms, for example, Pixhawk or DJI Matrice, without making changes as well as without having to learn internal protocols details.

For a wider range of tasks it is possible to use a set of simulators developed by Google DeepMind, Alphabet Inc.’s artificial intelligence division. DeepMind Lab is a 3D learning environment based on id Software’s Quake III Arena via ioquake3 and other open source software.

DeepMind Lab provides a suite of challenging 3D navigation and puzzle-solving tasks for learning agents. Its primary purpose is to act as a testbed for research in artificial intelligence, especially deep reinforcement learning.

DeepMind code (github.com/deepmind/lab) can be easily used by software engineers for training and testing their developments.

Elon Musk’s OpenAI is another testing ground for AI developers. The company provided open access to OpenAI Gym, a toolkit for developing and comparing reinforcement learning (RL) algorithms. It consists of a growing suite of environments (from simulated robots to Atari games), and a site for comparing and reproducing results (github.com/openai/gym).

OpenAI Gym is compatible with algorithms written in any framework, such as Tensorflow and Theano. The environments are written in Python but company says that they soon will make them easy to use from any language. OpenAI Gym is a tool to accelerate company own RL research but it will be just as useful for the broader community.

So, what should AI developers do at the initial stage of development? They should have an idea how a development process will be accepted by customers. It is hardly possible that someone will put the developed AI to operation in the natural environment. We will have to pay the price if any mistakes occur. Therefore, it is a good idea to have a testing ground for demonstration of AI operation and evaluation of possible mistakes.

It is also important to have understanding of the training process of AI. As a rule, there may also occur problems with simulating all possible situations and evaluating decisions taken by AI. Therefore, it is better to use a special testing ground.

The most effective way to build it is program implementation and search of ready simulation environments. This should be done already at the stage of the specification study. If it is not possible to find ready environments for tasks from the application field, it is necessary to select a suitable engine for independent development of the testing ground.

The experience of the leading companies shows that it is common practice to use well-known engines built for computer games. Therefore, having a computer games specialist in AI project teams can definitely become an advantage.