Fully neuromorphic vision and control for autonomous drone flight

Our work on fully neuromorphic vision and control for autonomous drone flight has been published in the Science Robotics May 2024 issue.

In this work, we have developed a drone that flies autonomously using neuromorphic image processing and control based on the workings of animal brains.  Animal brains use less data and energy compared to current deep neural networks running on GPUs (graphics chips). Neuromorphic processors are therefore very suitable for small drones because they don’t need heavy and large hardware and batteries. The results are extraordinary: during flight the drone’s deep neural network  processes data up to 64 times faster and consumes three times less energy than when running on a GPU. Further developments of this technology may enable the leap for drones to become as small, agile, and smart as flying insects or birds.

Learning from animal brains: spiking neural networks

Artificial intelligence holds great potential to provide autonomous robots with the intelligence needed for real-world applications. However, current AI relies on deep neural networks that require substantial computing power. The processors made for running deep neural networks (Graphics Processing Units, GPUs) consume a substantial amount of energy. Especially for small robots like flying drones this is a problem, since they can only carry very limited resources in terms of sensing and computing.

Animal brains process information in a way that is very different from the neural networks running on GPUs. Biological neurons process information asynchronously, and mostly communicate via electrical pulses called spikes. Since sending such spikes costs energy, the brain minimizes spiking, leading to sparse processing.

Inspired by these properties of animal brains, scientists and tech companies are developing new, neuromorphic processors. These new processors allow to run spiking neural networks and promise to be much faster and more energy efficient.

Whereas digital spiking neurons only need to add integers, standard neurons have to multiply and add floating point numbers. This makes spiking neural networks quicker and more energy efficient. This energy efficiency is further boosted if neuromorphic processors are used in combination with neuromorphic sensors, like neuromorphic cameras. Such cameras do not make images at a fixed time interval. Instead, each pixel only sends a signal when it becomes brighter or darker. The advantages of such cameras are that they can perceive motion much more quickly, are more energy efficient, and function well both in dark and bright environments. Moreover, the signals from neuromorphic cameras can feed directly into spiking neural networks running on neuromorphic processors. Together, they can form a huge enabler for autonomous robots, especially small, agile robots like flying drones.

First neuromorphic vision and control of a flying drone

In the spirit of utilizing biological principles for efficient drone flight, we demonstrate for the first time a drone that uses neuromorphic vision and control for autonomous flight. Specifically, we developed a spiking neural network that processes the signals from a neuromorphic camera and outputs control commands that determine the drone’s pose and thrust. We deployed this network on a neuromorphic processor, Intel’s Loihi neuromorphic research chip, on board of a drone. Thanks to the network, the drone can perceive and control its own motion in all directions.

“We faced many challenges,” says Federico Paredes-Vallés, one of the researchers that worked on the study, “but the hardest one was to imagine how we could train a spiking neural network

To make the network both train sufficiently fast and function well on the real robot, we designed a network consisting of two modules. The first module learns to visually perceive motion from the signals of a moving neuromorphic camera. It does so completely by itself, in a self-supervised way, based only on the data from the camera. This is similar to how also animals learn to perceive the world by themselves. The second module learns to map the estimated motion to control commands, in a simulator. This learning relied on an artificial evolution in simulation, in which networks that were better in controlling the drone had a higher chance of producing offspring. Over the generations of the artificial evolution, the spiking neural networks got increasingly good at control, and were finally able to fly in any direction at different speeds.

With its neuromorphic vision and control, the drone is able to fly at different speeds under varying light conditions, from dark to bright. It can even fly with flickering lights, which make the pixels in the neuromorphic camera send great numbers of signals to the network that are unrelated to motion.

Open-access paper
Datasets
Code

@article{<LineBreak>doi:10.1126/scirobotics.adi0591,

title = {Fully neuromorphic vision and control for autonomous drone flight},

author = {F. Paredes-Vallés and J. J. Hagenaars and J. Dupeyroux and S. Stroobants and Y. Xu and G. C. H. E. Croon},

url = {https://www.science.org/doi/abs/10.1126/scirobotics.adi0591},

doi = {10.1126/scirobotics.adi0591},

year  = {2024},

date = {2024-01-01},

urldate = {2024-01-01},

journal = {Science Robotics},

volume = {9},

number = {90},

pages = {eadi0591},

abstract = {Biological sensing and processing is asynchronous and sparse, leading to low-latency and energy-efficient perception and action. In robotics, neuromorphic hardware for event-based vision and spiking neural networks promises to exhibit similar characteristics. However, robotic implementations have been limited to basic tasks with low-dimensional sensory inputs and motor actions because of the restricted network size in current embedded neuromorphic processors and the difficulties of training spiking neural networks. Here, we present a fully neuromorphic vision-to-control pipeline for controlling a flying drone. Specifically, we trained a spiking neural network that accepts raw event-based camera data and outputs low-level control actions for performing autonomous vision-based flight. The vision part of the network, consisting of five layers and 28,800 neurons, maps incoming raw events to ego-motion estimates and was trained with self-supervised learning on real event data. The control part consists of a single decoding layer and was learned with an evolutionary algorithm in a drone simulator. Robotic experiments show a successful sim-to-real transfer of the fully learned neuromorphic pipeline. The drone could accurately control its ego-motion, allowing for hovering, landing, and maneuvering sideways—even while yawing at the same time. The neuromorphic pipeline runs on board on Intel’s Loihi neuromorphic processor with an execution frequency of 200 hertz, consuming 0.94 watt of idle power and a mere additional 7 to 12 milliwatts when running the network. These results illustrate the potential of neuromorphic sensing and processing for enabling insect-sized intelligent robots. A fully neuromorphic vision-to-control pipeline enables fast and energy-efficient ego-motion control of a flying drone. Despite the ability of visual processing enabled by artificial neural networks, the associated hardware and large power consumption limit deployment on small flying drones. Neuromorphic hardware offers a promising alternative, but the accompanying spiking neural networks are difficult to train, and the current hardware only supports a limited number of neurons. Paredes-Vallés et al. now present a neuromorphic pipeline to control drone flight. They trained a five-layer spiking neural network to process the raw inputs from an event camera. The network first estimated ego-motion and subsequently determined low-level control commands. Real-world experiments demonstrated that the drone could control its ego-motion to land, hover, and maneuver sideways, with minimal power consumption. —Melisa Yashinski},

keywords = {},

pubstate = {published},

tppubtype = {article}

}