Learning process as a Solution to Fundamental Optical-flow Problems

Optical flow for small flying robots

Flying insects heavily rely on optical flow for visual navigation and flight control. Roboticists have endowed small flying robots with optical flow control as well since it requires just a tiny vision sensor. However, when using optical flow, the robots run into problems that insects appear to have overcome.

Fundamental problems of optical flow

Today Nature Machine Intelligence has published our article (featuring it on the cover – [link]), in which we propose a solution to two fundamental problems of optical flow. The first problem is that optical flow only provides mixed information on distances and velocities. This means that using it directly for control leads to oscillations when getting closer to obstacles. The second problem is that optical flow provides very little information on obstacles in the direction of motion. This means that the hardest obstacles to detect are the ones the robot is actually going to collide with!

Learning process as a solution

We tackled these problems with the help of a learning process. Specifically, the robot exploits self-induced oscillations to learn what the objects in its environment look like at different distances. In this way, it can for example learn how fine the texture of grass is when looking at it from different heights during landing. For obstacle avoidance, it can learn how thick tree barks are at different distances when navigating in a forest.

After learning, drones can adapt their control to the perceived distance. This results in faster and smoother optical flow landings. It also allows them to detect obstacles in the direction of flight. This results in a faster and safer flight in cluttered environments.

Hypothesis on insect intelligence

The findings are not only relevant to robotics, but also provide a new hypothesis for insect intelligence. Insect flight control for tasks such as landing or obstacle avoidance is typically described in terms of pre-wired optical flow control laws. Our findings suggest that flying insects may learn the visual appearance of their environment to improve their flight and navigation skills over their lifetime.

The article is featured on the cover of Nature Machine Intelligence. Reprinted by permission from Springer Nature, “Enhancing optical-flow-based control by learning visual appearance cues for flying robots”, by G.C.H.E. de Croon, C. De Wagter, and T. Seidl, Nature Machine Intelligence, 3 (1), 2021.

• Press release: https://www.tudelft.nl/en/2021/tu-delft/appreciating-a-flowers-texture-color-and-shape-leads-to-better-drone-landings
• Article: https://www.nature.com/articles/s42256-020-00279-7
• Press media folder: https://surfdrive.surf.nl/files/index.php/s/qxTF9sFW0xEcetd
• Youtube: https://youtu.be/A50Wl311rmU about the solution to fundamental optical-flow problems

Publications

@article{nature_ai_optical_flow,

title = {Enhancing optical-flow-based control by learning  visual appearance cues for flying robots},

author = {G.C.H.E. de Croon, C. De Wagter, T. Seidl},

url = {https://www.nature.com/articles/s42256-020-00279-7},

doi = {10.1038/s42256-020-00279-7},

year  = {2021},

date = {2021-01-19},

urldate = {2021-01-19},

journal = {Nature Machine Intelligence},

volume = {3},

number = {1},

abstract = {Flying insects employ elegant optical-flow-based strategies to solve complex tasks such as landing or obstacle avoidance. Roboticists have mimicked these strategies on flying robots with only limited success, because optical flow (1) cannot disentangle distance from velocity and (2) is less informative in the highly important flight direction. Here, we propose a solution to these fundamental shortcomings by having robots learn to estimate distances to objects by their visual appearance. The learning process obtains supervised targets from a stability-based distance estimation approach. We have successfully implemented the process on a small flying robot. For the task of landing, it results in faster, smooth landings. For the task of obstacle avoidance, it results in higher success rates at higher flight speeds. Our results yield improved robotic visual navigation capabilities and lead to a novel hypothesis on insect intelligence: behaviours that were described as optical-flow-based and hardwired actually benefit from learning processes.},

keywords = {},

pubstate = {published},

tppubtype = {article}

}