Integrating advanced technologies into autonomous drones

1. Summary of the impact

The HKUST Robotics Institute is a multidisciplinary platform for integrating, facilitating and enabling University-wide programs in robotics-related research, development and education. One of its core focus areas is unmanned aerial vehicles (UAVs). HKUST RI professors Zexiang LI and Shaojie SHEN have developed advanced technology for intelligent flight control and navigation that have improved UAV maneuverability and enabled autonomous operation in challenging environments. These have been incorporated into UAVs developed and sold internationally by DJI Technology, giving its products the advanced capabilities that have enabled it to become the world leader in commercial UAVs during the census period.

2. Underpinning research

Two technologies are critical for the commercial success of unmanned aerial vehicles (UAVs): flight control systems enabling complex acrobatic maneuvers and navigation systems enabling autonomous flight with obstacle avoidance ability even under GPS-denied environments. Researchers at the HKUST Robotics Institute (RI), Prof. Zexiang LI and Prof. Shaojie SHEN, have developed algorithms that have led to significant advances in both technologies.

Prior to 2015, algorithms for multirotor UAV control used Euler Angles to parametrize the rotation of a UAV, and controlled the yaw-roll-pitch angles separately. While this is adequate for small deviations of around the hover orientation, it limits the maneuverability required for complex motions such as throw start and flipping in the area. To address this limitation, Prof. Li supervised Yun YU the invention of a novel quadrotor control algorithm based on a viewpoint that threats attitude control as an tracking problem for a second order dynamic system on SO(3). This algorithm outperforms conventional algorithms in terms of response time and in enabling complex maneuver behaviors [1]. This improvement yields better operation UAVs operating at high speeds and during dynamic flights.

As principal investigator of UAV research at the HKUST RI, Prof. Shen, has been supervising various research projects since 2014 to develop a “visual-inertial navigation system” (VINS) that relies only on a lightweight sensor suite consisting of just an inertial measurement unit (IMU) and a single (monocular) camera. He addressed two important challenges: initialization and calibration. The nonlinearity of VINS makes monocular estimators heavily reliant on the accuracy of initial state estimates of gravity, velocity, bias, feature depth, etc. Poor initialization slows convergence, and may even lead to totally incorrect estimates. Professor Shen developed a robust on-the-fly estimator initialization algorithm by loosely aligning a vision-only structure with pre-integrated IMU measurements to recover the critical states, and then applies further bootstrapping. In 2016, Professor Shen proposed a tightly-coupled monocular visual-inertial fusion methodology for accurate state estimation with online calibration refinement, which boosts accuracy, precision and robustness significantly,

In 2017, his team open sourced the simultaneous localization and mapping (SLAM) package ‘VINS-Mono’ on GitHub under the GNU GPLv3 license [2][3]. VINS-Mono swiftly became popular, receiving over 2000 stars on GitHub. The code fostered two conference papers (P. Li et al ISMAR 2017; T. Qin et al IROS 2017) and a journal paper [4]. It has been tested, evaluated and used by major research laboratories and industry users, including the Carnegie Mellon University Robotics Institute, the Graduate Aerospace Laboratories of California Institute of Technology (GALCIT), the University of Pennsylvania GRASP Laboratory, Google Inc. and Tencent.

Professor Shen has also developed algorithms for trajectory planning for intelligent navigation. His research in robust dynamic object localization 3D camera ego-motion estimation [5] and in safe navigation through unknown cluttered environments [6] have both attracted wide citation since publication. His research results make UAVs capable of obstacle avoidance and intelligent tracking in GPS-denied environments, ultimately resulting in safe autonomous flight. He has secured over HKD19 million in external research funding from academia, industry and local government institutes.

3. References to the research

(best indicators of research quality are marked *)

[1] Y. Yu, S. Yang, M. Wang, C. Li and Z. Li, High performance full attitude control of a quadrotor on SO (3), in Proc. of the IEEE International Conference on Robotics and Automation (ICRA), pp.1698-1703, Seattle, USA, May 2015. https://ieeexplore.ieee.org/document/7139416/*

[2] T. Qin, P. Li and S. Shen, VINS-Mono: A robust and versatile monocular visual-inertial state estimator, IEEE Transactions on Robotics, 34(4), pp.1004-1020, July 2018. https://ieeexplore.ieee.org/document/8421746 *

[3] T. Qin, P. Li, Z. Yang and S. Shen, (Open source code) VINS-Mono: a robust and versatile monocular visual inertial state estimator, GitHub, May 2017. https://github.com/HKUST-Aerial-Robotics/VINS-Mono *

[4] Z. Yang and S. Shen, Monocular visual-inertial state estimation with online initialization and camera-IMU extrinsic calibration, IEEE Transactions on Automation Science and Engineering, 14(1), pp.39-51, April 2016. https://ieeexplore.ieee.org/document/7463059/?reload=true&arnumber=7463059

[5] P. Li, T. Qin and S. Shen, Stereo vision-based semantic 3D object and ego-motion tracking for autonomous driving, in Proc. of the European Conference on Computer Vision (ECCV), pp.664-679, Munich, Germany, September 2018. https://link.springer.com/chapter/10.1007/978-3-030-01216-8_40

[6] Y. Lin, F. Gao, T. Qin, W. Gao, T. Liu, W. Wu, Z. Yang and S. Shen, Autonomous aerial navigation using monocular visual-inertial fusion, Journal of Field Robotics, 35(1), pp.23-51, July 2017. http://dx.doi.org/10.1002/rob.21732 *

4. Details of the impact

While a Master’s student under Prof. Li at HKUST, Frank Wang conducted initial research on an autonomous flight control system for UAVs. He developed the unmanned miniature helicopter that achieved the world’s first flight on Mount Everest, almost reaching the summit. The team also used the exponential coordinate theory to improve the flight controller. This directly led Wang to commercialize UAVs (drones) through the founding of DJI Technology. As described below, DJI has integrated the flight control system developed by Prof. Li and the navigation technology developed by Prof. Shen into its products, resulting in improved performance and enabling it to become the world leading manufacturer of UAVs during the census period.

Yun Yu joined DJI to integrate the flight control system he developed under Prof. Li’s supervision at HKUST into DJI’s drones. This system was first integrated into the Inspire 1 series of drones, and now forms the basis for all subsequent series. After several years of development, DJI’s flight control system now enables much more stable flight of UAVs under much higher speeds. For example, the Inspire 2 can achieve a maximum speed of 94km/h [S1]. DJI’s UAVs also achieve maximum wind resistance of about 10m/s and perform well under exceptional conditions, e.g. propeller breakdowns, sudden impacts or abrupt load changes. The Inspire 2 series was listed as one of the top five drones with autonomous flight [S2].

Based on the foundational research described in Section 2, Prof. Shen developed an autonomous flight system that included a robust camera position tracker using direct semi-dense alignment of image geometric structures and subsequent integration of IMU measurements using an optimization-based fusion method. He also developed a novel algorithm for dimensionality reduction of dense 3D-range data to enable real-time dense visual-inertial 3D-mapping. This built the foundation for navigation and obstacle avoidance in cluttered environments. After sufficient online experiments in real-world environments to verify its performance, this autonomous flight system was added into DJI’s products, such as the Phantom 4 and Inspire 2 series.

The autonomous flight system developed by Prof. Shen has enabled many of the intelligent flight functions now found in DJI products. For example, a user can simply draw a route onscreen. The Phantom 4 Pro will follow it at fixed altitude, then choose the best route to return automatically depending on environmental conditions. These functions are fulfilled by using the FlightAutonomy core module, a comprehensive system for perception, navigation and control, consisting of multiple sensors and computing devices.

Impact since 2013

DJI has introduced various UAV products to the international market, all possessing the complex maneuverability enabled by the flight controller and the intelligent safe flight enabled by the autonomous navigation system. Following almost a decade of continuous research and development, DJI won the prestigious Emmy® Award for Technology and Engineering from the National Academy of Television Arts & Sciences in 2017 [S3]. This recognized DJI’s contribution to the industry, specifically for creating ‘Low latency remote controlled airborne video platforms for television’. In the same year, DJI reported CNY18 billion (US$2.83 billion) revenue, a growth of 80% from 2016 [S4]. DJI’s achievements and dominant market influence motivated its biggest US competitor, 3D Robotics, to cooperate, rather than compete, in the commercial arena [S5].

DJI’s drones have consistently been at the forefront of camera drone technology by creating safe, reliable, easy-to-use high-quality aerial imaging systems for creative professionals around the world. Time listed the Phantom as one of the ‘50 Most influential gadgets of all time’ in 2016 [S6]. Engadget listed the Spark as one of ‘The best gadgets of 2017’ [S7].

The flight control system developed at HKUST is an indispensable key technology of the controller boards made by DJI. The significance of Professor Shen’s work is best summarised by DJI’s Vice President of Research and Development [S8]. “His developed tightly-coupled visual-inertial fusion algorithm promises robust state estimation for autonomous system and encourages development of DJI drones such as Phantom 4 and Mavic series, enabling Phantom 4 series to achieve novel functions including object avoidance and intelligent track and therefore lead the quad-rotor into computer vision era. Nowadays, DJI is leading in segment of consumer drones worldwide with market share of 70% [S9], and its facilities prevailing throughout the world has renewed the definition of ‘Made of China’ to the higher level.”

To summarize, technologies originated at the HKUST Robotics Institute, and further enhanced through collaboration with DJI have led to tremendous commercial success, and continued exploration of wider-range research development and technology transfer.

5. Sources to corroborate the impact

[S1] ‘Inspire 2 Spec’, DJI, 16 November 2016. https://www.dji.com/hk/inspire-2/info#specs

[S2] ‘Top 5 drones with autonomous flight – reviews and FAQ’, Dronethusiast, 21 October 2017. https://www.dronethusiast.com/top-drones-autonomous-flight/#DJI%20Inspire%202

[S3]‘The National Academy of Television Arts & Sciences announces recipients of the 69^th Annual Technology & Engineering Emmy® Awards’, The National Academy of Television Arts & Sciences, 29 August 2017. http://emmyonline.com/tech_69th_recipients

[S4] Wang Ying, ‘Drone-maker DJI to develop more industry applications,’ China Daily, 27 January 2018. http://www.chinadaily.com.cn/a/201801/27/WS5a6bd252a3106e7dcc1371b0.html

[S5] Mike Murphy, ‘America’s top drone company couldn’t beat China’s DJI, so now they’re partners’, Quartz, 2 August 2017. https://qz.com/1042831/americas-top-drone-company-couldnt-beat-chinas-dji-so-now-theyre-partners/

[S6] Lisa Eadicicco, Matt Peckham, Alex Fitzpatrick, John Patrick Pullen, Victor Luckerson, and Claire Howorth, ‘The 50 most influential gadgets of all time’, Time, 3 May 2016. http://time.com/4309573/most-influential-gadgets/

[S7] Chris Velazco, ‘The best gadgets of 2017’, Engadget, 24 December 2017. https://www.engadget.com/2017/12/24/best-gadgets-2017/

[S8] Statement from Vice President of Research and Development, DJI, 6 March 2019.

[S9] Louise Lucas, ‘World’s biggest drone maker DJI eyes move to commercial applications’, Financial Times, 10 August 2017. https://www.ft.com/content/0d87a148-7d7f-11e7-ab01-a13271d1ee9c