Motion tracking and recognition using Internet-of-Things MEMS-based inertial sensors

1. Summary of the impact

The work of Prof. W. J. Li’s research team related to “Motion tracking and recognition using Internet-of-Things MEMS-based inertial sensors” since 2000 has had significant academic and economic impact. The team received more than HK$10M grants for related research and published in highly-visible journals. During the assessment period, the team members have also started several companies (currently employing ~100 employees total) commercializing products that merged MEMS sensing and artificial intelligence technologies:

• Dr. WANG Yufan founded AI Motion Sports – the first company in Hong Kong to receive funding from Entrepreneur First (British VC) and HAX (San Francisco+Shenzhen VC).
• Dr. SHI Guangyi founded BEWISE Sensing (Wuxi, China) – the company has reached a milestone of over $1M RMB sales income per month recently.
• Dr. FEI Fei founded Ningbo Baite Baize Information Technology, Ltd. (Ningbo, China) and Chengdu Shichuang Information Technology, Ltd. (Chengdu, China).

2. Underpinning research

Over the past years, Prof. W. J. Li’s team conducted comprehensive research activities related to nano/micro/MEMS sensors -- spanning from novel device fabrication, device characterization and integration, sensor fusion and signal processing, to more recently in combining IoT sensor data collection and AI-based algorithms to develop novel applications. An example of the team’s pioneering effort in device fabrication is their development of the world’s first convective thermal motion sensor using carbon-nanotubes as sensing and heating elements. This sensor uses CNT (carbon nanotube) bundles manipulated by dielectrophresis as both a heater and thermal detector, and demonstrated that only picowatt level input power is necessary to active the sensing element as opposed to conventional polysilicon-based convective sensing elements that require microwatt power input. This sensor could respond to sinusoidal vibrations up to at least 10Hz and detected linear acceleration as low as 0.02m/s. The team has also more recently explored different nanomaterials to create various high-responsivity force and pressure sensors (e.g., see [1]). In terms of creating novel applications by integrating sensors and using advanced signal processing techniques (e.g., using adjustable measurement noise covariance, extended Kalman filter, etc.) the team has developed and an intelligent sensing system for human airbag deployment and a 3D digital writing instrument.

More recently, the team pioneered the development of using machine learning related algorithms (e.g., feature extraction, data segmentation, dynamic time warping, template matching, support vector machine, etc.) to classify and recognize motions related to human arterial pulse wave forms, mice motions, hand gesture motions, and sports activity motions. For example, the team developed a nonspecific-user hand gesture recognition algorithm for MEMS motion sensors, which was capable of recognizing seven hand gestures, i.e., up, down, left, right, tick, circle, and cross, based on the input signals from MEMS 3-axes accelerometers in real-time. To compress data and to minimize the influence of variations resulted from gestures made by different users, a basic feature based on sign sequence of gesture acceleration was extracted. This method reduces hundreds of data values of a single gesture to a gesture code of 8 numbers. Finally, the gesture is recognized by comparing the gesture code with the stored templates. The team achieved an overall recognition accuracy of 95.6%, with the correct recognition accuracy of each gesture ranging from 91% to 100%. They further extended this work by merging MEMS and vision sensors (see [2]) published in 2014, which was a top 50 most downloaded paper of IEEE Sensor Journal since 2001) and showed that inertial microsensor data sampled at 100 Hz and vision sensor data at 5 frames/s could be fused by an extended Kalman filter and used for accurate human hand gesture recognition and tracking. A novel adaptive algorithm was developed to adjust measurement noise covariance according to the measured accelerations and the angular rotation rates of human hand motions. The experimental results verify that the proposed method is capable of reducing the velocity error and position drift in an MEMS-based inertial sensor when aided by the vision sensor. Compensating for the time delay due to the visual data processing cycles, a moving average filter is applied to remove the high frequency noise and propagate the inertial signals. The team showed that reconstructed trajectories of the 10 Arabic numerals could be recognized using dynamic time warping with a direct cosine transform for feature extraction, resulting in an accuracy of 92.3% and individual numeral recognition within 100ms.

The team’s research work in the past decade culminated into their recent exploration of IoT framework for next-generation racket sports training (e.g., see [3] and [4]). Wireless wearable sensing devices (WSDs) based on MEMS motion sensors were developed to recognize different badminton strokes and classify skill levels from different badminton players. The WSDs is low-cost, easy-to-use, and computationally efficient compared to video-based methods for analyzing badminton strokes; they offer the advantage of dynamic monitoring of multiple players in indoor and outdoor environments. This IoT framework aims to change the way of racket sports training from experience-driven (subjective) to data-driven (objective), and which can be easily extended to analyze the motions and skill levels of players in other racket sports (e.g., tennis, table tennis, and squash) for training and/or practice.

3. References to the research

[1] Qi-Jun Sun, Jiaqing Zhuang, Shishir Venkatesh, Ye Zhou, Su-Ting Han, Wei Wu, Ka-Wai Kong, Wen Jung Li, Xianfeng Chen, Robert Li, Roy A. L. Vellaisamy, “Highly Sensitive and Ultra-Stable Skin-Sensors for Bio-Pressure and Bio-Force Measurements Based on Hierarchical Microstructures”, ACS Applied Materials & Interfaces, Jan 2018, DOI:10.1021/acsami.7b16611 (impact factor ~8.45 in 2018).

[2] Shengli Zhou, Fei Fei, Guanglie Zhang, John D. Mai, Yunhui Liu, Jay Y J Liou, Wen J. Li, "2D Human Gesture Tracking and Recognition by the Fusion of MEMS Inertial and Vision Sensors", IEEE Sensors Journal, 14(4), 2014, pp 1160-1170. (One of the 50 most downloaded IEEE Sensors Journal papers in the month of October 2014; included in this count are all IEEE Sensors Journal papers published since its foundation, i.e., about 4,000 papers in total.)

[3] Yufan Wang, Meng Chen, Xinyu Wang, Rosa H. M. Chan and Wen Jung Li, “IoT for Next-Generation Racket Sports Training”, IEEE Internet of Things Journal, DOI: 10.1109/JIOT.2018.2837347, May 2018 (impact factor ~9.515 in 2018).

[4] Yufan Wang, Yuliang Zhao, Rosa HM Chan, Wen J Li, “Volleyball Skill Assessment using a Single Wearable Micro Inertial Measurement Unit at Wrist”, IEEE Access, January 2018, DOI: 10.1109/ACCESS.2018.2792220 (impact factor ~4.08 in 2018).

4. Details of the impact

• AI Motion Sports (AIMS) Ltd. (Shenzhen, China) [A]

  AIMS is co-founded by Dr. WANG Yufan (a former PhD student of Prof. Li) and has received funding from Entrepreneur First (British VC) and HAX (San Francisco+Shenzhen VC) [B]. The company current provides a youth sports education and management platform powered by artificial intelligence, big data analysis and motion capture technology. AIMS products monitor, record and analyze students’ sports and health data, further provide insight-reports for schools, coaches, and parents. AIMS digitizes the youth sports performance to improve efficiency in sports training, and management. While a PhD student at CityU, Yufan used the “IoT Sensors for Sports Motion Recognition” work to win Third Prize in the International Innovation and Entrepreneurship Contest in December 2017. The contest was organized by the Taiyuan Municipal People's Government and China Center for Information Industry Development. The team competed with over 1000 submissions and 300 finalists from industries and universities worldwide.

• BEWIS Sensing LLC. (Wuxi, China) [C]

  BEWIS Sensing was co-founded by Dr. SHI Guangyi (a former PhD student of Prof. Li), with the headquarters in Wuxi (Jiangsu Province). The company’s first products were inclinations sensors for high-rise crane tilt-detection. About a year after it was founded, BEWIS Sensing became a top-4 supplier of inclination sensors in China. The company has since then expanded its product lines to include electronic compass, attitude and heading reference systems, inertial measurement unit, integrated navigation system, etc. The company’s current customer base span across industries in intelligent transportation, construction engineering, engineering machinery, aerospace, and energy grid. The technology used to develop BEWIS Sensing’s initial products were based on Prof. Li’s over 10 years of experience in developing MEMS motion sensors for motion tracking and recognition. More recently, Prof. Li has served as a technical advisor to BEWIS Sensing and convinced them to develop algorithms to enhance MEMS sensor accuracy performance and motion recognition using artificial intelligence related algorithms. As a China National High-tech Enterprise, BEWIS Sensing is a world-leading inertia sensor supplier. Focusing on inertial sensing technology, the company has persisted in continuous innovation and open cooperation with international companies since 2013. The company has automation manufacturing bases in Zhejiang Province, and currently wholly owned subsidiaries in the United States, Germany, Dalian and Xiamen. At present, it’s business covers more than 50 countries and regions, serving more than 5,000 customers. To adapt to the revolutionary changes that are taking place in IoT, BEWIS Sensing is constantly innovating around customer demands. Since its inception, the company has developed hundreds of types of tilt sensors, inclinometer switch, electronic compass, AHRS (attitude and heading reference systems), inertial measurement unit, etc. BEWIS Sensing has recently won the Gold Award for the 2018 World IoT Expo in September 2018. The expo was held in Wuxi, China, by the Ministry of Industry and Information Technology, Ministry of Science and Technology, and Jiangsu Provincial People's Government. The company has recent reached a sales income of more than 1Million RMB/month. As mentioned above, CityU researchers (Prof. Li and postdoctoral fellows Dr. ZHOU Shengli and Dr. FEI Fei) investigated several technologies to enhance sensor tracking accuracy and developed a technique to combine optical-tracking and MEMS-sensor-based tracking, with the implementation of Kalman Filtering algorithms. Based on the findings from this work, Dr. FEI Fei has co-founded the following 2 companies in the past 4 years:

• Ningbo Baite Baize Information Technology Co., Ltd. (宁波百特百泽信息科技有限公司)

  Date of establishment: 2015-04-02
  Unified Social Credit Code:
  91330204316922861D Company Type: Limited
  Company
  Industry: Information Transmission, Software and Information Technology Services
  Address: No. 13, Building 13, Lane 21, Dasheng Road, Yinzhou District, Ningbo, Zhejiang
  Province
  Number of employees: 20
  Main products: Inertial sensing unit, rehabilitation data gloves, myoelectric sensor
  

• Chengdu Shichuang Information Technology Co., Ltd. (成都势创信息技术有限公司)

  Date of establishment: 2016-09-13
  Unified Social Credit Code: 91510100MA61XL0U19
  Company Type: Limited Company
  Industry: Information Transmission, Software and Information Technology Services
  Address: 6th and 7th floors, Building A, Rongchuang Plaza, No. 200, Tianfu 5th Street,
  Chengdu High-tech Zone
  Number of employees: 8
  Main products: Inertial sensing unit, wearable heart rate and pulse sensor
  

• Collaborations with other companies and universities

  With the team’s globally-leading work related to MEMS-based motion tracking and recognition sensing technologies, two local companies, i.e., AIphonics Limited [D] and Cascube Limited [E], are currently funding the team to develop novel products. The outcome of these two collaborative projects will be further commercialized as products which will impact the LIDAR (light detection and ranging) and laboratory animal test industries. In addition, the CAS-HK Joint Laboratory on Robotics (headed by Prof. W. J. Li) has been recently granted a project to further develop embedded MEMS motion sensors with 3D integrated robotics and sensing structures [F]. The team will use an advanced multi-layered nano-ink circuit deposition technology to realize complicated but compact robotic sensing structures. Moreover, the team has also collaborated with Prof. Yuliang Zhao (a former Ph.D. student of Prof. Li) from the Northeastern University at Qinhuangdao on developing different applications of the MEMS motion sensors, including virtual keyboard [G] and indoor position tracking [H] technologies.

5. Sources to corroborate the impact

[A] https://www.aimotionsports.com/en/

[B] https://www.joinef.com/ef-hax-funding-the-worlds-best-founders/ (EF+HAX’s international announcement of funding AI Motion Sports)

[C] https://www.bwsensing.com/ (Company website of BEWIS Sensing)

[D] Principal Investigator, Hong Kong Innovation and Technology Fund — University-Industry Collaboration Programme (Matching Grant for Joint Research Scheme): “A MEMS-based Light Detection and Ranging (LIDAR) System with Super-resolution Microlens for Enhanced Structured-light 3D Imaging and Mapping”, Project Number: UM/382, HK$1.14Million, start date: July 15, 2019 (approved on June 10, 2019)

[E] Principal Investigator, Hong Kong Innovation and Technology Fund — University-Industry Collaboration Programme (Matching Grant for Joint Research Scheme): “An Implantable Micro- Sensing System for Tracking Animal Motion Behaviors”, Project Number: UM/326, HK$1,143,100 (HK$571,550 from ITC; HK$571,550 from private company), Nov 2017-Apr 2019.

[F] Principal Investigator, CAS-HK Joint Laboratory Scheme (JLFS): “Development of 3D Integrated Robotics and Sensing Structures using Multi-layered Nano-ink Circuit Deposition”, Project Number: JLFS/E-104/18, HK$2.99Million, start date: July 01, 2019 (approved on June 19, 2019).

[G] Yuliang Zhao, Chao Lian, Xueliang Zhang, Xiaopeng Sha, Guangyi Shi, and Wen J. Li, “Wireless IoT Motion-Recognition Rings and a Paper Keyboard”, IEEE Access, 7, 09 April 2019, pp 44514-44524, doi:10.1109/ACCESS.2019.2908835.

[H] Yuliang Zhao, Jiaqi Liang, Xiaopeng Sha, Jianing Yu, Hongjun Duan, Guangyi Shi, and Wen J. Li, “Estimation of Pedestrian Altitude Inside a Multi-Story Building Using an Integrated Micro-IMU and Barometer Device”, IEEE Access, 7, 24 June 2019, (DOE: 10.1109/ACCESS.2019.2924664).