Motion Capture and Assistive Systems

1. Summary of the impact

The research led to not only significant advancement of research in motion capture and assistive devices/systems but also nurture the establishment of one company commercializing the technologies and products of motion capture systems and motion assistive devices that have been used worldwide in film-making (special effects), animation and game interaction, sports training, and medical rehabilitation. The users of the developed products/technologies include Emmy Award winning ‘Game of Thrones’, 60% of US PGA top 100 coaches, and NASA for the commemoration of the 50^th Anniversary of Human Landing on Moon.

2. Underpinning research

At The Chinese University of Hong Kong (CUHK), we developed smart actuators for assistive knee braces to provide assistance to disabled or elderly people with mobility problems [1][3][5]. A magnetorheological (MR) actuator was developed to be used in assistive knee braces to provide controllable torque. The MR actuator can work as a brake or a clutch. When active torque is needed, the DC motor works and the MR actuator functions as a clutch to transfer the torque generated by the motor to the leg; when passive torque is desired, the DC motor is turned off and the MR actuator functions as a brake to provide controllable passive torque. The prototype of the developed MR actuator was fabricated and experiments were carried out to investigate the characteristics of the MR actuator. The results showed that the MR actuator is able to provide sufficient torque needed for normal human activities.

For gait assessment and rehabilitation, modeling and evaluation of human’s gait patterns is the basis. We have developed a convenient and real-time gait modeling, analysis, and evaluation method [2]. Gait states were defined based on the foot-ground contact forms of both legs. From the obtained gait state sequence and the duration of each state, the human gait was modeled as a semi-Markov process (SMP). Gait features derived from the SMP gait model were used for characterizing individual gait patterns. With this model, both the normal gaits of healthy people and the abnormal gaits of patients with impaired mobility were analyzed. Gait analysis experiments were conducted on 23 subjects with different ages and health conditions. The results showed that gait patterns are successfully obtained and evaluated for normal, age-related, and pathological gaits. Furthermore, patients suffering from neurological and orthopedic diseases or injuries usually have mobility impairment problems, and they require customized rehabilitation training to recover. We have proposed methods to determine user- adaptive assistance of assistive knee braces (AKBs) in gait rehabilitation [4]. A fuzzy expert system, which takes a patient’s physical condition and gait analysis results as inputs, was proposed to configure suitable levels of different assistive functions of the AKB. During gait rehabilitation, the AKB generated a reference knee trajectory according to the patient’s individual gait pattern, and the interaction force was controlled through a hybrid impedance controller considering the individual assistive function configuration. The proposed methods were verified through clinical pilot studies. Experimental results showed that AKB with the proposed control strategies can provide effective assistance during gait rehabilitation.

A multifunctional magneto-rheological actuator with power regeneration capability, named regenerative magnetorheological actuator (RMRA), was designed for gait assistance in the knee joint [5]. RMRA has motor and MR brake parts working in parallel that can harvest energy through regenerative braking. This novel design provided multiple functions with good energy efficiency. Geometrical optimization of the MR brake was performed based on a parameterized model, and multiple factors were considered in the design objectives: braking torque, weight, and power consumption. In addition, an innovative actuator named magneto-rheological series elastic actuator (MRSEA) was designed [6] to reduce the mechanical impedance of the motion assistive devices and filter out unwanted collisions. It can also provide large controllable braking torque with low power, and hence improve the system energy efficiency. Prototype of MRSEA was fabricated. Tests were performed to investigate the characteristics of the torsion spring pack and MR brake, and walking experiments were conducted to evaluate the performance. As compared with the electric motor, the energy efficiency of the innovative MRSEA was improved by 52.8% during a gait cycle. Novel actuators can be used in assistive knee braces and robotic exoskeletons [6].

The developed gait modeling and analysis as well as MR actuators described in the above paragraphs provide underpinning research for motion capture and assistance.

3. References to the research

(1) J. Z. Chen and W. H. Liao, “Design, Testing and Control of a Magnetorheological Actuator for Assistive Knee Braces,” Smart Materials and Structures, Vol. 19, 035029, 2010. DOI:10.1088/0964-1726/19/3/035029

(2) H. Ma and W. H. Liao, “Human Gait Modeling and Analysis Using a Semi-Markov Process with Ground Reaction Forces,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, Vol. 25, No. 6, pp. 597-607, 2017. DOI: 10.1109/TNSRE.2016.2584923

(3) H. Ma, B. Chen, L. Qin, and W. H. Liao, “Design and Testing of a Regenerative Magnetorheological Actuator for Assistive Knee Braces,” Smart Materials and Structures, Vol. 26, 035013, 2017. DOI:10.1088/1361-665X/aa57c5

(4) H. Ma, C. H. Zhong, B. Chen, K. M. Chan, and W. H. Liao, “User-Adaptive Assistance of Assistive Knee Braces for Gait Rehabilitation,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, Vol. 26, No. 10, pp. 1994-2005, 2018. DOI: 10.1109/TNSRE.2018.2868693

(5) B. Chen, X. Zhao, H. Ma, L. Qin, and W. H. Liao, “Design and Characterization of a Magneto-rheological Series Elastic Actuator for a Lower Extremity Exoskeleton,” S mart Materials and Structures, Vol. 26, 105008 (17pp), 2017. DOI: 10.1088/1361-665X/aa8343

(6) B. Chen, C. H. Zhong, X. Zhao, H. Ma, X. Guan, X. Li, F. Y. Liang, C. Y. J. Cheng, L. Qin, S. W. Law, and W. H. Liao, “A Wearable Exoskeleton Suit for Motion Assistance of Paralyzed Patients,” Journal of Orthopaedic Translation, Vol. 11, pp. 7-18, 2017. DOI: 10.1016/j.jot.2017.02.007

4. Details of the impact

Noitom Technology Ltd. was established in 2012 to redefine the motion capture paradigm (see [1] website, 1.1 and 1.2). The company focuses its research and development on mapping the human body, its movements and interaction with the environment. Now, Noitom is formed by more than 300 employees, and technicians occupied 60% of the company. It is worth to mention that Noitom has establishments in China and USA. Noitom owns more than 40 patents. The company’s name, Noitom, the word motion spelled backwards, embraces its dedication to the “art” of motion capture. In 2016, Noitom finished B+ round equity financing of RMB 200 million, with RMB 2 billion valuation in market. Eventually, Noitom became the biggest motion capture and VR technology company regarding to equity financing. Dr. Tristan Ruoli Dai is the Co-Founder of Noitom Ltd. and serving as Chief Technology Officer (CTO). He obtained his MPhil and PhD degrees from The Chinese University of Hong Kong (CUHK) in 2004 and 2007, respectively. Dr. Dai possesses rich experience in developing cutting-edge technologies, with R&D focus on vibration measurement and control, motion capture technology, motion sensing/interactive technology, virtual reality technology, wearable device, etc. The “wireless high-speed full-body motion capture system”, developed under his leadership, reached top international standards; it is an influential product in fields such as film-making (special effects), animation and game interaction. Dr. Dai received the Outstanding Alumni Award from the Department of Mechanical and Automation Engineering, CUHK in September 2014, and the Distinguished Alumni Award from the Faculty of Engineering, CUHK in March 2017 [4].

The research and development done at CUHK in motion capture and assistive devices/systems has made tremendous advances in these areas. In particular, the PhD theses by JZ Chen and H Ma [5][7], related journal papers [6][8][9] and patent [10] have impacted gait modelling and analysis as well as medical rehabilitation technologies in Noitom. Novel magneto-rheological (MR) actuators have been designed and tested for assistive knee braces (AKB) and robotic exoskeletons (EXO). Mechatronic design and control of AKB and EXO have been developed and implemented for gait rehabilitation and motion assistance. Human gait modeling and analysis have been performed using a semi-Markov process with ground reaction forces. This provides a convenient and real-time gait modeling, analysis, and evaluation method. Both Dr. J. Z. Chen and Dr. H. Ma joined Noitom, earlier and later, as core members of the company R & D team, after obtained their Ph.D. degrees from CUHK with expertise in assistive knee braces for medical rehabilitation as well as gait modeling and analysis. They have made impactful contributions to the intricate world of sensor technology to create systems that capture the level of detail and body movements, as well as intelligent rehabilitation equipment [2]. For gait assessment and rehabilitation, modeling and evaluation of human’s gait patterns is the basis. A convenient and real-time gait modeling, analysis, and evaluation method has been developed at CUHK [8]. Gait patterns were obtained and evaluated for normal, age-related, and pathological gaits. Furthermore, user-adaptive assistance of AKB has been successfully implemented in gait rehabilitation, to provide effective assistance during gait rehabilitation [9]. Those have been adopted by Noitom for the product “Dr. Joint’. Dr. Ma is the project leader of “Dr. Joint” (ref. slide #11 of sources [1] and [2] - letter from Noitom).

Noitom’s product ‘Perception Neuron’ is the world’s most affordable, adaptable and versatile prosumer motion capture system. It has been used in films ‘Game of Thrones’ (won Emmy Awards in Best Visual Effects), ‘LOGAN’, ‘Data Peking Opera’ (China), ‘Kizuna AI’ (Japan), Twenty One Points (Show Me Shorts Film Festival, New Zealand). The ‘mySwing Professional’ is the premier full-body 3D motion capture system for golf training. It provides players with capability performance and quantitative analysis. 60% of US PGA top 100 coaches became users of ‘mySwing’. ‘Dr. Joint’ has been developed to be an intelligent rehabilitation equipment to connect doctors and patients. The recent ‘Project Alice’ is a scalable commercial VR total solution for multi-users with physical props and motion capture. This has been used in NASA commemoration of the 50^th Anniversary of Human Landing on Moon. Noitom’s products and technologies have been reported by BBC, CCTV, CNN, Discovery, Bloomberg, Road to VR, etc.

The PR China Premier, LI Keqiang, also highly complimented Noitom’s technologies and products [3].

Noitom is planning to establish offices and labs in both Hong Kong and Shenzhen to further enhance the collaboration between CUHK and Noitom.

5. Sources to corroborate the impact

(1) http://www.noitom.com/; Noitom Intro

(2) Letter from Dr. Ruoli Dai, Co-Founder/CTO, Noitom

(3) Awards and recognition of Noitom

(4) Certificates of the Outstanding Alumni Award (MAE, CUHK) and the Distinguished Alumni Award (FoE, CUHK) for Dr. Ruoli Dai

(5) PhD Thesis: J. Z. Chen, A Magneto-rheological Actuator for Assistive Knee Braces, The Chinese University of Hong Kong, 2009.

(6) Journal paper: J. Z. Chen and W. H. Liao, “Design, Testing and Control of a Magnetorheological Actuator for Assistive Knee Braces,” Smart Materials and Structures, Vol. 19, 035029, 2010.

(7) PhD Thesis: H. Ma, Mechatronic Design and Control of Assistive Knee Braces for Gait Rehabilitation, The Chinese University of Hong Kong, 2015.

(8) Journal paper: H. Ma and W. H. Liao, “Human Gait Modeling and Analysis Using a Semi- Markov Process with Ground Reaction Forces,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, Vol. 25, No. 6, pp. 597-607, 2017.

(9) Journal paper: H. Ma, C. H. Zhong, B. Chen, K. M. Chan, and W. H. Liao, “User-Adaptive Assistance of Assistive Knee Braces for Gait Rehabilitation,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, Vol. 26, No. 10, pp. 1994-2005, 2018.

(10) Patent: W. H. Liao, B. Chen, L. Qin, X. Zhao, and H. Ma, “Magneto-Rheological Series Elastic Actuator,” CUHK Technology Disclosure 17/ENG/757; China Patent, ZL201710242884.5, 16 August 2019; US Non-Provisional Application No. 15/942,256, 30 March 2018 (US Patent Application Publication No. US 2018/0298973, 18 October 2018); PCT/CN2018/080467, 26 March 2018.