Immersive Media Technologies for Education - from Special Education to Massive Open Online Courses (MOOC) and Beyond

1. Summary of the impact

The research on interactive immersive media technologies for education, which focuses on the development of new motion tracking and gesture recognition frameworks in a virtual reality environment, enabled a series of educational programmes which have benefited over 7000 learners by September 2019 through partners such as Caritas Hong Kong, the YMCA of Hong Kong and the Catholic Diocese of Hong Kong. Many of the outputs have been adapted to benefit Mainland students with the support from both regional government bodies and nation-wide charitable organisations. Randomized controlled trials (RCT) utilising standardized psychometric tests have strongly demonstrated the effectiveness of the programmes.

2. Underpinning research

Prof. Horace Ho Shing Ip, MH, joined the Department of Computer Science, City University of Hong Kong in 1989, and became Chair Professor in 2000. He also served as the department head during 1997 to 2002. In 2000, Prof. Ip and his colleagues developed the computer algorithms that could track human gestures via infrared cameras in real time, which catalysed the follow-up research on using immersive media technologies for educational purposes. The then state-of-the-art algorithm enables the emergence of the interactive immersive media environment called Smart Ambience. The first Smart Ambience room was built with the technical facilitation from the Department in 2001. Body Brush was then designed to showcase Smart Ambience by transforming an artist’s body movements into three-dimensional abstract paintings. Body Brush was demonstrated at the Emerging Technologies Exhibition of SIGGRAPH conference in 2003 [1] with financial support from the Department, and the underlying technologies were published in the conference proceedings of the 10th ACM International Conference on Multimedia [2].

In 2006, the team started to explore the feasibility of using the technologies for children with special education needs (SEN). The Smart Ambience Therapy (SAT) programme for abused children developed in 2006 also employed the gesture-based interaction approach, enabling the children to express their feelings, learn to face threats, and embrace the sense of safety in the immersive environment. Following the success of SAT, the Interactive Sensory Programme for Affective Learning (InSPAL) was developed to meet the learning needs of students with severe intellectual disabilities. InSPAL was designed to respond to students’ movements and voices by providing multisensory stimulations, aiming to improve their safety awareness, understanding of cause and effects, balance and coordination, and attention span through eight virtual scenarios. Evaluations of the programme were conducted by adopting the intra-group repeated measurement approach on 72 participants during the school year 2012 - 2013. Results showed that the participants required significantly less support to perform physical or cognitive functional tasks after joining the programme [3]. InSPAL was integrated into the curriculum of the Mental Health Association of Hong Kong (MHAHK) Cornwall School after its launch in 2012, and the assets have been transferred from the Department to the School to form the InSPAL resource centre to continuously serve the community.

In 2015, the team started the development of the Virtual Reality-based Training Programme (vPAD) for social adaptive and emotional training on school-aged children with Autism Spectrum Disorder (ASD) in an inclusive education setting. Six learning scenarios were developed and delivered to the students via a Cave Automatic Virtual Environment (CAVE) with non-intrusive motion tracking. During 2015 to 2017, a total of 134 school-aged children with ASD participated in the study, among which 72 subjects were included for the final evaluation, making the study one of the largest studies worldwide in terms of the number of participants. Using Psychoeducational Profile Third Edition (PEP-3) as the instrument and adopting the RCT experiment design, results showed statistically significant improvements in social reciprocity and affective expression from the participants who received interventions [4].

3. References to the research

[1] Ip, H. H., Young, H., & Tang, A. C. (2003). Body brush. Emerging Technologies Exhibition, Siggraph, 2003.

[2] Ip, H. H., Hay, Y., & Tang, A. C. (2002, December). Body-Brush: a body-driven interface for visual aesthetics. In Proceedings of the tenth ACM international conference on Multimedia (pp. 664-665). ACM.

[3] Ip, H. H. S., Byrne, J., Lau, K. S. Y., Li, R. C., Tso, A., & Choi, C. (2013, August). Interactive sensory program for affective learning (InSPAL): An innovative learning program combining interactive media and virtual reality for severely intellectually disabled students. In International Conference on Hybrid Learning and Continuing Education (pp. 199-207). Springer, Berlin, Heidelberg.

[4] Horace H.S. Ip, Simpson W.L. Wong, Dorothy F.Y. Chan, Julia Byrne, Chen Li, Vanessa S.N. Yuan, Kate S.Y. Lau, Joe Y.W. Wong (2018). Enhance emotional and social adaptation skills for children with autism spectrum disorder: A virtual reality enabled approach. Computers & Education, 117, 1-15.

[5] Ip, H. H., Li, C., Leoni, S., Chen, Y., Ma, K. F., Wong, C. H. T., & Li, Q. (2018). Design and Evaluate Immersive Learning Experience for Massive Open Online Courses (MOOCs). IEEE Transactions on Learning Technologies.

4. Details of the impact

Context

Students with SEN need special educational support because of learning or adjustment difficulties such as Autism Spectrum Disorders (ASD) and Intellectual Disability (ID). According to a report from the Education Bureau (https://www.aud.gov.hk/pdf_e/e70ch03.pdf), there were about 42890 students with SEN studying in 844 local schools in 2016/17. Interactive immersive media technologies, which could enable a safe and authentic learning environment, recently become a new trend in the special education sector. The key to its success is its capability to fulfil students’ manifold learning needs by applying appropriate instructional and pedagogical designs, while keeping the technologies accessible. Before the InSPAL and vPAD projects, there was a lack of empirical studies with concrete evidences showing the effectiveness or the efficacy of using the technologies for special education purposes.

Pathways to Impact

The research started to demonstrate its impact in 2007 when SAT won a gold medal at Geneva’s Salon International des Inventions. In the same year, Prof. Ip was recognized by TIME magazine as one of the world’s top 100 innovators [A]. The success of SAT was covered by international media such as BBC [B] and CNN [C]. In 2015, South China Morning Post published a full-page report on the InSPAL programme [D]. In 2016, the local television network Cable TV broadcasted a television programme with interviews of the research team and the participating parents and teachers [E]. The TV programme not only shared the successful stories of the vPAD programme, but also increased public awareness of ASD. In 2012 and 2013, over 100 students at the Cornwall School with severe intellectual disabilities joined the InSPAL programme. InSPAL was integrated into the curriculum of the School afterwards and still benefits the community today. From 2015 to 2017, a total of 134 children with ASD joined the vPAD programme, making it the world’s largest immersive media enabled learning programme for these children. Testimonials were received from the children’s schools and their parents or caretakers [F]. Selected teachers and professional staffs from 12 local schools were also trained to equip them with necessary skills and knowledge, so that they were able to independently deliver the programme. As a result, the research team successfully transformed the study into a self-sustained programme, which significantly increased the number of potential beneficiaries. In 2018, the team established a laboratory under City University of Hong Kong Chengdu Research Institute in Sichuan province, China. Equipped with the state-of-the-art facilities, the laboratory aims not only to bring SEN learning resources to Mainland China, but to attain scalability [G].

Reach and Significance of Impact

Today, the team continuously develops new technologies for students with SEN. The project “On School-sites Virtual Reality-based Training Programme for social adaptive and emotional training on children with ASD” (vPAD-II), supported by the Quality Education Fund (QEF), is based on the success of vPAD. The three-year project commenced in 2018 will further expand the number of beneficiaries by using more portable virtual reality devices. Like vPAD, vPAD-II also provides training resources to teachers and professional staff for improving its sustainability. By September 2019, 161 students joined the programme, and 80 teachers and professional staff from 15 schools had been trained and qualified to deliver the programme independently [H].

With financial support from the University and external agencies such as the Hong Kong Jockey Club Charities Trust, the team is also applying the technologies for the greater benefit of the society, promoting social inclusion, preserving local culture and ritual practices, and bringing more public attention to under-privileged classes. The team works closely with NGOs, such as Caritas Hong Kong [I] and the YMCA of Hong Kong [J], to deliver various interactive immersive media-enabled programmes to users. By September 2019, educational workshops have been conducted in 32 primary and secondary schools covering 1348 students. Moreover, 13 NGOs and service centres have also participated in the collaborative workshops. In 2017, the team further expanded the boundaries of interactive immersive media-enabled learning by launching the online course “Virtual Hong Kong: New World, Old Traditions”, which was originally produced in the cross-media laboratory of the Department. Immersive media technologies were employed to engage learners all around the world in culture-specific festivals, carnivals and ritual practices. The course quickly became a hit on FutureLearn and attracted over 5742 learners all around the world in its year of launch.

5. Sources to corroborate the impact

[A] TIME Magazine - The world’s top 100 innovators (https://tinyurl.com/y2rbjsvj)

[B] BBC - Go Digital - Art Out of Body Movements (https://tinyurl.com/y4yxtr75)

[C] CNN - Live at Daybreak - New Art Fleshed Out in Hong Kong (https://tinyurl.com/y4j3mnt5)

[D] South China Morning Post (SCMP) on InSPAL (https://tinyurl.com/y5g4dsjk)

[E] Cable TV programme “至 FIT 男女” on vPAD (https://tinyurl.com/y4r7dpsy)

[F] Testimonials from parents and caretakers of children who joined the vPAD programme

[G] Monocle on AIMtech’s projects in Chengdu

[H] Testimonial from Catholic Diocese of Hong Kong on vPAD and vPAD-II

[I] Testimonial from Caritas Hong Kong on COSI

[J] Testimonial from the YMCA of Hong Kong on COSI