Using learning analytics for enhancing teaching and learning

1. Summary of the impact

Professor Pong and Professor Qu have developed innovative technological and pedagogical approaches for both online and on-campus education. Since 2014, HKUST has offered 53 MOOCs (Massive Open Online Courses), attracting over 1.8 million learners from around the world. We used MOOC data for analyzing students’ learning activities, including clickstream patterns and forum postings for reasoning and predicting student performance. We developed a system, VisMOOC, a set of intuitive, interactive and comprehensive visualization tools for users to analyze learning data. VisMOOC has been embedded into HKMOOC (a MOOC platform developed at HKUST) and has successfully provided formative and summative feedback to MOOC instructors, students and developers.

2. Underpinning research

We have published numerous peer-reviewed research papers on lecture video analysis and learning analytics of MOOC data, including three in top journals.

When students engage with online materials, they leave a record of their engagement. With thousands of students engaged with a MOOC, or even with a normal class-size for a blended-learning course, a huge amount of data can be collected with a much finer granularity than data generated in a traditional classroom. These data include clickstream data – how students watch lecture videos; forum data – what students post and how they interact with each other; and performance data – how they perform in online assessments. The HKMOOC platform (https://www.hkmooc.hk/), developed by Pong’s team based on OpenedX technology, has been customized using the visualization and learning analytics tools developed by Qu’s research group.

These learning analytics have the potential to enable teachers and course designers to better understand how students learn, providing just-in-time feedback for enhancing the delivery of teaching and learning. But such data are large, complex and heterogeneous, posing special challenges for analytics. In addition, end users of the analytics systems – instructors, students and education researchers – typically have little understanding of data analytics techniques. Qu, an expert in big data and the visualization of data, took up the problem. With his help, a set of interactive, comprehensive visualization tools has been developed, now integrated through a platform – VisMOOC – providing visualizations enabling end users, including course instructors and education experts, to analyse the data intuitively and efficiently.

Our research practice in the respective e-learning through learning analytics is further elaborated as follows:

• With the aim of understanding the teaching approaches employed by the instructors, we developed techniques for the automatic analysis of lecture videos.[1]

• For analysis of clickstream data collected by MOOC platforms, we developed the Event graph and Seek diagram.[2–3]

• For analysis of video data, we developed PeakVizor to help instructors and education experts analyse the ‘peaks’ or the video segments that generate numerous clickstreams.[4]

• For analysis of social network in Forum data, we developed NetworkSeer to help evaluate why students use forums and what they do in forums. Specifically, interactions in forums can be visualized, including where, when and why the interactions happen.[5]

• For analysis of temporal patterns in Forum data, we developed iForum to present three interleaving aspects of MOOC forums (i.e. posts, users and threads) at three different scales for effectively discovering and understanding temporal patterns in MOOC forums.[5]

• For reasoning and prediction of dropout, we developed DropoutSeer to visualize learning patterns in MOOCs.[6]

• For analysis of learning sequence, we developed ViSeq which analyses and visualizes learning sequences of different learner groups and enables better understanding of learners’ behaviours.[7]

3. References to the research

[1] Feng Wang, CW Ngo, TC Pong. ‘Simulating a Smartboard by real-time gesture detection in lecture videos’, IEEE Transactions on Multimedia, Vol. 10, No. 5, August 2008

[2] Huamin Qu, Qing Chen. ‘Visual Analytics for MOOC Data’, IEEE Computer Graphics and Applications (CG&A), 2015

[3] Conglei Shi, Siwei Fu, Qing Chen, Huamin Qu. ‘VisMOOC: Visualizing video clickstream data from massive open online courses’, PACIFICVIS, 2015, Visualization Symposium, IEEE Paci c, Visualization Symposium, IEEE Paci c 2015

[4] Qing Chen, Yuanzhe Chen, Dongyu Liu, Conglei Shi, Yingcai Wu, Huamin Qu. ‘PeakVizor: Visual analytics of peaks in video clickstreams from Massive Open Online Courses’, IEEE Transactions on Visualization and Computer Graphics, 2016

[5] Siwei Fu, Jian Zhao, Weiwei Cui, Huamin Qu. ‘Visual analysis of MOOC Forums with iForum’, IEEE Transactions on Visualization and Computer Graphics, 2017

[6] Yuanzhe Chen, Qing Chen, Mingqian Zhao, Sebastien Boyer, Kalyan Veeramachaneni, Huamin Qu. ‘DropoutSeer: Visualizing learning patterns in Massive Open Online Courses for dropout reasoning and prediction’, IEEE VAST 2016

[7] Qing Chen, Xuanwu Yue, Xavier Plantaz, Yuanzhe Chen, Conglei Shi, Ting-Chuen Pong, Huamin Qu. ‘ViSeq: Visual analytics of learning sequence in Massive Open Online Courses’, IEEE Transactions on Visualization and Computer Graphics, Oct. 2018

4. Details of the impact

Through our research and development in e-learning during the past 20 years, we have positioned ourselves as a leader in this important area which links technology and education. Our work started in 2000 with an internal grant from the Sino Software Research Institute to develop a platform for distance learning. In 2001 this led to external funding from the Hong Kong Quality Education Fund to set up a Cyber University for secondary school students and three years later to set up a Hong Kong Virtual University Campus for UGC institutions with UGC funding. These projects produced research results for analysing lecture videos automatically.[1]

Recognizing the changing educational landscape with the 2012 launching of MOOCs initiated by Coursera and edX, HKUST joined both platforms as their first university partner from Asia. Since 2014, HKUST has offered more than 50 MOOCs, attracting over 1.8 million learners from around the world. Pong also spearheaded setting up the HKMOOC platform with UGC funding of HK$10 million in 2015.

To date, this work has attracted a high level of research and development funding – exceeding US$10 million – with Qu and Pong serving as the PI or co-PI. Supported by the Hong Kong Innovation Technology Center, an ongoing project to build an open-source, visualization platform, combining a learning studio module, a data-analytics module for MOOC data, and a data-visualization module, has drawn in industry matching-funds from local and multinational companies.

The following projects were based on the pedagogies and technological tools that we developed.

STEM education for K-12 sector

We have initiated collaborative projects between universities and secondary schools in Hong Kong to nurture STEM literacy of senior secondary students by providing them with exposure to university education through blended learning. Through the Chemists Online project (https://learn.hkmooc.hk/courses/course-v1:HKVU+COSAS+2019_Q4_R1/about), Hong Kong university professors delivered a set of online seminars on topics related to the secondary school curriculum in Chemistry. During the past three years, the program has attracted over 20,000 students from around 170 secondary schools.

In 2018, in collaboration with HKedCity, we launched the Code2App Challenge program which introduces computational thinking (CT) to the K-12 sector using the App Inventor MOOC developed at HKUST (https://www.hkedcity.net/code2app/en). Teachers and students acquire CT skills through the MOOC and then participate in an App development competition with sustainability themes. Over 10,000 teachers and students registered for the MOOC and around 250 students participated in the competition with encouraging results. The program continued in 2019.

In both the Chemists Online and Code2App projects, learning analytics obtained from the HKMOOC platform were used to improve the content delivery and the design of training workshops offered to teachers and students.

MOOC on Java programming

Developed by Pong, this course was developed based on the approaches to lecture video analysis.[1] First offered in 2014, it has been re-offered continuously in an on-demand mode since 2015. The course content has been enhanced based on the learning analytics tools developed by Qu’s research group. Attracting over 400,000 learners internationally, the course was selected from among several thousand courses offered on edX as one of the top 10 most popular courses in 2016 (https://blog.edx.org/10-most-popular-courses-edx-courses-in-2016) and one of the top 17 courses in 2017 (https://blog.edx.org/17-top-edx-courses-2017).

Impact of HKMOOC beyond Hong Kong

Following our success in MOOC developments, Pong hosted a Workshop at HKUST in 2017 with university representatives from the USA, Europe and Australia to discuss establishing a Global Virtual Exchange Alliance (GVEA). Implementation details were finalized during the Workshop and the agreement was signed by nine HE institutions from the USA, Netherlands, Switzerland, Australia and HKUST for offering credit-bearing MOOCs to students from partner institutions. The HKMOOC is being used as the platform for offering HKUST courses to students from the GVEA. (For more information about the Alliance visit: https://studyabroad.ust.hk/outbound/programs/virtual_exchange)

To encourage closer collaboration in teaching and learning innovations among HE institutions in Asia, an alliance similar to the GVEA was launched in 2018. A program on Extended Flipped Education is being piloted using the HKMOOC platform for the Association of East Asian Research Universities (AEARU), which was established by 17 top East Asian research universities. Through the program, two HKUST MOOCs and one Tokyo Tech MOOC have been offered using the extended blended learning approach. (Visit: https://hkvu.hk/hkvu/aearu/)

Our MOOCs were initially offered free as a service to education but more recently have started to generate revenue, estimated at ~US$4 million to date. Research results from our projects have been published in six peer-reviewed journals and presented in conferences including the Coursera Asia Partner Symposium, MOOC Conference in Big China Area, IEEE VIS Conferences and IEEE PacificVis Conference.

5. Sources to corroborate the impact

1. Dr Una-May O’Reilly, PhD Principal Research Scientist, CSAIL, MIT (http://people.csail.mit.edu/unamay/)
2. Mr Tin-Ming Kwok, CEO, Trumptech Limited, Hong Kong (Email: ming.kwok@trumptech.com)
3. Dr Raymond Fong, Senior curriculum development officer (Science), Hong Kong Education Bureau (Email: raymondfong@edb.gov.hk)
4. Dr Chi Keung Mak, Former principal, Lok Sin Tong Young Ko Hsiao Lin Secondary School (Email: chemmck@netvigator.com)
5. Professor Cai Lingru, Associate Professor, Department of Computer Science & Technology, Shantou University, Shantou, China
6. Mr Victor Cheng, Executive Director, Hong Kong Education City (HKedCity)
7. Significant impacts achieved through our work have been well recognized by the community, as evidenced by the awards received, including: 1st place in the Natural Sciences Award and 2nd place in the Hybrid Learning Award of the Wharton-QS Stars Awards Global Competition in 2014; the Silver Award for Best Innovation (Innovative Technology) in the 2015 Hong Kong ICT Awards; the Merit Award in E-Learning in the 2015 Asia Pacific ICT Awards; the Best Visualization Notes Award in the IEEE Pacific Visualization Symposium in 2016; and Best Lightning Speaker Award for presenting the invited Lightning talk Using MOOCs and Experiential Learning for Competency Development during the conference IEEE LWMOOCS 2019 and news scripts.