Interactive, Immersive Technologies for Cultural Heritage Preservation and Display

1. Summary of the impact

Dr. Jeffrey Shaw has pioneered the development of new forms of interactive and immersive 3D technologies for the purpose of preserving and displaying tangible and intangible cultural heritage. Shaw’s research led to the creation of a cultural heritage museum in Hampi, India, instigated the first Dunhuang Caves digital heritage preservation project which was subsequently taken over by the Chinese government, and has fundamentally reshaped the nature of cultural heritage preservation and display in Hong Kong and elsewhere over the past six years, through the impact of carefully staged exhibitions and through collaboration with both public and private institutions.

2. Underpinning research

Shaw, who pioneered interactive art in the 1960s, has spent the second half of his career developing practical digital media technology applications in the fields of virtual and augmented reality and immersive visualization. Since 2013, Professor Shaw has developed interactive, immersive technologies for the preservation and exhibition of tangible and intangible cultural heritage in collaboration with Professor Sarah Kenderdine of the École Polytechnique Fédérale de Lausanne. His calibration of interactive, immersive technologies to showcase specific cultural heritage forms and sites has yielded a cumulative impact.

Between 2006 and 2008, Professor Shaw conceived and developed the AVIE system, a 3D, 360-degree, VR, interactive and immersive projection and exhibition display funded by the Australian Research Council (R2). This system “immerses users within a projection space, allowing them to physically interact across its entire breadth, while enabling natural interaction with visualizations via a full-body motion-tracking interface” (S1). It has been exhibited at 48 international venues, including The Smithsonian in Washington. It is permanently installed at the Museum Victoria in Melbourne, and at ZKM in Karlsruhe. It also shaped new safety and learning procedures in the Australian Mining Industry (S1).

From 2012, drawing upon his expertise with AVIE, and with funding from JSW Foundation, Mumbai, India, Shaw teamed with Kenderdine to develop a digital cultural heritage installation on the Hampi UNESCO World Heritage Site in India. Place Hampi provides an interactive, 3D, immersive visualization of the entire site (R1). The viewer navigates to a virtual location and enters an immersive 360 panorama where they can activate 3D renderings of Hindu deities. Place Hampi was exhibited at 11 international venues. A permanent museum was then created at Kaladham Art Village, India, to house it (S3).

Between 2012 and 2014, Shaw developed Pure Land: Inside the Mogao Grottoes at Dunhuang, an interactive, immersive visualization project based on the Dunhuang Caves funded by the Friends of Dunhuang, Hong Kong (R3, R4). Pure Land 360 presents a 3D, 360- degree, 1:1 digital rendering of Cave 220. Elements may be enlarged for a closer view through a virtual magnifying glass and animated in color and sound. In Pure Land AR, the visitor interacts in close up with their iPad the digitally rendered copy of Cave 220 (R5). These exhibits were shown at 11 international venues, led the Dunhuang Academy to initiate a digital cultural heritage program, and transformed museum practices in Hong Kong and elsewhere. Washington Post’s Philip Kennicott reported: “At last we have a virtual reality system that is worthy of inclusion in a museum devoted to the real stuff of art” (S2).

Since 2016, Shaw and Kenderdine have teamed up with Mr. Hing Chao of the Guoshu Foundation, Hong Kong, to develop long term strategies for the preservation and exhibition of Hakka Kung Fu. Shaw’s motion capture of kung fu masters has created an archive of more than 9 recordings, and he has developed innovative strategies to present their artistry in 3D visualizations (R6). This material has informed three widely-seen exhibitions that will form the basis of a permanent museum (S6).

3. References to the research

(R1) Jeffrey SHAW, Sarah KENDERDINE, and Roderick COOVER. 2011. “Re-place: The Embodiment of Virtual Space.” In Switching Codes: Thinking Through Digital Technology in the Humanities and the Arts, edited by Thomas Bartscherer and Roderick Coover, 218-237. Chicago: University of Chicago Press.

(R2) Sarah KENDERDINE, Jeffrey SHAW, and Tobias GREMMLER. 2012. “Cultural Data Sculpting: Omnidirectional Visualization for Cultural Datasets.” In Knowledge Visualization Currents: From Text to Art to Culture, edited by Francis Marchese and Ebad Banissi, 199-220. London: Springer-Verlag.

(R3) Leith Kin Yip CHAN, Sarah KENDERDINE, and Jeffrey SHAW. 2013. “Spatial User Interface for Experiencing Mogao Caves.” SUI 2013 – Proceedings of the ACM Symposium on Spatial User Interaction: 21-24.

(R4) Sarah KENDERDINE, Leith Kin Yip CHAN, and Jeffrey SHAW. 2014. “Pure Land: Futures for Embodied Museography.” ACM Journal on Computing and Cultural Heritage, 7:2 (July): 1-15.

(R5) Sarah KENDERDINE and Jeffrey SHAW. 2014. “A Cultural Heritage Panorama: Trajectories in Embodied Museography.” In Digital Heritage and Culture: Strategy and Implementation, edited by Herminia Din and Steven Wu, 197-218. Singapore: World Scientific Publishing Company.

(R6) Sarah KENDERDINE and Jeffrey SHAW. 2016. “A Digital Legacy for Living Culture.” In 300 Years of Hakka Kung Fu: Digital Vision of its Legacy and Future, edited by Hing Chao, Jeffrey Shaw and Sarah Kenderdine, 165-189. Hong Kong: International Guoshu Association.

Selected Awards and Citations

(A1) 2010. Gold Medal, American Association of Museums MUSE Awards; (A2) 2014. Lifetime Achievement Award, Society of Art and Technology, Montreal, Canada; (A3) 2015. Hong Kong Arts Development Council Arts Promotion Award 2015; (A4) 2015. Prix Ars Electronica Golden Nica for Visionary Pioneer of Media Art, Linz, Austria

Selected External Competitive Grant Funding (Since 2011):

(G1) 2011-15. AUD 224,309. Australian Research Council (ARC). On interactive cinema. Co-I; (G2) 2012-15. Euro 21,000. Europeana Foundation. Ecloud WW1. Co-PI; (G3) 2014-17. AUD 628,400. ARC. Museological Data in Interactive Systems. Co-PI; (G4) 2016-17. HKD 475,496. ARC. Landscape as a User-Generated Interactive Aesthetic. Co-I; (G5) 2018-21. HKD 267,104. RGC Hong Kong. Navigational Deep Mapping of Cultural Atlases in Museums. PI.

Selected Other Grants (Since 2012):

(O1) 2012-16. HKD 770,000. Animating the Pirates Scroll. Hong Kong Maritime Museum. PI; (O2) 2012-14. HKD 800,000. Dunhuang Academy. Hong Kong. Grant for ALiVE. PI; (O3) 2013. HKD 500,000. Moonchu Foundation. Confucian Rites. PI; (O4) 2018. HKD 2,696.980. Ocean Park. 360-degree 3D Video Production. PI; (O5) 2018-19. HKD 250,000. HK Police College. Immersive Environments; (O6) 2018-21. HKD 27,792.000. HK Tourist Board. PI.

4. Details of the impact

Jeffrey Shaw has pioneered the development of new forms of interactive and immersive 3D Technologies which have transformed the way in which museums have approached the display of cultural heritage both in Hong Kong and elsewhere. In the words of Professor Harold Thwaites, Shaw’s research has been “of central importance to the major change of direction that took place in cultural organizations’ outlook towards digital heritage.” According to Louis Ng, Deputy Director of Leisure and Cultural Services, Hong Kong, in the past museums “had been resistant to putting anything but the ‘real’ object on display” (S4). However, he wrote that Shaw’s interactive digital displays have “transformed visitor experience and audience engagement” and they have also aided preservation of original artifacts which can now be accessed “virtually rather than literally” (S4). The impact of Shaw’s work can be assessed through the cultural heritage installations that Shaw has created and inspired.

The Creation of a Heritage Museum at Hampi UNESCO World Heritage Site, India

The buildings of Hampi, a vast 4,000 hectare 14th century site of the capital city of the Vijayanagara Empire in Southern India, form the content of Shaw’s navigable, immersive installation Place Hampi. Sangita Jindal, Chairman of the JSW Foundation, one of India’s largest business organizations, was inspired by Shaw and Kenderdine’s installation in Melbourne to build a Museum near Hampi to house it. “The Shaw-Kenderdine exhibition showed to me how the immersive visualization of a cultural heritage site has the power to engage visitors in an understanding and experience of cultural heritage that a traditional museum and visitor’s site cannot provide” (S3). Since 2012 Shaw’s panoramic, immersive installation has been the main feature within the Museum which receives up to 1400 visitors a month. Noting visits from government delegations from the Indian offices of tourism and archeology, as well as museum professionals from South Africa, Italy, Switzerland and the United States, Jindal comments that Shaw’s work “has transformed how those of us working with UNESCO World Heritage understand the potential for digital resources in preserving and representing cultural heritage.” The installation received extensive and favorable press reviews. BK Ramesh of the Deccan Herald writes “one gets the feeling of entering a sophisticated theatre that unveils the drama of mythological, historical as well as archeological details of a medieval Indian metropolis, Hampi” (S3).

The Creation of a Digital Cultural Heritage Strategy in China

Between 2012 and 2014, Shaw created and refined the first digital, interactive and immersive representation of the Dunhuang caves, a site of 492 Buddhist caves constructed over a span of 1000 years. Shaw’s 1:1 scale visualization system of Cave 2020 integrating interactive, 3D augmented reality features was displayed at the Dunhuang academy and Shaw conducted workshops on how he created the system. Shaw’s innovation kick-started China’s Dunhuang digital heritage initiative. Dr. Louis Ng, writes: “[Shaw’s] accomplishments in promoting the cultural heritage of Dunhuang through cutting-edge multimedia technology have directly inspired, and provided a template for the development of a broader Dunhuang digital preservation project supported by the Chinese government” (S4). Professor Harold Thwaites attests: “This work predates (and sets the paradigm for) the flurry of exhibitions of the caves they are doing now, both in the museum environment and in the content on the Dunhuang Academy website. It opened a once “closed door” for the research-creation of rich, meaningful, digitally preserved cultural heritage of China that heretofore would not have been accessible to the public” (S2).

The Dunhuang project was also showcased at the Hong Kong International Art Fair in 2012 and inspired the development of digital museology in Hong Kong by demonstrating the possibilities of interactive, immersive visualization as an alternative to object and text-based exhibition strategies. As a direct result of Shaw’s installation, two exhibitions were staged at the Hong Kong Heritage Museum on the Dunhuang Caves in 2015 and 2018, which attracted more than 200,000 visitors. These exhibitions “would not have taken place were it not for the leadership and inspiration Shaw provided in the field,” Ng writes.

In 2013, when the Hong Kong Maritime Museum moved its headquarters from Stanley to Pier 8 in Central district, its director Richard Wesley encountered Shaw’s work on Hampi and the Dunhuang Caves and recruited Shaw to develop a digital visualization strategy for the new museum. Shaw’s immersive visualizations of the museum’s “pirate scrolls” formed a central feature of the inaugural show. Since then, under the guidance of Professor Shaw, the Maritime Museum has made a significant investment in immersive technologies for exhibition and teaching programs, such that, according to Wesley, “Jeffrey has had a major impact on the trajectory of the museum’s interpretation and public communication strategy and programming” and this was noted by his colleagues in the International Congress of Maritime Museums (S5).

Since 2016, in collaboration with Hing Chao, Shaw has created an archive of motion captured movements of Kung Fu masters and with Kenderdine he has developed innovative and engaging ways to visually render and exhibit the motion captured material. Shaw’s archive and exhibits have formed the basis of three shows in Hong Kong: 300 Hundred Years of Hakka Kung Fu (2016), Lingnan Kung Fu in Cinema and Community (2018) and Safeguarding the Community: An Intangible Cultural Heritage New Media Exhibition (2018). Hing Chao writes that these exhibitions “have served to transform public perception of Chinese martial arts, [and] given the martial arts a renewed visibility within Hong Kong.” (S6). As a result, a permanent museum of Hakka Kung Fu is planned for Hong Kong to house these exhibits.

Gobal Impact

Under Shaw’s guidance, Professor Harold Thwaites led a project on the digital conservation and exhibition of Mah Meri mask culture in Malaysia, designed an exhibition on The Textile Tales of Pua Kumba, and led a team of researchers at Sunway University in Kuala Lumpur to digitally conserve and display other aspects of Malaysian cultural heritage. Thwaites, who is writing A History of Virtual Heritage concludes that “Professor Shaw’s numerous internationally exhibited and critically acclaimed digital cultural heritage projects are milestones of technological and cultural innovation. They have had a seminal impact on the theory, design and application of digital media in art, society and industry the world over.”

5. Sources to corroborate the impact

(S1) Statement from Dennis Del Favero, Scientia Professor, University of New South Wales.

(S2) Statement from Harold Thwaites, Professor of Creative Digital Media, Head, Center for Research Creation in Digital Media.

(S3) Statement from Sangita Jindal, Chairman of the JSW Foundation.

(S4) Statement from Louis Ng, Deputy Director, Leisure and Cultural Services Department, Hong Kong.

(S5) Statement from Richard Wesley, Director, Maritime Museum, Hong Kong.

(S6) Statement from Hing Chao, Director, International Guoshu Association Limited.