From Physical to Virtual: Enhancing the Representation of Intangible Cultural Heritage Using Mixed Reality

Abstract

Intangible Cultural Heritage (ICH), including oral traditions, performing arts, and rituals, is at risk of fading due to modernization and limited preservation methods. Existing digital preservation methods often lack multi-modal integration and fail to provide a fully immersive representation of ICH. The aim is to develop an advanced framework that integrates Mixed Reality (MR) and Deep Learning (DL) techniques to create an immersive and accurate digital representation of ICH. The objective is to enhance the accessibility and longevity of cultural heritage by leveraging multi-modal data sources, including audio, textual, and visual data. First, pre-processing techniques such as noise reduction, tokenization, and data augmentation are applied to improve data quality. Next, feature extraction is performed by using Convolutional Neural Networks (CNN) for audio and visual data and Word2Vec for textual data, ensuring an accurate understanding of cultural expressions. These extracted features are then fused at the feature level, integrating multiple data modalities for a coherent and enriched representation of ICH. The processed data is integrated into a Mixed Reality environment using platforms like Unity and Unreal Engine, allowing real-time interaction through gesture recognition and voice-based controls. A Backpropagation Neural Network (BPNN) is employed to optimize content generation, ensuring adaptive and high-fidelity digital reconstruction. The proposed method is implemented by using Python 3.10.1. The proposed approach is evaluated based on realism, cultural authenticity, system performance, and knowledge retention rate, demonstrating MR-driven ML models that significantly enhance the accessibility, preservation, and educational impact of ICH.