Rapid development in China has destroyed massive cultural heritage sites, such as traditional old family houses in sub-urban towns, along with the artefacts, stories, memories, and people connections those heritage sites once nurtured. Meanwhile, intangible cultural heritage is also disappearing, as urbanisation and the accompanying lifestyle changes hinder younger generations from experiencing traditional culture, namely speaking local dialects, understanding family stories, practising traditional customs, etcetera. However, preserving and passing on cultural heritage is essential as it helps transmit cultural knowledge, enrich life through cultural connections and people’s relationships, shape identity and lead to the future. Under such urgent circumstances, this project aims to develop a prototypical immersive cultural experience that not only meets my big family’s aspirations but also serves as a model for cultural initiatives across China.

My great-grandmother’s old house in Ningbo, China, was demolished two years ago, and its furniture and artefacts were dispersed among sub-families long before that. This house, which witnessed my grandmother’s birth, was built when my great-grandmother married and would have been over a hundred years old today. It accommodated my grandmother, her elder sister, and my uncles, holding many cherished memories. To fulfil our big family’s collective aspirations, we decided to digitally reconstruct the house and its artefacts and develop an interactive VR experience to reconnect with the vanished ‘home’ and pass on our cultural legacy to younger generations. We also wanted to enable a participatory process that strengthens our family bonds.

In 2020, an initial meeting was held to identify the family’s needs, followed by two fieldwork to document the site through measuring, mapping, photos, and videos before the building was demolished. Although we lacked 3D scanning devices for accurate site scans, this challenge inspired us to explore rendering techniques in remodelling that might resonate with family elders’ memories and multimodal experiences, including visual, physical, and auditory interactive elements that enhance the level of immersion.

In late 2023, after I returned from abroad, we resumed this project. The basic architecture model with materials was created with Rhino and Blender and integrated into Unity 3D. Artefact scanning techniques photogrammetry and Gaussian splatting with easily accessible phones and cameras were tested. In March 2024, the four sub-families gathered in Ningbo for a participatory workshop. We checked the correctness of the model, documented and scanned all the artefacts and furniture, discussed the virtual spatial configurations, and formulated plans for the next steps. Specifically, elders were actively engaged in remembering the original condition of the building, sharing stories behind the space, artefacts and furniture, helping move the objects to the outdoor scanning place and discussing the possibilities of placement and interaction strategies in virtual reality through image-based hand-drawing. We concluded that placing the artefacts and furniture should balance the original layout and the exhibition requirements. We put more public ones, such as the traditional table and porcelain vases, on the ground floor together with old photos hanging on the surrounding walls, while private ones, such as cabinets and stoves, are on the first floor. We divided all the digital collections into three categories based on their preservation conditions: well-preserved, lost but documented with photos, and lost without documentation but in memory – resulting in three different reconstruction approaches: Photogrammetric modelling, 3D modelling with authentic textures from photos, and 3D modelling with simulated textures. Later on, starting from these collective ideas, we completed model refinement and placement, focusing on adding decorative details and remodelling missing furniture based on AI-enhanced photos.

In November 2024, we completed the first phase of this project – a multimodal, interactive and immersive experience about these cultural legacies. Wearing the headset, people are empowered to roam and explore this two-story building and courtyard through walking and hand-detection-based teleportation. A right-hand palm up and a pinch can easily activate the teleportation function while avoiding misoperation. Wall colliders have been implemented to prevent users from getting stuck during teleportation. In the virtual space, our hands are equipped with colliders and can be detected to touch, push and hold artefacts, as well as grab and distance-grab them through pinching. We incorporated stereo background sounds and views of the old paddy field outside windows, evoking a familiar environment from memory. We added different post-processing effects to the outdoor and indoor spaces to better simulate realistic perceptions of natural lights. In these ways, we created visual, physical and auditory sensories that immerse users. During five participatory tests with family members and other participants, I first introduced the project background and operational instructions for using VR. When they started their experience, I first observed them without interference, followed by follow-up questions about their experience. All of them were new to VR, were surprised and impressed by its level of reality and details, and expressed their willingness to experience more. They particularly enjoyed the hand-based locomotion method and three hand interaction methods with artefacts – they were having lots of fun interacting with artefacts as if in a game. Their actions include grabbing the artefacts closer to look at, moving them by rays (distance-grab), arranging them on tables, stacking one on top of another and taking them to other places. Through the tests, participants gave constructive feedback for improvement, such as the discomfort when getting stuck in walls, some difficulties in posing correct hand gestures for teleportation, and a sense of floating and unrealistic looking outside the door, all of which have been fixed.

In the next phase, we plan to collect more multi-media data on the family stories, such as old photo scannings, video recordings and sound recordings from elders, to enrich the virtual experience and dig into the history and material culture. We aim to develop a more gamified and interactive experience with a narrative focus. Besides, we will optimise hand physics for a more natural and intuitive interaction. These will be done in an iterative and participatory way as we have done. We will seek exhibition opportunities not only in local galleries and museums to raise cultural awareness and resonate with other Ningbo families but also worldwide for cultural exchange and technology exchange. We aim to collaborate more with practitioners and scholars on this topic.