Cross media Narrative of Piano and Digital Vision A Study on the Design of Ice and Snow themed Immersive Concerts
Main Article Content
Abstract
Traditional piano concerts mainly rely on auditory perception, while music and visual images often remain independent and cannot present an integrated ice-and-snow narrative. To address this limitation, this paper first constructs a four-part ice-and-snow narrative script. It then extracts piano pitch, dynamics, and rhythm in real time and maps them to visual parameters of a particle system. Six-channel projection mapping, optical image propagation, and motion-capture technology are used to generate aurora and glacier-crack effects linked to performer gestures. Finally, millisecond-level audio-visual synchronization is achieved through the OSC protocol. Experimental results show that, compared with traditional concerts, the immersive concert increases the immersion score by 18.15 points (70.52 versus 52.37), the narrative comprehension score by 4.10 points (11.23 versus 7.13), and produces a significant increase in physiological response indicators (p<0.001). The findings show that the cross-media narrative strategy improves ice-and-snow expression and audience immersion, with advantages in accurate real-time audio-visual mapping and multi-sensory perception fusion.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
References
L. Bishop and W. Goebl, “Visual information shapes the perception of musical expressiveness,” Psychology of Music, vol. 49, no. 5, pp. 1221–1236, 2021, doi: 10.1177/0305735620927863.
J. London and J. A. Sloboda, “Visual attention in live music performance,” Music Perception, vol. 39, no. 3, pp. 245–261, 2022, doi: 10.1525/mp.2022.39.3.245.
J. K. Vuoskoski and T. Eerola, “The role of visual information in emotional responses to music,” Musicae Scientiae, vol. 25, no. 2, pp. 143–160, 2021, doi: 10.1177/1029864919867342.
R. T. Dean and F. Bailes, “Real-time audio-to-visual mapping in interactive performance,” Organised Sound, vol. 26, no. 2, pp. 184–195, 2021, doi: 10.1017/S1355771821000209.
J. Françoise and N. Schnell, “Augmented piano performance through real-time gesture and audio analysis,” Journal of Mathematics and Music, vol. 15, no. 2, pp. 112–128, 2021, doi: 10.1080/17459737.2021.1890231.
M. Schedel and J. Yoo, “Immersive music: A survey of technologies and aesthetics,” Computer Music Journal, vol. 46, no. 3, pp. 8–22, 2022, doi: 10.1162/comj_a_00635.
Z. Cui, X. Chen, and Y. Wang, “Audio-driven visual effects for live piano concerts based on MIDI mapping,” The Visual Computer, vol. 39, no. 6, pp. 2345–2357, 2023, doi: 10.1007/s00371-022-02478-6.
A. Tanaka and M. Donnarumma, “Immersive concert design using projection mapping and motion capture,” International Journal of Performance Arts and Digital Media, vol. 19, no. 1, pp. 56–73, 2023, doi: 10.1080/14794713.2022.2146875.
S. Lee and K. Lee, “A critical review of visual augmentation in classical music concerts,” Journal of Music Technology & Education, vol. 14, no. 2–3, pp. 145–162, 2021.
H. Frisk and P. Nilsson, “Beyond decoration: The need for narrative integration in audiovisual performance,” Organised Sound, vol. 27, no. 1, pp. 45–56, 2022, doi: 10.1017/S1355771822000083.
V. Zappi and A. McPherson, “The role of real-time visual feedback in piano pedagogy and performance,” The Journal of the Acoustical Society of America, vol. 153, no. 3, pp. 1801–1812, doi: 10.1121/10.0017235.