A Multi-modal Platform for Semantic Music Analysis

Visualizing Audio-and Score-Based Tension

Dorien Herremans, Ching-Hua Chuan

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)

Abstract

Musicologists, music cognition scientists and others have long studied music in all of its facets. During the last few decades, research in both score and audio technology has opened the doors for automated, or (in many cases) semi-automated analysis. There remains a big gap, however, between the field of audio (performance) and score-based systems. In this research, we propose a web-based Interactive system for Multi-modal Music Analysis (IMMA), that provides musicologists with an intuitive interface for a joint analysis of performance and score. As an initial use-case, we implemented a tension analysis module in the system. Tension is a semantic characteristic of music that directly shapes the music experience and thus forms a crucial topic for researchers in musicology and music cognition. The module includes methods for calculating tonal tension (from the score) and timbral tension (from the performance). An audio-to-score alignment algorithm based on dynamic time warping was implemented to automate the synchronization between the audio and score analysis. The resulting system was tested on three performances (violin, flute, and guitar) of Paganini's Caprice No. 24 and four piano performances of Beethoven's Moonlight Sonata. We statistically analyzed the results of tonal and timbral tension and found correlations between them. A clustering algorithm was implemented to find segments of music (both within and between performances) with similar shape in their tension curve. These similar segments are visualized in IMMA. By displaying selected audio and score characteristics together with musical score following in sync with the performance playback, IMMA offers a user-friendly intuitive interface to bridge the gap between audio and score analysis.

Original languageEnglish (US)
Title of host publicationProceedings - IEEE 11th International Conference on Semantic Computing, ICSC 2017
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages419-426
Number of pages8
ISBN (Electronic)9781509048960
DOIs
StatePublished - Mar 29 2017
Externally publishedYes
Event11th IEEE International Conference on Semantic Computing, ICSC 2017 - San Diego, United States
Duration: Jan 30 2017Feb 1 2017

Other

Other11th IEEE International Conference on Semantic Computing, ICSC 2017
CountryUnited States
CitySan Diego
Period1/30/172/1/17

Fingerprint

Semantics
Clustering algorithms
Synchronization

Keywords

  • Multimodal system
  • music analysis
  • music representation
  • online interface
  • tension

ASJC Scopus subject areas

  • Computer Science Applications
  • Information Systems
  • Computer Networks and Communications

Cite this

Herremans, D., & Chuan, C-H. (2017). A Multi-modal Platform for Semantic Music Analysis: Visualizing Audio-and Score-Based Tension. In Proceedings - IEEE 11th International Conference on Semantic Computing, ICSC 2017 (pp. 419-426). [7889573] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICSC.2017.49

A Multi-modal Platform for Semantic Music Analysis : Visualizing Audio-and Score-Based Tension. / Herremans, Dorien; Chuan, Ching-Hua.

Proceedings - IEEE 11th International Conference on Semantic Computing, ICSC 2017. Institute of Electrical and Electronics Engineers Inc., 2017. p. 419-426 7889573.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Herremans, D & Chuan, C-H 2017, A Multi-modal Platform for Semantic Music Analysis: Visualizing Audio-and Score-Based Tension. in Proceedings - IEEE 11th International Conference on Semantic Computing, ICSC 2017., 7889573, Institute of Electrical and Electronics Engineers Inc., pp. 419-426, 11th IEEE International Conference on Semantic Computing, ICSC 2017, San Diego, United States, 1/30/17. https://doi.org/10.1109/ICSC.2017.49
Herremans D, Chuan C-H. A Multi-modal Platform for Semantic Music Analysis: Visualizing Audio-and Score-Based Tension. In Proceedings - IEEE 11th International Conference on Semantic Computing, ICSC 2017. Institute of Electrical and Electronics Engineers Inc. 2017. p. 419-426. 7889573 https://doi.org/10.1109/ICSC.2017.49
Herremans, Dorien ; Chuan, Ching-Hua. / A Multi-modal Platform for Semantic Music Analysis : Visualizing Audio-and Score-Based Tension. Proceedings - IEEE 11th International Conference on Semantic Computing, ICSC 2017. Institute of Electrical and Electronics Engineers Inc., 2017. pp. 419-426
@inproceedings{c0f27c0375b049079ab63622ee96096e,
title = "A Multi-modal Platform for Semantic Music Analysis: Visualizing Audio-and Score-Based Tension",
abstract = "Musicologists, music cognition scientists and others have long studied music in all of its facets. During the last few decades, research in both score and audio technology has opened the doors for automated, or (in many cases) semi-automated analysis. There remains a big gap, however, between the field of audio (performance) and score-based systems. In this research, we propose a web-based Interactive system for Multi-modal Music Analysis (IMMA), that provides musicologists with an intuitive interface for a joint analysis of performance and score. As an initial use-case, we implemented a tension analysis module in the system. Tension is a semantic characteristic of music that directly shapes the music experience and thus forms a crucial topic for researchers in musicology and music cognition. The module includes methods for calculating tonal tension (from the score) and timbral tension (from the performance). An audio-to-score alignment algorithm based on dynamic time warping was implemented to automate the synchronization between the audio and score analysis. The resulting system was tested on three performances (violin, flute, and guitar) of Paganini's Caprice No. 24 and four piano performances of Beethoven's Moonlight Sonata. We statistically analyzed the results of tonal and timbral tension and found correlations between them. A clustering algorithm was implemented to find segments of music (both within and between performances) with similar shape in their tension curve. These similar segments are visualized in IMMA. By displaying selected audio and score characteristics together with musical score following in sync with the performance playback, IMMA offers a user-friendly intuitive interface to bridge the gap between audio and score analysis.",
keywords = "Multimodal system, music analysis, music representation, online interface, tension",
author = "Dorien Herremans and Ching-Hua Chuan",
year = "2017",
month = "3",
day = "29",
doi = "10.1109/ICSC.2017.49",
language = "English (US)",
pages = "419--426",
booktitle = "Proceedings - IEEE 11th International Conference on Semantic Computing, ICSC 2017",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - A Multi-modal Platform for Semantic Music Analysis

T2 - Visualizing Audio-and Score-Based Tension

AU - Herremans, Dorien

AU - Chuan, Ching-Hua

PY - 2017/3/29

Y1 - 2017/3/29

N2 - Musicologists, music cognition scientists and others have long studied music in all of its facets. During the last few decades, research in both score and audio technology has opened the doors for automated, or (in many cases) semi-automated analysis. There remains a big gap, however, between the field of audio (performance) and score-based systems. In this research, we propose a web-based Interactive system for Multi-modal Music Analysis (IMMA), that provides musicologists with an intuitive interface for a joint analysis of performance and score. As an initial use-case, we implemented a tension analysis module in the system. Tension is a semantic characteristic of music that directly shapes the music experience and thus forms a crucial topic for researchers in musicology and music cognition. The module includes methods for calculating tonal tension (from the score) and timbral tension (from the performance). An audio-to-score alignment algorithm based on dynamic time warping was implemented to automate the synchronization between the audio and score analysis. The resulting system was tested on three performances (violin, flute, and guitar) of Paganini's Caprice No. 24 and four piano performances of Beethoven's Moonlight Sonata. We statistically analyzed the results of tonal and timbral tension and found correlations between them. A clustering algorithm was implemented to find segments of music (both within and between performances) with similar shape in their tension curve. These similar segments are visualized in IMMA. By displaying selected audio and score characteristics together with musical score following in sync with the performance playback, IMMA offers a user-friendly intuitive interface to bridge the gap between audio and score analysis.

AB - Musicologists, music cognition scientists and others have long studied music in all of its facets. During the last few decades, research in both score and audio technology has opened the doors for automated, or (in many cases) semi-automated analysis. There remains a big gap, however, between the field of audio (performance) and score-based systems. In this research, we propose a web-based Interactive system for Multi-modal Music Analysis (IMMA), that provides musicologists with an intuitive interface for a joint analysis of performance and score. As an initial use-case, we implemented a tension analysis module in the system. Tension is a semantic characteristic of music that directly shapes the music experience and thus forms a crucial topic for researchers in musicology and music cognition. The module includes methods for calculating tonal tension (from the score) and timbral tension (from the performance). An audio-to-score alignment algorithm based on dynamic time warping was implemented to automate the synchronization between the audio and score analysis. The resulting system was tested on three performances (violin, flute, and guitar) of Paganini's Caprice No. 24 and four piano performances of Beethoven's Moonlight Sonata. We statistically analyzed the results of tonal and timbral tension and found correlations between them. A clustering algorithm was implemented to find segments of music (both within and between performances) with similar shape in their tension curve. These similar segments are visualized in IMMA. By displaying selected audio and score characteristics together with musical score following in sync with the performance playback, IMMA offers a user-friendly intuitive interface to bridge the gap between audio and score analysis.

KW - Multimodal system

KW - music analysis

KW - music representation

KW - online interface

KW - tension

UR - http://www.scopus.com/inward/record.url?scp=85018298850&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85018298850&partnerID=8YFLogxK

U2 - 10.1109/ICSC.2017.49

DO - 10.1109/ICSC.2017.49

M3 - Conference contribution

SP - 419

EP - 426

BT - Proceedings - IEEE 11th International Conference on Semantic Computing, ICSC 2017

PB - Institute of Electrical and Electronics Engineers Inc.

ER -