A hierarchical sonification framework based on convolutional neural network modeling of musical genre

Shijia Geng, Gang Ren, Mitsunori Ogihara

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Convolutional neural networks have satisfactory discriminative performances for various music-related tasks. However, the models are implemented as "black boxes" and thus their processed representations are non-Transparent for manual interactions. In this paper, a hierarchical sonification framework with a musical genre modeling module and a sample-level sonification module has been implemented for aural interaction. The modeling module trains a convolutional neural network from musical signal segments with genre labels. Then the sonification module performs sample-level modification according to each convolutional layer, where lower sonification levels produce auralized pulses and higher sonification levels produce audio signals similar to the input musical signal. The usage of the proposed sonification framework is demonstrated using a musical stylistic morphing example.

Original languageEnglish (US)
Title of host publication141st Audio Engineering Society International Convention 2016, AES 2016
PublisherAudio Engineering Society
StatePublished - 2016
Event141st Audio Engineering Society International Convention 2016, AES 2016 - Los Angeles, United States
Duration: Sep 29 2016Oct 2 2016

Other

Other141st Audio Engineering Society International Convention 2016, AES 2016
CountryUnited States
CityLos Angeles
Period9/29/1610/2/16

Fingerprint

Sonification
Network Modeling
modules
Neural Networks
Neural networks
Labels
Module
audio signals
music
boxes
Morphing
interactions
Black Box
Music
Interaction
Modeling
Framework
pulses

ASJC Scopus subject areas

  • Electrical and Electronic Engineering
  • Modeling and Simulation
  • Acoustics and Ultrasonics

Cite this

Geng, S., Ren, G., & Ogihara, M. (2016). A hierarchical sonification framework based on convolutional neural network modeling of musical genre. In 141st Audio Engineering Society International Convention 2016, AES 2016 Audio Engineering Society.

A hierarchical sonification framework based on convolutional neural network modeling of musical genre. / Geng, Shijia; Ren, Gang; Ogihara, Mitsunori.

141st Audio Engineering Society International Convention 2016, AES 2016. Audio Engineering Society, 2016.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Geng, S, Ren, G & Ogihara, M 2016, A hierarchical sonification framework based on convolutional neural network modeling of musical genre. in 141st Audio Engineering Society International Convention 2016, AES 2016. Audio Engineering Society, 141st Audio Engineering Society International Convention 2016, AES 2016, Los Angeles, United States, 9/29/16.
Geng S, Ren G, Ogihara M. A hierarchical sonification framework based on convolutional neural network modeling of musical genre. In 141st Audio Engineering Society International Convention 2016, AES 2016. Audio Engineering Society. 2016
Geng, Shijia ; Ren, Gang ; Ogihara, Mitsunori. / A hierarchical sonification framework based on convolutional neural network modeling of musical genre. 141st Audio Engineering Society International Convention 2016, AES 2016. Audio Engineering Society, 2016.
@inproceedings{ee478e1b56cd403398b0785aa01ee272,
title = "A hierarchical sonification framework based on convolutional neural network modeling of musical genre",
abstract = "Convolutional neural networks have satisfactory discriminative performances for various music-related tasks. However, the models are implemented as {"}black boxes{"} and thus their processed representations are non-Transparent for manual interactions. In this paper, a hierarchical sonification framework with a musical genre modeling module and a sample-level sonification module has been implemented for aural interaction. The modeling module trains a convolutional neural network from musical signal segments with genre labels. Then the sonification module performs sample-level modification according to each convolutional layer, where lower sonification levels produce auralized pulses and higher sonification levels produce audio signals similar to the input musical signal. The usage of the proposed sonification framework is demonstrated using a musical stylistic morphing example.",
author = "Shijia Geng and Gang Ren and Mitsunori Ogihara",
year = "2016",
language = "English (US)",
booktitle = "141st Audio Engineering Society International Convention 2016, AES 2016",
publisher = "Audio Engineering Society",

}

TY - GEN

T1 - A hierarchical sonification framework based on convolutional neural network modeling of musical genre

AU - Geng, Shijia

AU - Ren, Gang

AU - Ogihara, Mitsunori

PY - 2016

Y1 - 2016

N2 - Convolutional neural networks have satisfactory discriminative performances for various music-related tasks. However, the models are implemented as "black boxes" and thus their processed representations are non-Transparent for manual interactions. In this paper, a hierarchical sonification framework with a musical genre modeling module and a sample-level sonification module has been implemented for aural interaction. The modeling module trains a convolutional neural network from musical signal segments with genre labels. Then the sonification module performs sample-level modification according to each convolutional layer, where lower sonification levels produce auralized pulses and higher sonification levels produce audio signals similar to the input musical signal. The usage of the proposed sonification framework is demonstrated using a musical stylistic morphing example.

AB - Convolutional neural networks have satisfactory discriminative performances for various music-related tasks. However, the models are implemented as "black boxes" and thus their processed representations are non-Transparent for manual interactions. In this paper, a hierarchical sonification framework with a musical genre modeling module and a sample-level sonification module has been implemented for aural interaction. The modeling module trains a convolutional neural network from musical signal segments with genre labels. Then the sonification module performs sample-level modification according to each convolutional layer, where lower sonification levels produce auralized pulses and higher sonification levels produce audio signals similar to the input musical signal. The usage of the proposed sonification framework is demonstrated using a musical stylistic morphing example.

UR - http://www.scopus.com/inward/record.url?scp=85010818625&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85010818625&partnerID=8YFLogxK

M3 - Conference contribution

BT - 141st Audio Engineering Society International Convention 2016, AES 2016

PB - Audio Engineering Society

ER -