Graphic notation and synesthesia

To discover potentially innovative techniques to incorporate music filters based on moods/emotions, an expert interview was done with a professional sound artist/composer¹ who referred to the topics of synesthesia and graphic notation.

To consider their potential application in the design of the advanced music discovery tool, a mix of literature study and brainstorming was conducted.

To evaluate the impact of this research, a co-creation session was conducted with a filmmaker.

Graphic notation is the representation of music through the use of images, abstract symbols, graphic elements, illustration, and text. Compared to traditional notation which is linear and rigid, graphic notation is a more expressive way of explaining music. For instance, emotions can be visualised as colours or tempo can be illustrated by various patterns. This could give a different view of the depth of music.²

Example of traditional music notation

Example of graphic notation

Synesthesia is a rare phenomenon which makes people experience a blending of the five senses (sight, smell, touch, hearing, taste) in which the stimulation of one modality simultaneously produces sensation in a different modality. Examples include seeing colours upon hearing music or tasting words and shapes.^{3 4} Research⁵ has shown that music and colour have common emotional qualities and people with or without synesthesia can match colours and songs that contain overlapping emotional qualities. For example, slow melancholic piano music and the colour “blue” can be associated with the feeling of sadness. Such research results can be useful for creating new musical experiences so that “listening” to music can become richer and more vivid by “seeing” and “feeling” it as well.

Results from research about music-color associations.⁴

Approximately both non-synesthetes and synesthetes associate overall warm colours with fast-paced major music and cold colours with slow-paced minor music.

Brainstorming: Since graphic notation and the knowledge of synesthesia can offer such open and richer ways of interpreting and experiencing music, how can this be used to discover music in a more intuitive and emotionally informed way? Similar questions kept coming while learning about graphic notation and synesthesia:

• What if the filmmaker can choose moods for the music by selecting different colours and shapes to provide a search prompt to the tool?

• How can music be visualised in an intuitive way to give a breakdown of the emotional value?

• How can interpreting music be made more accessible? What if instead of majors and minors, crescendos and diminuendos, people could use more of their senses to get lost in the stories the music is telling?

Although particularly interesting, the findings about graphic notation and synesthesia, alongside the brainstorming questions, did not have a direct impact on the design of the advanced music discovery tool. Initially, the intention was to use the findings during the Concept Creation phase by making the tool more delightful and intuitive through the visualisation of colours, shapes and imagery while the music is playing. After a co-creation session with a filmmaker, it was decided that such additions would not significantly change the experience for the users as the main goal of a filmmaker using such a tool is to find music as fast as possible. However, these additions could be added optionally to make the tool stand out from other platforms like Artlist or Musicbed.