Visualizing the Semantic Structure in Classical Music Works

Wing-Yi Chan, Huamin Qu, Wai-Ho Mak

IEEE Transactions on Visualization and Computer Graphics (TVCG), vol. 16, no. 1, pp. 161-173, Jan.-Feb. 2010


Department of Computer Science and Engineering

The Hong Kong Unviersity of Science and Techology (HKUST)


full paper demo on youtube


One of the major obstacles to appreciating classical music is that extensive training is required to understand musical structure and compositional techniques towards comprehending the thoughts behind the musical work. In this paper, we propose an innovative visualization solution to reveal the semantic structure in classical orchestral work, such that users can gain insights into musical structure and appreciate the beauty of music. We formulate the semantic structure into macro-level layer interactions, microlevel theme variations, and macro-micro relationship between themes and layers to abstract the complicated construction of a music composition. The visualization has been applied with success in understanding some classical music work as supported by highly promising user study results with the general audience and very positive feedback from music students and experts, demonstrating its effectiveness in conveying the sophistication and beauty of classical music to novice users with informative and intuitive displays.


fig1 fig2 fig3
Figure 1download Figure 2download Figure 3download
fig4 fig5 fig6
Figure 4download Figure 5download Figue 6download
fig7 fig8 fig9
Figue 7download Figure 8download Figure 9download
fig10a fig10b fig10c
Figue 10adownload Figure 10bdownload Figure 10cdownload
fig11 fig12 fig13
Figure 11download Figure 12download Figure 13download
fig14 fig15 fig16a
Figure 14download Figure 15download Figure 16adownload
fig16b fig16c fig17
Figure 16bdownload Figure 16cdownload Figure 17download
Figure 1

Traditional ways of learning music with a musical score (left) and a technical essay (right).

Figure 2

Color Scheme for layers. Instrument families are represented by their physical hues, except that percussions are assigned an outlier color. Lightness indicates the pitch range of an instrument for intra-family contrast.

Figure 3

Basic type anatomy in typography.

Figure 4

The complete glyph variation board for the musical symbol glyph.

Figure 5

Layer braid for the first movement of Mozart’s Symphony No. 40. Layer threads are plaited according to their functional roles over time. The threads are blended (α = 0.7) to better reveal the braid pattern.

Figure 6

Layer braid prototype. Layer threads interwine to show how layers interact at the macro-level. Thread clusters of various functional groups are seen over time. Control points for interpolating the cello thread are marked.

Figure 7

The string threads are highlighted by adjusting the thread opacity values. The visual clutter in Figure 5 can be reduced.

Figure 8

Deploying some focus+context technique like bifocal display to the long layer braid for better navigation.

Figure 9

Theme fabric prototype. The glyphs are first placed in a 2D plane according to when and where the theme appears at the micro-level. The glyphs of the same themes can then be connected aesthetically in various weaving styles.

Figure 10

Theme fabric styles in textile art. (a) Plain weaving. (b) Butterfly stitch in knitting that bundles yarns. (c) Open strips produced by drop-stitch technique.

Figure 11

Theme fabric with plain style weaving.

Figure 12

Theme fabric in bundled style for the first movement of Mozart’s Symphony No. 40. Each theme occurrence is represented by a musical symbol glyph encoding its variation. Identical glyphs are connected by bundled threads.

Figure 13

Generating bundled style from plain style theme fabric. Vertical threads (warp) are removed and horizontal threads (weft) are bundled, retaining the betweenness centrality of the theme glyphs with a concise layout.

Figure 14

Collapsing a theme fabric by removing duplicated glyphs. Layer information is now encoded by rainbow arc.

Figure 15

Collapsed theme fabric for a segment of the first movement of Beethoven’s Symphony No. 5. Macro-micro relationships are shown by mapping arc heights to layer roles. In this example, no theme occurrence is associated with the dominant or accompany layer roles. Pale reflection of the arcs is added for visual reinforcement.

Figure 16

User study result with the general audience to evaluate the effectiveness of the proposed visualization prototypes in revealing musical structure.

Figure 17

Survey results with music experts on a 5-point scale about the usefulness, effectiveness, and aesthetics of each prototype in visualizing musical structure.