Enhancing Performance Music Games in Education through the IEEE 1599 Standard

Veiw figure View Figure

Figure 1, an updated version of the one in ^[13], presents a graphical comparison of the principal music games. This analysis reveals that some performance games privilege expressiveness, others technique, and finally meta-instruments try to mix the two approaches, being closer to the real music instruments.

Nowadays, research activities in this field aim at improving both control and meta-control processes, in order to allow an expressive performance without asking the game player to be a virtuoso, but addressing instrumental technical skills too. On one hand, performance-oriented meta-instruments such as the Metapiano can provide a simplification of some performance aspects, and on the other hand they can even extend expressive potentialities through ad-hoc supports and standards. In our opinion a new learning paradigm for music based on advanced performance games is extremely innovative since it could remove any starting barrier between the player and the instrument.

In order to further develop the concept of music meta-instrument, in the following we will discuss a framework that embeds meta-instrument notation into the IEEE 1599 format. IEEE 1599 is an XML-based international standard aiming at the representation of music in all its aspects. Before providing details about our proposal and showing a case study, the key concepts of IEEE 1599 will be recalled.

3. Key Concepts of the IEEE 1599 Standard

IEEE 1599 is a standard internationally recognized by the IEEE, sponsored by the Computer Society Standards Activity Board and designed by the Technical Committee on Computer Generated Music. IEEE 1599 adopts XML (eXtensible Markup Language) in order to describe a music piece in all its aspects ^[17]. This challenging goal is achieved through the following concepts:

• A multi-layer layout for content description;

• A common data structure to index and refer to different descriptions of the same music entity, known as the spine;

• Full synchronization among different descriptions of the same music entity.

All the mentioned issues will be briefly discussed in the following subsections. For further details, please refer to ^[18].

3.1. Multi-layer Structure

Music description is a complex task which can embrace a number of different facets. For example, when people address a music work, they usually identify it through commonly-accepted metadata, such as its title, authors, performers, instrumental ensemble, etc.

As regards the visual aspects of music, a piece can be described through notation, i.e. any system that represents scores through the use of written symbols. Depending on cultures, traditions and geographical areas, many methods are in use for notated music, ranging from modern staff notation ^[19] to neumes ^[20], from asian solmization ^[21] to Indian sargam ^[22], from lute tablature ^[23] to African dance drumming ^[24]. Moreover, the same music piece – namely the same set of music symbols – can be represented with different page layouts (e.g. different printed editions), or through different kinds of notation (colored music notation, Braille, etc.).

Audio contents represent another kind of description for music. Once again, the same music piece can be performed in many ways, as regards interpretation, score version, ensemble, etc. Depending on the music genre, this fact presents a number of implications: in classical music, even if score symbols are (usually) common to any performance, players adopt their own interpretative model; in jazz, improvisation is a key aspect, so that each performance of a jazz standard becomes unique; in pop/rock music, acoustic versions, covers, etc. are quite common; and so on.

The previous examples intuitively show the richness implied by the locution “music description”. The number of heterogeneous ways to describe music is potentially high, involving data, metadata, and structures.

A multilayer view is fit for treating complex information entities by keeping contents properly organized within a unique framework. In fact, music information is made of heterogeneous facets, whose degree of abstraction may range from a purely logical definition in terms of symbols to the physical description of audio signals. Adopting the IEEE 1599 terminology, music information can be organized into the following layers: General, Logic, Structural, Notational, Performance, and Audio. Figure 2 shows the typical 6-layer structure of an IEEE 1599 document, together with an example of contents. Please note that the multi-layer approach for music has been discussed in earlier works, such as ^[25], ^[26] and ^[27].

It is worth mentioning that not all layers are necessarily present for a given music piece. Of course, the higher is the number of available layers, the richer is the music description. For instance, many jazz pieces have no full score and a given performance could only be transcribed a posteriori. Similarly, some folk songs or popular tunes do not present commonly-accepted metadata or instrumentation. Besides, no hierarchical relationship is implied among layers, since they show different facets of the same music piece.

The goal of layers is providing ad-hoc descriptions for the set of music events that constitute the score. Layers can contain 0, 1 or n descriptions of each music event. In order to support music coming from different traditions, cultures, geographical areas, historical periods it is necessary to adopt a format both powerful and flexible. In this sense, a key idea is defining the concept of music event in a loose way, as explained in the following.

Figure 2. The IEEE 1599 6-layer structure

Download as

Veiw figure View Figure

3.2. The Spine

The spine is the main data structure in an IEEE 1599 document, and its presence is mandatory for a file to be valid. Its purpose is listing music events and identifying them univocally by assigning unique labels.

In the official documentation, the concept of music event is left intentionally vague. In general terms, a music event can be defined as the occurrence of something that is considered important by the author of the encoding. For instance all the notes and rests of a score could be defined as music events: each symbol will be univocally identified and inserted into the spine. However, there are many other possibilities. For instance, in jazz music the only available form of score could be a harmonic grid, and music events could simply refer to the occurrence of chords. Similarly, in a musicological context, events of interest could be the occurrence of the fugue subject or the beginning of a dodecaphonic series. Please note that, thanks to the flexible definition of music event, neither traditional score notation nor a complete encoding of the piece are required to generate a valid IEEE 1599 document.

The spine is a sort of glue among IEEE 1599 layers, since the events listed and described inside different layers have to refer to the common event list provided by the spine itself. The name of this data structure recalls its function of “backbone” for the music work. This concept was first used in 1975 by D.A. Gomberg ^[28], who based a system for electronic music publishing on a similar structure, also called spine.

The spine simply lists events without defining them. For example, in a piece made of n music events – whatever they are – the spine would list n entries without defining them from any point of view. The multiple and heterogeneous descriptions of events are demanded to the other layers.

3.3. Synchronization among IEEE 1599 Layers

If we consider the aforementioned 6-layer layout, the idea of heterogeneity seems to be merely related to the number of supported media types. But in addition each layer could contain many instances sharing the same kind of description or media type. For example, the Audio layer could contain different audio tracks, and the Structural layer could provide many different analyses for the same piece.

Each music event – univocally identified by the spine – can be described in different layers (e.g. the graphical aspect of a note and its sound), as well as multiple times within a single layer (e.g. multiple music performances of the same event). Consequently, in the multilayer environment provided by IEEE 1599, one can recognize two synchronization modes:

• Inter-layer synchronization – It takes place among contents described in different layers, which have by definition heterogeneous data types. Applications involving multi-media and multi-modal fruition, such as score following, karaoke, didactic products, and multimedia presentations, can be realized thanks to this kind of synchronization;

• Intra-layer synchronization – It occurs among the contents of a single layer. Each layer contains - by definition - homogeneous information. Thanks to this feature, one can jump from an instance to another (e.g. from an audio performance to another) in real time, without losing synchronization.

By using the aforementioned kinds of synchronization, it is possible to design and implement advanced frameworks for music. In this context, the most relevant aspect is the possibility to integrate and synchronize heterogeneous kinds of description, including any form of meta-instrument notation. In other words, a particular music encoding – expressively conceived for a meta-instrument – can interact with other kinds of notation and other media materials. The relevant consequences on music performance games and music education will become clear soon.

4. The Proposed Educational Framework

The goal of our research is fostering music education through the design and implementation of performance games based on the concept of meta-instrument and on the IEEE 1599 standard. On one hand, meta-instruments are currently the most expressive and challenging surrogates of real music instruments; on the other hand, the adoption of the IEEE 1599 standard allows countless didactic applications ^{[29, 30]}.

In our approach, two activities can be clearly distinguished: i) music encoding, and ii) music performance. Usually they are asynchronous, and the latter step follows in time the former one. For our goals music encoding implies the production of a “rich” IEEE 1599 document, namely a single XML file containing both the spine and the meta-instrument notation, mutually linked. Encoding music in a proper way during a live performance would be a hard task, even if theoretically feasible ^[31].

Needless to say, an IEEE 1599 document can contain much more, as explained in Section 3. For instance, the Audio layer could host a number of pre-recorded audio tracks referring to other performances of the piece, or conceived as a background for the current performance. Similarly, the Notational layer could host evocative graphics together with a traditional score in common Western notation.

With respect to other IEEE 1599 applications oriented to music education, here the novelty is the presence of meta-instrument notation. Usually meta-instruments require two kinds of input: performer’s gesture and a custom notation to drive the system. Depending on the technical characteristics of the framework, notation may vary significantly. As an example, Figure 3 shows a Metapiano score excerpt containing basic symbolic information, i.e. notes, rests, and a few articulation signs.

Figure 3. An example of plain-text notation for J. Haury's Metapiano. The score contains voice, pitch and velocity encoding, together with basic information on articulations

Download as

Veiw figure View Figure

It is worth underlining that the information contained in a meta-instrument score is potentially redundant with the contents of the IEEE 1599 Logic layer, and actually the specifications of encoding rules make an automatic conversion among formats possible. Software tools and plug-ins have been developed to compile the Logic layer starting from commonly adopted formats (e.g. MusicXML and MIDI) as well as score editing software (e.g. MuseScore, MakeMusic Finale and Sibelius). Similarly, computer applications could be implemented to support custom meta-instrument scores, too.

Since IEEE 1599 supports any representation of score symbols, also new notation for music meta-instruments can be embedded and synchronized with all the other contents.

As regards the second phase - i.e. music performance - before the design of this framework, two independent modules were available:

• An IEEE 1599 viewer, namely an environment oriented to a multi-layer and synchronized musical experience. This software is able to present simultaneously information contents from multiple layers, allowing the user to enjoy them together and to choose the material to bring to front. The user is active in the choice of current materials (scores, audio tracks, video clips, etc.), and he/she can use standard navigation controls (start, stop, pause, change current position); however, from the performance point of view, the user can only experience already prepared materials;

• A meta-instrument parser, where a symbolic score is loaded and the user can interact through the interface of the musical instrument. The parser is not standard, since it is customized for the specific meta-instrument. Besides, it usually gets input only from the external controller and from a digital score representation. Consequently, other interactions with related materials are demanded to a posteriori processing of its output, which limits the expressive possibilities of the framework.

The idea is creating a unique framework where the two modules can be integrated, in order to create a performance game which takes advantage by both the approaches.

These two environments could be (and actually have been) implemented under different HW/SW architectures. Since IEEE 1599 is a standard, implementation details are not relevant for our proposal. For instance, IEEE 1599 players have been developed for multi-platform off-line fruition as well as embedded in Web portals. Similarly, there are some meta-instruments entirely implemented via software and others based on the communication between Arduino and Max/Msp environment. An example will be discussed in the final section of this paper as a case study. The music meta-instrument can be any hardware or software device capable of sending computer-interpretable messages: MIDI controllers, external peripherals such as computer keyboards, graphical interfaces, and so on.

Figure 4. Flow chart of the proposed framework

Download as

Veiw figure View Figure

In the proposed framework, the role of the parser is interpreting both the IEEE 1599 and the controller input, producing a sequence of commands to drive the player. One of the key roles is disambiguating synchronization. As mentioned before, most contents in an IEEE 1599 document have intrinsic timing information, such as all audio and video tracks. On the contrary, in this context metronome could be one of the parameters provided by the human player; in this case the parser has to match human gesture with meta-instrument notation, and other contents must be consequently timed.

Figure 4 illustrates the proposed framework. The upper half corresponds to music encoding, whereas the lower half represents music performance. The two approaches are matched by the parser, whose purpose is combining the information coming from the physical interface of the meta-instrument with the data contained in the IEEE 1599 document, now including also the meta-instrument score.

5. Case Study: the Metapiano

The Metapiano is a musical meta-instrument made of only nine piano keys that can be played with a few fingers, or even with only one finger ^[32]. This music meta-instrument can produce music and sound by playing a limited number of keys. The musician can instantly interpret music with his/her own style by applying his/her rhythm, tempo, articulation, accent, dynamic and agogic phrasing. Figure shows the Metapiano’s simplified interface.

The Metapiano notation stores only the notes to be performed by the player, encoding them according to the pianotechnie rules ^[33]. Thanks to the integration with IEEE 1599, now it is possible to provide a synchronized accompaniment too.

In this kind of meta-instrument pitch information is received from the score, and consequently reconstructed at parser level. The 9-key interface is provided only to allow more effective gestures. For instance, quick sequences of notes are easier to be obtained using many fingers, independently from the melodic contour. Similarly, a legato effect can be obtained only using at least two keys.

Experiments have shown that such an interface is extremely intuitive for complete beginners and inexpert players, who are not used to associate keys to sounds ^[34]. On the contrary, for skilled piano players this abstraction is harder to be managed. However, the latter category is not the typical recipient of a music performance game initiative.

In order to apply the IEEE 1599-based framework to the Metapiano case, a meta-instrument oriented language layer has been designed. This language is based on the syntactic and semantic encoding defined by Jean Haury. Starting from an XML encoding of the score, an integrated IEEE 1599 document is generated to feed the parser controlled by a musical meta-instrument interface. In this way, a 3-level hierarchy of music representation has been realized:

• A low-complexity encoding for complete beginners, namely people unable to read music scores and to play any music instrument;

• A medium-complexity encoding for learners, namely people who can read scores but with no instrumental skill;

• Finally, a high-level encoding for musicians, namely people interested in improving their music abilities and experiencing new kinds of music interfaces.

This approach can be easily ported to any other music meta-instrument, thanks to the extensibility of IEEE 1599 format. Currently, we are working on the software and hardware prototype of a meta-guitar, allowing the application of specific gestures and the control of typical guitar parameters. To this end, ad hoc custom notation is now under development.

Figure 5. Jean Haury playing his 9-key Metapiano

Download as

Veiw figure View Figure