Evaluation of a Piano Learning Support System Focusing on Visualization of Keying Information and Annotation

Kentaro Ueda, Yoshinari Takegawa & Keiji HirataDept. of Systems Information Science

Future University HakodateJapan

{g2115005, yoshi, hirata}@fun.ac.jp

Abstract: It is difficult for beginners to practice the piano efficiently because they are mostly unable to recognize their own weak points and advances in the proficiency of their piano performance. As a result, their motivation for practicing the piano decreases because they cannot improve their skills in spite of their great efforts. In this paper, we propose a piano learning support system that offers comprehensive visualization of the practice information, and an annotation function, to achieve efficient piano learning. We conducted a system evaluation experiment with fourteen subjects over five successive days (30 minutes per day), using the proposed learning system. The results of the experiment reveal that even though the performance time of subjects who were practicing the set piece with the system became shorter, the total number of experiment days was also significantly shorter than that of subjects who were practicing the set piece without any support from the system.

IntroductionPiano players need to master various techniques and skills, such as reading a score, correct keying,

proper fingering, correct rhythm, keeping tempo and dynamics. Therefore, players generally need long-term training. In addition, even if beginners take piano lessons, independent practice is important because the length of time of independent practice is usually longer than that of a lesson. However, when practicing in a place in which no skilled pianists are present, such as their own home, beginners are hardly aware of their weak points and the daily development of their performance. Due to this reason, beginners tend to unconsciously adopt inefficient practice methods, such as the repeated practice of only a specific phrase that they can play smoothly. Unfortunately, beginners often give up because of these reasons.

There are many studies of methods to support piano learners. For example, Piano Tutor (Dannenberg et al., 1990) is an interactive expert system that uses multimedia technology, and has functions such as automatic page-turning based on score-following technology, and creating performance support information. Piano Tutor is based on score, but there are many systems that are based on piano roll (Chow et al., 2013; Raymaekers et al., 2014). PIANO (Rogers et al., 2014), which shows a piano roll using interactive projection, does not require any score-reading skill. The aim of PIANO is to develop visual tracking and rhythmic performance.

It has been suggested that meta-cognition and self-reflection are necessary for long-term and independent learning (Dewey, 1997). Based on this, Suwa considers verbalization of physical skill (Suwa, 2008). However, there are only a few studies of methods to support piano learners with a system which consider these findings. Wikivatoire (Social College of Music) is an application for piano learners. The learners can write down comments on the score and share the comments with other users. Because target users of Wikivatoire are intermediate and advanced-level learners, this system does not have a function providing auxiliary information for writing comments. Therefore, it seems that Wikivatoire is difficult for beginners to use efficiently.

Based on these findings, we built the hypothesis that reflection on learning strategy and meta-cognition are promoted by writing about learning strategy and focusing on the observations and realizations which are obtained during piano practice (Takegawa et al., 2013). Moreover, we have implemented a piano learning system which has the following two functions: a function which visualizes the piano practice situation based on keying information, a function which enables the user to write down their observations regarding the practice.

According to the results of the system evaluation experiment, learners utilizing the system effectively tend to introspect their learning strategy as they reach a higher level. However, due to the small number of

subjects, it is unclear whether or not the system is valid for shortening the required time period for mastery of the piano.

In this study we investigate the learning process of learners who are utilizing the system effectively, by carrying out the same system evaluation experiment with more subjects.

System DesignThe proposed system shows keying information representing the number of instances of correct

keying, mis-keying and performance suspension rate (the number of suspended keying per total number of performances) to support writing up of learning strategies and writing observations on the score.

Visualization of Keying InformationThere are previous systems which focus on visualizing keying information. For example, pianoFORTE

(Smoliar et al., 1995) visualizes keying information such as velocity, tempo, articulation and coincidence of keying timing of right and left hand. We can obtain useful knowledge from pianoFORTE, but this system has not been evaluated adequately.

Similar to pianoFORTE, the proposed system shows keying information representing the number of instances of correct keying, mis-keying and the performance suspension rate, to support writing up of learning strategies and writing observation on the score. There are various indicators of mastery of the piano and each of these indicators can be understood in a different way. If learners are able to become aware of the quality of their practice and judge their achievement degree for the task appropriately by listening to their playing, they can upgrade their learning strategies.

We use the heat map style to visualize keying information. The keying information must be mapped in notes on the score because the target users of the system are beginners. The heat map style is suitable for data mapping and makes it easy for beginners to identify data trends. However, the system visualizes the difference of keying information using a line graph because the heat map style is not suitable for presenting this kind of data (Ferster, 2012).

Annotation on ScoreIn the educational field, the importance of annotation has been insisted on because it makes learners

aware of the sources of their mistakes (Blayney, 2006). In addition, it is believed that annotation increases depth of engagement with learning (Lebow et al., 2004).

We implement an annotation function on the system which enables learners to write observations about their piano practice on the score. This is because it is assumed that observations obtained during practice, though possibly trivial, give learners a new viewpoint for reflection on their piano practice, and complement their learning strategy. The more trivial observations are liable to go unnoticed as time goes by.

Therefore, whenever possible, as soon as learners make a new observation they should immediately annotate the score. However, when learners wish to record their learning strategy as a long sentence, annotation should be carried out after practice. This is because it is assumed that awareness obtained during practice and awareness obtained by considering the state of an entire practice have different factors from each other.

System ImplementationThe input data of the system is the keyboard’s MIDI output, representing pitch data and keying

velocity data. The output data of the system are two graphs: the line graph and the heat map graph. These graphs represent the number of instances of correct keying, mis-keying and the performance suspension rate (the number of suspended keying per total number of performances). We have implemented the system with JavaScript, HTML5, CSS3 under Mac OS X.

Display Function of Keying Information in Heat Map StyleAs shown in Figure 1, the system displays keying information representing the number of instances of

correct keying, mis-keying and performance suspension rate as a heat map created from MIDI data. Each heat

Figure 2: Line graphs showing the difference of keying information on the score

Figure 1: Screenshot of annotations shown on the score in the heat map style

Figure 4: Guide for writing the learning strategyFigure 3: Arrangement of the experimental equipment

map presents the differences of those values as differences in color. In the ascending order of transparency, blue, green and red, these colors represent the numbers of any of those values.

The first heat map from the top in Figure 1 shows the total number of instances of correct keying and mis-keying of all the notes at the given moment. The second heat map shown in Figure 1 shows the total number of instances of keying of the first voice. The third and fourth heat maps show the total number of instances of correct keying of the second and third voice. The fifth heat map shows the total number of instances of mis-keying of all the voices. The sixth shows performance suspension rate. Performance suspension rate is made of Inter-Onset Interval.

Display Function of Keying Information in Line Graph StyleThe line graphs also represent keying information on the score. Whereas the heat maps present the

differences of data values as color differences, the line graphs represent them in the height of the y coordinates of the points at which the lines bend (Fig. 2). The system can superimpose the line graphs to inform learners about the differences of keying information. Figure 2 is an example of superimposed line graphs. In Figure 2, the red, green and blue line graphs show the total number of instances of keying since the beginning of practice, which is obtained at 8, 17 and 30 minutes into practice. Learners can choose the line graphs they want to superimpose using the lower-left pull-down menus. The system can superimpose up to six line graphs to enable learners to understand their own progress more intuitively.

Annotation FunctionLearners can make annotations regarding their awareness of their practice on the score when they type

sentences in the text box (Fig. 1). This function has been implemented to record observations which learners made during practice, and the learner's learning strategies.

System Evaluation ExperimentThere are two methods that we prepared for system evaluation: practicing the piano with the proposed

system, and practicing the piano without any system. We evaluated the validity of the piano practice visualization method and whether the function of annotation on the score raises learner’s awareness of their learning strategies and promotes learner’s meta recognition, by comparing the learning process under each method.

Experimental EquipmentThe practice environment was established using the piano learning support system as explained in

Section 1, and the input data of the system was MIDI output from the piano learning support system. As shown in Figure 3, the piano learning support system uses a projector which is set above the piano and can display correct keying location information along the entire piano keyboard.

Experimental ProcedureFourteen subjects (A to N) took part in this experiment, all of them had no formal piano training and

were not able to read a score. All participants were university students and aged between 18 and 23. We explained to them how to read pitch from notes on the musical score and how to use the system functions.

In this experiment, we used Turkish March (Piano Sonata No. 11 in A major, K. 331: III (W. A. Mozart)) as a set piece. Although all of the subjects had heard this piece, they had never played it before. The subjects practiced the Turkish March from the beginning to bar 17, playing with both hands.

Subjects A to G practiced the training piece for 30 minutes with the system and Subjects H to N practiced the training piece for 30 minutes without using any system. At the time, Subjects A to G input the MIDI data into the system at arbitrary times and noted down their observations on the score with keying information visualized by the system. After the practice, all of the subjects took an achievement evaluation test. In this test, we instructed the subjects to play the training piece without keying errors, and in natural tempo, within five minutes. We permitted them to skip forward to the next note when they did not understand correct keying positions, but we prohibited them from replaying notes.

After the achievement evaluation test, Subjects A to G input the MIDI data from the past 30 minutes into the system and wrote their learning strategies in accordance with the guide shown in Figure 4. These data were saved in HTML format as a practice portfolio. Subjects H to N followed the same procedures, writing by hand, and the experimenter converted their data into HTML format with the system, and saved them as a practice portfolio. Each of the subjects repeated the series of procedures until mis-keying was eliminated on the achievement evaluation test.

In addition, we permitted each of the subjects to browse their own practice portfolio but prohibited them from browsing the practice portfolios of other subjects. This is because if learners were to browse other learner’s practice portfolios, their learning strategy would have various influences and as a result be difficult for the experimenter to analyze.

Table 2: Results of Subjects H to N who were practicing the set piece without the system

Table 1: Results of Subjects A to G who were practicing the set piece with the system

　 # of experiment

days

# of MIDI input times

Keying time per practice

(minutes)Subject A 3 11 10.30Subject B 3 3 5.40Subject C 3 6 4.84Subject D 5 9 6.49Subject E 5 5 8.06Subject F 6 9 2.40Subject G 9 27 6.10Average 4.86 10.01 6.22Standard Deviation 2.03 7.39 2.31

　 # of experiment

days

Keying time per

practice (minutes)

Subject H 5 6.55Subject I 5 12.50Subject J 6 11.79Subject K 9 6.96Subject L 10 9.31Subject M 11 6.46Subject N 11 16.23Average 9.97 8.71Standard Deviation 2.19 3.43

Figure 7: Extract from practice portfolio of Subject G on day 3

Figure 5: Extract from practice portfolio of Subject F on day 3

Figure 6: Extract from practice portfolio of Subject K on day 9

Results of Experiment and DiscussionTable 1 shows the total number of experiment days, number of MIDI input times, and keying time per

practice of each practice of Subjects A to G, who were practicing with the system. Table 2 shows the total number of experiment days and keying time per practice of each practice of Subjects H to N, who were practicing without the system.

Firstly, the total keying time per single practice of Subjects A to G was significantly shorter than that of Subjects H to N (t (12) =2.22, p=.046<.05). It seems that Subjects A to G spent time browsing their practice portfolios and writing annotations. Although the performance time of Subjects A to G became shorter because of this, the total number of experiment days of Subjects A to G was also significantly lower than that of Subjects H to N (t (12) =3.17, p=.008<.01). It is thus assumed that the degree of learning effect per practice time is increased by browsing practice portfolios and writing annotations.

Secondly, Subjects A to G tended to write their learning strategies with reference to their annotation written during the practice. For example, in the practice portfolio of Subject F on day 3 (Fig. 5), Subject F mentions the difficulty of fingering and disturbance of rhythm on bar 10 to bar 15, and this awareness was reflected in the learning strategy.

Thirdly, in the practice portfolios of Subjects H to N, there were many misidentifications of mis-keying on the achievement evaluation test, as shown in the practice portfolio of Subject K on day 9 (Fig. 6). It seems that Subjects H to N were not able to refer to keying information because of the conditions of the experiment. On the other hand, in the practice portfolio of Subject G on day 3 (Fig. 7 (a)), he mentioned that there was more mis-keying in the performance on that day, but he eventually realized that it was a misunderstanding, when he was writing his learning strategies after practice. In addition, as shown in the practice portfolio of Subject G on day 3 (Fig. 7 (b)), most of Subjects A to G saw the difference of the number of instances of mis-keying by using the system on that day. As shown in Table 1, Subjects A to G looked at the visualized keying information more than once per day. This trend was notable in Subject G as well as the other learners permitted to use the system. Therefore, it seems that Subjects A to G were able to practice the set piece while being conscious of their mis-keying, and the state of the entire practice session, by using the system. This led to a reduction in the number of days required to attain a high level of piano performance.

ConclusionsWe evaluated the proposed learning system, focusing on the visualization of keying information and

annotation. The system shows keying information representing the number of instances of correct keying, the number of instances of mis-keying, and the performance-suspension rate, to support the creation of learning strategies. The system also has an annotation function which enables learners to write down observations about their piano practice on the score. The results of the system evaluation experiment revealed that even though the performance time of subjects who were practicing the set piece with the system became shorter, the total number of experiment days was also significantly lower than that of subjects who were practicing the set piece without the system.

In the experiment, we permitted each of the subjects to browse their own practice portfolio but prohibited them from browsing the practice portfolios of other subjects. This is because if learners were to browse the other learner’s practice portfolios, their learning strategy would have various influences and as a result be difficult for the experimenter to analyze. Therefore, future work will involve the creation of a practice environment which enables learners to share their practice portfolios with each other. If the environment is created, learners who are fixed on a certain learning strategy could get a new perspective on their piano learning.

References

Blayney, P. (2006). Interactive Learning with Adaptive Annotation: An Investigation of the Outcomes, In Pearson, E., & Bohman, P. (Eds.), Proceedings of EdMedia: World Conference on Educational Media and Technology 2006, Association for the Advancement of Computing in Education (AACE), pp. 2276-2281.

Chow, J., Feng, H., Amor, R., & Wünsche, B. C. (2013). Music Education using Augmented Reality with a Head Mounted Display, Proceedings of the Fourteenth Australasian User Interface Conference (AUIC2013), Adelaide, Australia, pp. 73-79.

Dannenberg, R. B., Sanchez, M., Joseph, A., Capell, P., Joseph, R., & Saul, R. (1990). A Computer-Based Multi-Media Tutor for Beginning Piano Students, Journal of New Music Research, 19 (2-3), pp. 155-173.

Dewey, J. (1997). Experience and Education, Free Press.

Ferster, B. (2012). Interactive Visualization, The MIT Press, pp.213-220.

Lebow, D., Lick, D., & Hartman, H. (2004). Interactive Annotation for Teaching and Learning, In Ferdig, R., Crawford, C., Carlsen, R., Davis, N., Price, J., Weber, R., & Willis, D. (Eds.), Proceedings of Society for Information Technology and Teacher Education International Conference 2004, Association for the Advancement of Computing in Education (AACE), pp. 1781-1786.

Raymaekers, L., Vermeulen, J., Luyten, K., & Coninx, K. (2014), Game of Tones: Learning to Play Songs on a Piano Using Projected Instructions and Games, In Proc. CHIEA ’14, ACM (2014), pp. 411-414.

Rogers, K., Röhlig, A., Weing, M., Gugenheimer, J., Könings, B., Klepsch, M., Schaub, F., Rukzio, E., Seufert, T., & Weber, M. (2014). PIANO: Faster Piano Learning with Interactive Projection, Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces, pp. 149-158.

Suwa, M. (2008). A Cognitive Model of Acquiring Embodied Expertise Through Meta-Cognitive Verbalization, Information and Media Technologies, Vol. 3, No. 2, pp. 399-408.

Smoliar, S., Waterworth, J., & Kellock, P. (1995). pianoFORTE: A System for Piano Education Beyond Notation Literacy, Proceedings of the 3rd ACM International Conference on Multimedia, pp. 457-465.

Takegawa, Y., E. Tayanagi, M. Tsubakimoto and K. Hirata (2013). Evaluation of a Piano Learning Support System Focusing on the Learning Process, Proceedings of EdMedia 2013 (25th World Conference on Educational Media and Technology), pp.2306-2314.

Wikivatoire: Social college of music, http://wikivatoire.org/.