Polytechnic Institute of New York University

Department of Mechanical Engineering

Final Project Report, Mechatronics - ME5643

Air Piano

Professor

Sang-Hoon Lee

Students

Lucas Gregolin Dias - 0513304

Fabio Mantelli - 0513305

1

Table of Contents

Introduction.................................................................................................................................................................................. 3

Methodology - Hardware........................................................................................................................................................ 4

Microcontroller Module......................................................................................................................................................4

Keyboard Module...................................................................................................................................................................5

Control Module....................................................................................................................................................................... 6

Prototype Costs.......................................................................................................................................................................8

Methodology - Software...........................................................................................................................................................9

MIDI Protocol.......................................................................................................................................................................... 9

MIDI Messages........................................................................................................................................................................ 9

MIDI Digital Audio Workstation (DAW)...................................................................................................................11

Code........................................................................................................................................................................................... 11

Results...........................................................................................................................................................................................15

Conclusion...................................................................................................................................................................................16

Bibliography...............................................................................................................................................................................17

2

Introduction

Since it was developed, the MIDI (Musical Instrument Digital Interface [1]) protocol

has revolutionized the concept of making music, quickly becoming one of the main pillars

of music production and recording nowadays. The characteristics of this protocol, allied

with new and more accessible technologies, also changed the paradigm of digital

instruments. Previously, they were used exclusively to emulate analogic instruments (such

as pianos and drum kits). Now they are acquiring their own characteristics, in different and

more creative ways.

Another technology that has been getting a lot of attention lately is gesture

recognition. Different applications have been using the technology as a way of interacting

more naturally with the end-user.

Immersed in this context we find the “Air Piano” project. It is a musical instrument

that can sense finger gestures on top of an array of sensors to trigger notes and send them

to MIDI devices. It also has a control board that changes the volume and effects of the sent

notes depending on its distance to the ground and orientation.

This device was inspired by some unusual musical instruments which use sensor

measurements to produce tones, like the Theremin [2], the Martenot [3], and other more

recent ones [8]. Its gesture sensing is inspired on the recent advent of hands-free interfaces

like the Kinect [4], Leap [5] and Oculus Rift [6].

3

Methodology - Hardware

The device built is composed by a microcontroller, photoresistors, an accelerometer

and an ultrasonic distance sensor. For the structure, we used cardboard and breadboards.

Microcontroller Module

Figure 1 - Microcontroller Module

For the prototype’s microcontroller module a breadboard was used as the

mechanical structure. The microcontroller used was the Arduino Micro [7]. We chose this

over the suggested BASIC Stamp 2 microcontroller due to the necessity of 12 analog ports.

The specifications of this microcontroller are as indicated in Table 1.

Figure 2 - Arduino Micro

4

Microcontroller ATmega32u4

Operating Voltage 5V

Input Voltage (recommended) 7-12V

Input Voltage (limits) 6-20V

Digital I/O Pins 20

PWM Channels 7

Analog Input Channels 12

DC Current per I/O Pin 40 mA

DC Current for 3.3V Pin 50 mA

Flash Memory 32 KB (ATmega32u4), 4 KB used by bootloader

SRAM 2.5 KB (ATmega32u4)

EEPROM 1 KB (ATmega32u4)

Clock Speed 16 MHz

Table 1

Keyboard Module

Figure 3 - keyboard

The prototype keyboard was built as a cardboard box-like structure with 11

dividers, resulting in 12 holes corresponding to the 12 notes of one octave of the musical

scale (C to B), displayed on the subtitles. Plastic tape was used to stick the parts together,

and the wires were soldered together. Each hole contains a photoresistor, which was

connected to the microcontroller as portrayed on Figure 4.

5

Figure 4 - photoresistor circuit

Figure 5 - photoresistor

A photoresistor resistance decreases with increasing incident light intensity. So with

this circuit, the analog pin measurement will be inversely proportional to the amount of

light shed on it. The user will use his/her fingers to shade the photoresistor, changing the

state on the analog port and triggering the notes.

Control Module

Figure 6 - control module

6

The control module prototype also has a breadboard structure, and it is basically

composed by an ultrasonic distance sensor, which is used to detect its distance to the

ground, and an accelerometer, which measures the board spatial orientation. The

accelerometer used 2 microcontroller ports, one for each axis, and the ultrasonic sensor

used only one port. The other wires seen in Figure 6 are power supply and ground wires

for the digital sensors.

The accelerometer used is the MX2125 by parallax (Figure 7). It measures on a

range of ±3g on each axis and sends a PWM signal through its output ports with the width

modulated by the acceleration readings. The components of the g force in each direction

allow us to measure tilt in relation of two orthogonal axes. The electrical and mechanical

specifications of the MX2125 design met our requirements.

Figure 7 - accelerometer

The ultrasonic sensor used in this prototype was the PING by parallax. It is a simple

pulse in/pulse out device that measures distance on a range of 2 cm to 3 m, without

contact. Its convenience also resides in the fact that it doesn’t require special light

conditions and that it uses only one I/O digital pin port on the microprocessor. Attached on

the control module, the sensor can be used by pointing it to a flat surface and bringing the

module nearer or farther away from it.

7

Figure 8 - ultrasonic sensor

Prototype Costs

We estimated the components’ cost as shown in Table 2.

Component Quantity Unit Price

Arduino Micro 1 $22.95

MX2125 Accelerometer 1 $29.00

PING Ultrasonic Sensor 1 $25.00

Photoresistor 12 $1.25

1 k resistorΩ 12 $0.80

Breadboard 2 $5.00

Assorted wires ~35 $0.30

Cardboard - $0.00

Total cost $122.05Table 2

These costs can be reduced by mass production, by embedding the circuits in

printed circuit boards, cutting the breadboard and wires’ cost, and by using cheaper

accelerometer and ultrasonic sensors.

8

Methodology - Software

MIDI Protocol

Before the creation of MIDI protocol, many other proprietary standards were used

to interface between different digital instruments. They allowed the communication and

the exchange of messages between two instruments of the same brand, but failed to do that

when is dealing with devices that used different protocols.

The MIDI protocol was created in 1983 to standardize the interface between digital

instruments. It created requirements for the hardware and communication. The relevant

aspects of the protocol to the project are the message standard and communication

standard. MIDI messages don’t carry information about the sound being played, but they

tell which, when and for how long notes are being played, analog to music notation. Some

of the advantages of MIDI protocol over the previous standards are the fact that it is open-

source, simple, fast and lightweight (small messages). [1]

MIDI Messages

MIDI messages are delivered through serial port, and are 8 bit (1 byte) long. There

are two types of message: status messages and data messages.

The MSB (most significant bit) of the status messages is 1. The next 3 bits are the

command and the last 4 is the channel the message is being sent to. Depending on the

status message, 0 to 3 data bytes are expected to follow it.

The MSB of the data messages is 0. The next 7 bits are the data value for the

message. You can send 128 different values using one data message.

Figure 9

9

The two status messages used by the system are the “note on” and the “control

change” message. They are sent on the channel 1 (for example) and you can see their

structure in Table 3.

Status D7----D0

Data Byte(s)D7----D0

Description

1001nnnn 0kkkkkkk 0vvvvvvv

Note On event. This message is sent when a note is depressed (start). (kkkkkkk) is the key (note) number. (vvvvvvv) is the velocity.

1011nnnn 0ccccccc 0vvvvvvv

Control Change. This message is sent when a controller value changes. Controllers include devices such as pedals and levers. Controller numbers 120-127 are reserved as "Channel Mode Messages" (below). (ccccccc) is the controller number (0-119). (vvvvvvv) is the controller value (0-127).

Table 3

10

MIDI Digital Audio Workstation (DAW)

Since the MIDI messages don’t carry any sound information, in order to test and

play music, a MIDI instrument is needed. For this project, the Hairless MIDI to Serial Bridge

[10] computer driver was used to interpret the MIDI messages received through the serial

port and send it to Reason [11], a DAW, for the messages to be played in one of its MIDI

software instruments.

Code

The Arduino Micro is programmed using the C language.

The photoresistor ports are the 12 analog ports of the microcontroller. The initial values for the sensor measurements are zero.

// Photoresistor portsint light[] = {A6, A7, A8, A9, A10, A11, A5, A4, A3, A2, A1, A0};int lightValue[] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};

The MIDI protocol uses natural numbers from 0 to 127 to represent each note, so that one unit stands for one semitone. The central piano C note number is 60. The array noteValue stores the current state of each note, pressed (1) or not pressed (0).

// MIDI notes from C to Bint note[] = {60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71};int noteValue[] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};

The array noteThresholdOn will store the minimum sensor measurement necessary to trigger the note on. Likewise, the array noteThresholdOff will store the maximum value which will trigger the note off. This acts like a digital Schmidt trigger, necessary to reduce note triggering caused by noise in the sensor readings. This arrays are populated with the calibrate function.

// Ambient light adjustment valuesint noteThresholdOn[12], noteThresholdOff[12];

Sensor ports, initial values and auxiliary variables.

// Ultrasonic sensorint ultra = 7, ultraValue = 0x45, duration, velocity;

// Accelerometerint accelX = 5, accelY = 3, valueX = 0, valueY = 0, pulseX, pulseY;

11

We read the accelerometer and ultrasonic sensors once every 100 iterations to decrease lag when transmitting the signals though the serial port.

// Sensing interval counterint i = 0;

// Photoresistor calibrationint calibrated = 0;

void setup() { Serial.begin(31250); // MIDI protocol baud rate}

void loop() { if (calibrated == 0) calibrate(); // Read sensors every 100 iterations if (i == 100) {

readAccel();readUltrasonic(); // Read ultrasonic sensor and update ultraValuesendControl(0x03, valueY); // Send effect to MIDI channel 3sendControl(0x01, valueX); // Send effect to MIDI channel 1

i = 0;

} i++;

We read the values on the 12 photoresistors and decide if the corresponding keys were triggered.

for (int i = 0; i < 12; ++i) {

lightValue[i] = analogRead(light[i]);

if (lightValue[i] >= noteThresholdOn[i]) { // The key was "pressed"? if (noteValue[i] == 0) { noteValue[i] = 1; // Change state sendNote(0x90, note[i], ultraValue); // Turn on note }

}else if (lightValue[i] < noteThresholdOff[i]) { // The key was "released"?

if (noteValue[i] == 1) { noteValue[i] = 0; // Change state sendNote(0x90, note[i], 0x00); // Turn off note }

} }}

The functions sendNote and sendControl implement the MIDI messages and the serial communication to turn on or off a note and send a control message (which can be used for several purposes, in particular we will use it to produce sound effects).

// Send a MIDI notevoid sendNote(int cmd, int pitch, int velocity) { Serial.write(cmd); Serial.write(pitch); Serial.write(velocity); // Velocity = intensity or volume}

12

// Send a MIDI controlvoid sendControl(int control, int value) { Serial.write(0xB0); Serial.write(control); Serial.write(value);}

The calibration routine populates noteThresholdOn with sensor measurements when all holes are open and noteThresholdOff when all holes are closed. This is critical since ambient light drastically changes the photoresistor behavior.

// Photoresistors setupvoid calibrate() { while (!Serial.available()); int incoming = Serial.read(); Serial.println("Calibration starting. Leave all holes open and press any key."); while (!Serial.available()); incoming = Serial.read(); Serial.println("Values for threshold off will be: "); for (int i = 0; i < 12; ++i) {

noteThresholdOff[i] = analogRead(light[i]) + 30;Serial.print(noteThresholdOff[i]);Serial.print(" ");

} Serial.println(" "); Serial.println("Close all the holes and press any key."); while (!Serial.available()); incoming = Serial.read(); Serial.println("Values for threshold on will be: "); for (int i = 0; i < 12; ++i) {

noteThresholdOn[i] = analogRead(light[i]) - 20;Serial.print(noteThresholdOn[i]);Serial.print(" ");

} Serial.println(" "); calibrated = 1;}

The ultrasonic distance sensor works by sending one input pulse and measuring the time for the echo pulse to arrive. The function microsecondsToVelocity is used to convert this waiting time to a number from 0 to 127, which will be the MIDI velocity (volume).

// The ultrasonic sensor is used for measuring the velocity (volume) of the notes played.void readUltrasonic() { // The PING sensor is triggered by a HIGH pulse of 2 or more microseconds. // We give a short LOW pulse beforehand to ensure a clean HIGH pulse. pinMode(ultra, OUTPUT); digitalWrite(ultra, LOW); delayMicroseconds(2); digitalWrite(ultra, HIGH); delayMicroseconds(5); digitalWrite(ultra, LOW);

// The same pin is used to read the signal from the PING: a HIGH

13

// pulse whose duration is the time (in microseconds) from the sending // of the ping to the reception of its echo off of an object. pinMode(ultra, INPUT); duration = pulseIn(ultra, HIGH);

if (duration != 0) ultraValue = microsecondsToVelocity(duration);}

int microsecondsToVelocity(int microseconds) { // The speed of sound is 340 m/s or 29 microseconds per centimeter. // The ping travels out and back, so to find the distance of the // object we take half of the distance travelled. int cm = microseconds / 29 / 2;

// Velocity (intensity or volume) in MIDI goes from 0 to 127, and we measured the // minimum distance being 30cm and the maximum being 145cm if (cm >= 60) return 127; else if (cm <= 3) return 0; else return (127 * cm - 3 * 127) / 57;}

The accelerometer is read in this function. Likewise, there is a conversion from its measurements to a 0-127 scale understandable by the MIDI protocol. In particular, the X and Y accelerations are used to produce 2 MIDI sound effects.

// The accelerometer is used for 2 MIDI effect controlsvoid readAccel() { pinMode(accelX, INPUT); pinMode(accelY, INPUT); // read pulse from x- and y-axes: pulseY = pulseIn(accelY, HIGH); pulseX = pulseIn(accelX, HIGH); // convert the pulse width into acceleration // accelerationX and accelerationY are in milli-g's: // earth's gravity is 1000 milli-g's, or 1g. valueX = (((pulseX / 10) - 500) * 8); valueY = (((pulseY / 10) - 500) * 8); valueX = abs(valueX); valueY = abs(valueY); if (valueX >= 1072) valueX = 127; else valueX = (valueX / 1072.0) * 127.0; if (valueY >= 1072) valueY = 127; else valueY = (valueY / 1072.0) * 127.0;}

14

Results

After some debugging and adjustment, the prototype reliability was acceptable. The

calibration routine was essential to make it work, since ambient light turned to be a great

influence. Also, the introduction of two lighting thresholds (similar to a Schmitt trigger)

was important to reduce sensor jittering.

The advantages of using breadboards were the ease to assemble the prototype and

the possibility of quickly change the circuitry. However, faulty connections were a common

problem that could be solved by using printed circuit boards.

15

Conclusion

The device built shows that motion sensing does not necessarily need expensive

sensors and complicated architectures. It is a perfect demonstration of the creative

potential of the technology on music, and it brings a new way to create and perform music

on live or on studio. An improved version could be sold as a DJ equipment, toy or video

game. The involved gesture sensing technologies could also be used for other applications,

for example: socially assistive robots, remote controllers and alternative computer

interfaces.

Possible improvements on the prototype for the second version include replacing

the mechanical parts of it by more comfortable and ergonomic ones, specifically designed

for it. There is also room for more features, such as control knobs and faders on the control

module, for a more practical and complete control of other parameters of the music. Giving

the option of the user to change MIDI parameters of the system such as controller numbers

or channel numbers is another functionality that could be added on a future version of it.

This could be made coupling a LCD display and buttons to the system or through serial

communication to a computer or smartphone. Other sensors could be added to provide

even more ways to change the music through gesture, such as a 3 axis accelerometer and

gyroscopes.

16

Bibliography

[1] MIDI Manufacturers Association. “An Introduction to MIDI”. http://www.midi.org/aboutmidi/intromidi.pdf

[2] “What’s a Theremin?”. http://www.thereminworld.com/Article/14232/what-s-a-theremin-

[3] “Ondes Martenot”. http://www.soniccouture.com/en/product/g27-ondes/

[4] Microsoft Corporation. “Kinect Sensor”. http://www.xbox.com/en-US/kinect

[5] Leap Motion Inc. “Leap Motion 3D Controller”. https://www.leapmotion.com/

[6] Oculus VR Inc. “Oculus Rift Virtual Reality Headset”. http://www.oculusvr.com/

[7] Arduino. “Arduino Micro”. http://arduino.cc/en/Main/arduinoBoardMicro

[8] MIDI Manufacturers Association. “MIDI Products”.

http://www.midi.org/aboutmidi/products.php

[9] “Understanding Schmitt Triggers”. http://www.ti.com/lit/an/scea046/scea046.pdf

[10] “Hairless MIDI to Serial Bridge. http://projectgus.github.io/hairless-midiserial/

[11] “Reason Digital Audio Workstation”. http://www.propellerheads.se/products/reason/

17