Experimental Robot Musicians

Tarek Sobh, Kurt Coble and Bei Wang

School of EngineeringUniversity of Bridgeport

Table of Content Introduction Motivation: music expressiveness Robot Musician Architecture

Band overview Architecture overview Detailed module description A Robot Musician Example: Jasche

Mechanical changes in real-time performance Results Robot Musician Instrument Future Development Future Discussions Conclusion

Introduction

Robot musicians perform on real instruments through the usage of mechanical devices, such as servomotors and solenoids

Research innovations linking music, robotics and computer science

MotivationMusic Expressiveness

Offer the audience live-experience very similar to listening to a human musician.

Real instrument performance, such as the physical vibration of a violin string, provides a much stronger case in music expressiveness, versus electronic music synthesizers.

Mozart - eine kleine nacht musik: whole ensemble

MotivationMusic Expressiveness (cont.)

Bypass several technical difficulties that are typically encountered by human musicians

More degrees of freedom in real-time performances and reach a higher level of performance difficulty, flexibility and quality.

As an example, a violin is played by a robot musician with hands that have 12 fingers.

Robot Musicians ArchitectureRobot Musicians Band Overview

Each member of the band is a robot musician, which specializes in either a string or a percussion instrument

Robot Musician Name Instrument Played

Micky Drum set

Austin Percussion Ensemble

Dusty II Electric guitar

Jasche 2-bow violin

Drack Bass guitar

John Folk guitar

Stu Classical guitar

Zak White electric guitar

Dusty Red electric guitar

Gold member Gold electric guitar

Bernie Bot Cello

Silver 1-bow violin

Robot Musicians ArchitectureRobot Musicians Band Overview (cont.)

Robot musicians, “the P.A.M. band”, invented by Prof. Kurt Coble.

The moth: features violin solo, composed by Prof. Kurt Coble, companied by percussion ensemble, electric base and electric guitar

Robot Musicians ArchitectureRobot Musicians Band Overview (cont.)

Austin plays a Percussion Ensemble

Dusty plays a red electric guitar

Robot Musicians ArchitectureRobot Musician Architecture Overview

A three-module architecture

Robot Musicians ArchitectureRobot Musician Architecture Overview(cont.)

Software module: interacts with users, provides the programming and composition environment and sends motion commands to the control module through a serial port.

Control module involves rules that govern application processing and connects the software module with the motion module.

Motion module - the hands of the robot musician; powered mechanics such as servos and solenoids, which manage and provide access to the actual musical instrument.

Robot Musicians ArchitectureSoftware Module Infrastructure

Implemented as a set of Turbo C++ programs

The software communicates with the control module through serial port I/O.

Consists of two major components: composition component performance component.

Robot Musicians Architecture Software Module: Composition Component

Direct programming environment: based on a set of pre-defined syntax, users can create C++ programs which control each detailed motion of the robot musician

Compose music scores: through the keyboard and mouse

Robot Musicians Architecture Software Module: Composition Component (Cont.)

Recording function: detects and records details of each key-press or mouse-movement event, and the time delay in between.

Robot Musicians Architecture Software Module: Performance Component

Read and play the musical scores recorded by the composition component or directly composed by a user

Single-line interpretation: produce a musical experience that is almost identical to the original instrumental performance

Tuning component: adjust sound quality of the instruments before composition and performance

Robot Musicians ArchitectureControl Module

An RSV Converter and a costume-manufactured motion control card

The motion control card is built on an Aesthetika Robotika Servo Control Rev 4 board, powered by 12V and 500 mA. It is custom manufactured for this project to support up to 4 servos (4 axes) and 12 solenoids

The converter interfaces between the serial port and motion control card

Robot Musicians ArchitectureControl Module (Cont.)

Receives motion control commands from the software module and computes joint angles and servo motor velocities for each axis

Deliver these values to the motion module, ensuring smooth servo motor movements.

Robot Musicians ArchitectureControl Module (Cont.)

Control module with motion control card and converter, linked with the motion module of an incomplete keyboard robot musician hand

Robot Musicians ArchitectureMotion Module

The Motion module is provided by servos and solenoids

Two types of servos (Hitec Quarter Scale HS-700BB Servo and Hitec HS303 Servo) and three types of solenoids are used in its construction

Robot Musicians ArchitectureMotion Module (Cont.)

A violin is set up with two bows controlled by two servos and twelve fingers (solenoids). Servos are attached to bows of the violin. They move the bows back and forth across the violin strings. Solenoids, act as magnets to depress the strings when charged

Amazing grace: traditional American folk song.

Robot Musicians ArchitectureMotion Module (Cont.)

Servo attached to one bow of Jasche

Solenoid (with holding power of 1.5 pounds) attached to Jasche

Servomotor In Action

Robot Musicians ArchitectureMotion Module (Cont.)

A coffee container’s plastic lid is connected with a servo so it flutters against the body of a drum when the servo receives control command from the control module

Sample Motion Module Architecture: drumstick controlled by solenoid

Robot Musicians ArchitectureMotion Module (Cont.)

Sample Motion Module Architecture: chimes wand controlled by servo

Motion Module In Action Mozart - eine kleine nacht

musik: whole ensemble

Robot Musicians ArchitectureA Robot Musician Example: Jasche

The Robot Musician Jasche plays a 2-bow 12-finger 2-string violin tuned with 3 octave chromatic range (From F below middle C to two octaves above middle C), which produces 24 pitches.

Happy Birthday

Robot Musicians ArchitectureA Robot Musician Example: Jasche (Cont.)

Jasche In ActionAmazing grace: traditional American folk song

Robot Musicians ArchitectureA Robot Musician Example: Jasche (Cont.)

Software module: compose musical pieces through mouse movements

Keys on the third line of the keyboard, symbols “QWERTYUIOP” correspond to 10 speed levels of the bow movement.

Keys on the second line of the keyboard, symbols “1234567890-=” correspond to 12 solenoids used to press the strings, producing different pitches

Robot Musicians ArchitectureThe Robot Musician Band Architecture

Robot Musicians ArchitectureThe Robot Musician Band Architecture (Cont.)

8 MCC boards are chained together, meaning, 8 robot musicians can be controlled by one single computer terminal simultaneously

The Mechanical Changes in Real Time Performance The smooth transition between two

musical notes (continuation of music score): the improvement of mobility, elasticity and sensitivity of the robot hands

High speed communication between the PC and the motion control card becomes critical

Real Time PerformanceMechanical Issues

Results

The recent work on robot musicians has some promising results. The existing system is found to be a robust method in controlling complex musical mechanics

A musical piece collection will follow 7sg2: composed by Prof. Kurt Coble, the name of the musical

piece stands for "7 solenoids guitar part 2", 6-string electric guitar, tuned in a parallel major 7th chord

Results: drum set in action

Results: whole ensemble Botman: composed by Prof. Kurt Coble, featuring whole ensemble

Robot Musical InstrumentA New Generation and A New Art Form

Musical instruments that are more complex in nature for robot musicians

A combinational drum set: drums are specially positioned so that they are easily accessed by the robot hands.

A violin can be played with two bows simultaneously

Robot Musical InstrumentA New Generation and A New Art Form (Cont.)

Digitally-Manipulated Analogue Signal Systems (D-MASS) is proposed in this work as a description of this new art form. The following Table makes a comparison between MIDI (Multi Instrument Digital Interface) and D-MASS.

MIDI D-MASS

Sound resources Triggers sound samples from a sound bank 

Analogue signal from an acoustic instrument

System Requirement

No mechanical devices Requires mechanical devices

Sound quality Identical Allowing for subtle variances each event is unique

Sound sample manipulation

Samples manipulated electronically

Acoustic waves can be manipulated electronically

Future Development

Software/hardware tools: read and play traditional scores.

soft computing (fuzzy logic, neural networks, evolutionary and genetic algorithms) and artificial intelligence / expert systems techniques

Improving the emotional quality of the music performance

Having the robot musicians listen to various music pieces, recognize the tones, improve on them and then re-play them

Wireless control

Future Discussions

Appreciating the interaction of music, computer science, robotics and the new concepts that emerge from robot musicians.

The analysis of the degree of intelligence and sensitivity achieved by robot musicians

Conclusions

The experience of designing robot musicians is a breath-taking experiment.

A fully functional robot orchestra can be established in the near future, which will play not only simple tunes, but also masterpieces.

Acknowledgements

James Sedgwick Dr. Jeffrey Johnson Dung Hoang and Ying Guo Interdisciplinary Robotics, Intelligent

Sensing, and Control (RISC) laboratory

Arnold Bernhardt Center, Austin Mather Bubble Theatre

Current Project Status

2002 Sigma Xi Grant-in-Aid of Research recipient

Paper titled “Experimental Robot Musicians” has been accepted by the Journal of Intelligent & Robotic Systems

Thank You