PowerPoint Presentation

Violin Pedagogy for Finger and Bow Placement using Augmented RealityFrancois de SorbierHiroyuki ShiinoHideo SaitoIntroductionOverview of our systemViolin extraction and 3D registrationVirtual advisingConclusionOutlineViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier2Violin is a beautiful instrument but one of the most complicatedNo fret on the fingerboardNo help for the position the bow on stringsMotivation

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier3IntroductionMusic JacketVibro-tactile feedbackGuide the bowing arm

Guitar playing supportTracking with markerGuide with virtual hand Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier4Previous works

Y. Motokawa, H. Saito. Support system for guitar playing using augmented reality display. In Proceedings of the 5th IEEE and ACM ISMAR, 243-244, 2006van der Linden, J., Schoonderwaldt, E. and Bird, J. Good Vibrations: Guiding Body Movements with Vibrotactile Feedback.Proceedings of the Third International Workshop on Physicality, 13-18, 2009IntroductionOverlay virtual information on the violinVirtual fretsGuides for the bow and fingersSound analysisViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier5Our objectiveIntroduction

No intrusive deviceNo markerReal time feedback

Marker-free violin trackingusing a RGBD cameraViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier6Our constraintsIntroductionTracking of the violinFeedback displayed on the screenConstant view of the violinVirtual informationViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier7Overview of our system

Overview of our systemFeatures detection and extractionMany occlusions caused by the playerThe surface has a poor textureThe material is highly specular

Difficult to use features in this context

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier8Marker less tracking

Overview of our systemUse Kinect for tracking the violinDepth values for the pose estimation

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier9Depth-based pose estimation

Overview of our systemViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier10WorkflowOverview of our systemColorDepthSegmentationViolin detectionViolin extractionDatabaseRegistrationVirtual informationdisplayedDetect the brown color in the imageRemove noiseMany parts are missingOcclusionsSpecular materialStrings and fingerboardNot enough for tracking

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier11Color segmentationViolin extraction and 3D registration

Get the 3D points from extracted colorCompute the corresponding plane equationOptimized with RANSACAlign a 3D volume to the planeTypical dimensions of a violinKeep the 3D points in itViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier12Extended segmentation

Violin extraction and 3D registrationIterative Closest Point algorithmCompare input points with a modelSlow if too many pointsInaccurate if not enoughOur proposed solutionIncrease the number of modelsReduce the number of points per modelFast retrieval with a plane equation comparisonViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier133D registration with ICPViolin extraction and 3D registrationOffline phase25 modelsCompare the plane equationsStore candidate if the difference is big enoughStore also the plane equation

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier14Creating the databaseViolin extraction and 3D registration

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier15Result of the trackingViolin extraction and 3D registration

Real time (21 milliseconds)Pose also estimated using markersViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier16Evaluation of the trackingViolin extraction and 3D registrationRx(deg)Ry(deg)Rz(deg)T(mm)Minimum error0.120.250.200.22Maximum error13.298.277.8932.1Average error3.072.692.787.20Use the pose estimationLocation manually defined during the capture of the models Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier17Display the fretsVirtual advising

Emphasize the string to be pressedDisplay a red dot at the junction of the string and the fretDefine where the finger has to press the stringViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier18Fingering advicesVirtual advisingSuggest the violinist to play a given noteDisplay the fret/string to be pressedAnalyze the sound obtainedIf fingering is considered correctAdvice about the position of the bow given the difference of pitch (OK / LOW / HIGH)Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier19Other advicesVirtual advisingViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier20ResultsVirtual advising

Performed by confirmed playerBowing is correctCompute the difference of pitch on each fret with the expected oneViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier21Evaluation of the resultsFret number123456789averageDifference of pitch11.114.112.012.413.415.812.813.919.213.8Virtual advisingOriginal marker-free method for virtual advising on a violinMethod based on several pre-computed modelsReal-timeAccurateDisplay virtual guides on the fingerboardAnalyze the note played for further advicesViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier22ConclusionsVirtual advising22Perform a user based analysisValidate or improve our approachUse different kind of displaySee-through HMDSpatial augmented realityApply to other similar instrumentsJapanese shamisenViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier23Future works

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier24QuestionsThank you for your attention