sound/tracks:Real-Time Synaesthetic Sonification and Visualisationof Passing Landscapes聲音，軌跡火車上的即時的視覺與聽覺的通感翻譯系統

NTUST 指導老師 - 梁容輝 設計研究所資設組 鄭智云

PAPER FROM

http://mmart.iat.sfu.ca/exhibition.html

ABSTRACT

When travelling on a train, many people enjoylooking out of the window at the landscape passing by. The continuously changing view outside the window is captured with a camera and translated intoMIDI events that are replayed instantaneously.

This allows for a reflection of the visual impression, addinga sound dimension to the visual experience and deepeningthe state of contemplation.

The application is intended tobe run on both mobile phones (with built-in camera) andon laptops (with a connected Web-cam).

1. MOTIVATION AND CONTEXT

• 我們常常在坐火車時，看著窗外，看風景略過。另外常做的的事情是聽手機裡的 mp3 。

• 然而，這兩個活動印象中常常同時的上演，不相應對方，視覺和聲音保持獨立的實體，沒有邏輯和美學的連接。

• 該項目的目標是在這裡建立一個結合視覺和聽覺藝術的應用• 建立一個一致的旅遊體驗 - 一種共通經驗 , 在視覺和聽覺印象互相對應

2. GENERAL SCENARIO AND USER INTERFACE

• our goal is to give the passing landscape an active role, to make it the central player that controls the real-time production of music. In a sense, the passing landscape with its fleeting visual impressions acts as the musical score which is going to be interpreted based on outside conditions such as weather, season, lighting, the speed of the train, and the quality of the camera. Every journey is different and will yield a unique composition

3. TRANSFORMING VIDEO TO SOUND

Figure 2: Four example screenshots taken from the mobile version of the software running on a Nokia 6120.The right half of the screen displays the current video taken from the camera. The left half is the condensed history – the ‘score image’.

4. TRANSFORMATION APPROACHES

(IFFT)The filter is realized by applying an inverse Fast Fourier Transformation

4.1 Method 1: Filter Approach

Figure 4: Filter approach. Top: Short excerpt of ajourney as taken by the camera (all middle columnsof the processed frames). Bottom: Spectrogram ofa generated sound.

4.2 Method 2: Piano Roll Approach

Figure 5: Tone-to-colour mapping of Skrjabin’sClavier `a lumi`eres.

4.3 Method 3: Colour-based Approaches

Figure 6: The basic colour mapping used in the per-ceptually motivated approach.

5. EVALUATION OF THE SONIFICATIONALGORITHMS

1. 過濾器方法不使人信服與喜愛因為產生的聲音像是噪音以及吹口哨聲，此外計算每個 ifft 圖片與圖片的過渡耗費太久時間

2.piano roll 沒有經過精密計算所以導致複雜的風景呈現上聽起來很簡單

3.color-based 法 : 目前最好的結果

穩定率與基本節奏方面產生的樂曲和諧度為最高

Six short excerpts from visual ‘score images’.

8. APPLICATION AND PRESENTATION SCENARIOS

• 8.1 Private Enjoyment• 8.2 Public Installations and Performances

9. DISCUSSION

1. 未來將近一步研究其他圖像的色調翻譯方法

2. 計畫軟件開發

提供一種新的有趣的小工具使人們的旅途中更加的愉快

BONUS

靈感來自於 Chemical Brothers - Star Guitar

Chemical Brothers http://www.thechemicalbrothers.com/home

BONUS

化學兄弟 / 有志一同：新歌＋精選The Chemical Brothers / Brotherhood 【 Limited 2CD 】

Chemical Brothers: Let Forever Behttp://www.youtube.com/watch?v=Hmpxsk3dHaA&feature=related

http://www.cp.jku.at/projects/soundtracks/index.html

THANK YOU!More information : http://mew3290.blogspot.com/