audio engineering society 143rd convention · 2019. 3. 17. · aes143 iot smart speaker testing...

AES143 IoT Smart Speaker Testing

Audio Engineering Society143rd Convention

Challenges of IoT Smart Speaker Testing

Glenn Hess and Daniel Knighten

Property of Indy Acoustic Research LLC and Listen Inc. All Rights Reserved.

1


“Look Ma! No inputs or outputs.”

➢ How do we test a speaker where the only input or output is the cloud?


2


Test Setup Diagram


3


Physical Setup

IEEE 1329-2010 Test Setup


4


Microphone Measurement Steps

1. Combine recorded activation word with frequency sweep equalized at the MRP of the mouth simulator

– Sample stimulus

2. Playback stimulus, then download from the cloud service

– Virtual Audio Cable was used to capture the playback stream.

3. Analyze captured response recording


5


Speaker Measurement Steps1. Create activation phrase

2. Upload test stimulus to cloud service– The example cloud music service would not accept a linear .wav file.

Conversion to 320 kbps MP3 was used.

– The transparency of the MP3 was tested by encoding and decoding the file and comparing them using Transfer function analysis.

3. Play activation word via mouth simulator and capture response with reference microphone

4. Analyze captured response signal


6


Correcting for Sampling Error

✓ Smart speaker testing is an asynchronous process

✓ The analysis system and the device under test do not share a sample clock

✓ Clock skew or sampling rate mismatch between the systems can lead to measurement error

✓ All measurements were corrected for sampling rate mismatch by precisely measuring the actual playback frequencies and re-sampling the response waveforms to match the stimulus waveforms.


7


Sample Microphone Measurement Results

Freq. (Hz)

100 500 1k 5k 10k

Sens.(dB re 1 FS/Pa)

-15.2 -10.1 -9.6 -11.0 -164

Freq. (Hz)

100 500 1k 4k

THD(%)

3.7 0.2 0.3 0.01


8


Sample Speaker Measurement Results

Freq.(Hz) 100 500 1k 5k 10k

Sens.(dB re 20u Pa/FS)

44.4 88.0 88.6 85.3 84.6

Freq. (Hz)

100 500 1k 5k

THD(%)

5.9 1.2 1.4 0.3


9


Conclusions

• As long as there is a path to inject a test signal and extract a response signal, we can measure it!

• Sampling rate mismatch must be addressed between device and analysis system.

• Once the mystery surrounding recording and playback is removed, the actual behavior of the systems is relatively conventional.


10


Future Work

✓ Objective testing of voice recognition accuracy – Word error rate testing

✓ Activation behavior in the presence of interfering noise and talkers

✓ Effects on objective measurements with non-stationary test signals, e.g. noise, speech, etc…

✓ Directivity response


11


References▪ IEC 61094-4, “Measurement Microphones Part 4:

Specifications for Working Standard Microphones”, November 1995

▪ IEEE 1329-2010, “Standard Method for Measuring Transmission Performance of Speakerphones”, October 2010

▪ ITU-T Recommendation P.51, “Artificial Mouth”, August 1996

▪ Karlheinz Brandenburg, “MP3 and AAC Explained”, presented at AES17th International Conference on High-Quality Audio Coding, August 1999

▪ Steve Temme et al., “The Challenges of MP3 Player Testing”, presented at the AES 122nd Convention, May 2007


12


Questions?


13

audio engineering society 143rd convention · 2019. 3. 17. · aes143 iot smart speaker testing...

Documents