April 16, 2021
To contribute to comfortable listening environments, Hitachi has developed technology that uses a newly designed 27-channel microphone array based on a 3D sound propagation model to record acoustic environments that can be played back in a highly realistic manner through a HMD (head-mounted display) and a multi-channel loudspeaker system. In addition to recording and playback, this technology can analyze the direction and intensity of incoming sounds. By superimposing a visual representation of these sound sources on an omnidirectional image captured by a 360-degree camera, it is possible to create an audiovisual experience of a scene, including information about where sounds are coming from. In the future, we will combine this technology with acoustic simulation to produce acoustic environment design technology that can be applied to urban and residential spaces to contribute to the creation of comfortable, human-friendly living spaces where, for example, it is easy to hear and communicate with other people.
Fig.1 ：Explanatory video on the developed technology (in Japanese)
For 360°video demonstration (in Japanese), click here.
This work was presented at the 2020 AES International Conference on Audio for Virtual and Augmented Reality, which was held online from August 17th to 19th, 2020. (Title: Tesseral Array for Group Based Spatial Audio Capture and Synthesis).
This result was achieved through a joint study with Professor Toru Kamekawa and Professor Atsushi Marui of Tokyo University of the Arts. In addition, the acoustic environments inside the train and on the station platform were recorded with the cooperation of Hitachinaka Seaside Railway Co., Ltd.
Hitherto, one of the evaluation indexes that has been used in the design of acoustic environments has been the acoustic levels measured by an omnidirectional microphone (sound level meter) that does not have any directionality with regard to the incoming sounds. However, since humans have two ears and are able to hear sounds coming from all directions, sounds recorded in this way produce different sound pressure levels and create a different subjective impression of the acoustic environment.*1 In our technology, we measure sound pressure levels and the subjective impression of the sound environment by using a 26-channel omnidirectional microphone array with a tesseral system configuration*2 based on a three-dimensional sound propagation model*3 centered on the human listener. In this way, we have been able to record incoming sounds from all directions for the first time. By playing this audio back through a 26.1-channel loudspeaker system*4 with a similar configuration to the 27 microphones used for recording, or by converting it to stereo sound and playing it back through headphones, it is possible for the listener to experience the recorded acoustic environment with a high degree of realism. Furthermore, by recording video with a 360-degree camera, we can also perform subjective evaluations of acoustic environments in remote areas with the addition of visual information.
In working towards implementing a comfortable acoustic environment, it is essential to understand how humans interpret sound information such as the direction and intensity of incoming sounds. However, since sounds are not visible, they have to be analyzed by a highly skilled sound engineer. To make it easier to understand the physical information conveyed by sounds, we have developed technology that analyzes and visualizes the direction and intensity of incoming sounds by analyzing 27 channels of recorded sound. This technology first reconstructs the recorded sound by means of multiple spherical harmonics,*5 taking advantage of the spherical shape of the directional microphone array, which has an equiaxial geometrical configuration. Each of these spherical harmonics is then weighted to form a sharp directional peak that determines the intensity of sound from a particular direction. Then, by changing the direction of this directional peak to probe all directions, the intensity of the incoming sounds can be broken down by direction. The sound intensities obtained in this way can be visualized as a color map and superimposed on the recorded 360-degree image to provide a visual representation of the sounds coming from all directions. This makes it easy to understand the physical information of sound in a sound environment by augmenting the highly realistic acoustic reproduction with visual clarification of information such as the direction and intensity of incoming sounds.
For more information, use the enquiry form below to contact the Research & Development Group, Hitachi, Ltd. Please make sure to include the title of the article.
