Recently, the Department of Media Technology at Aalto University offered a seminar entitled Applied Data Analysis and Visualization. The course used spatial impulse response measurements from concert halls as the context to explore high-dimensional data visualization methods. Students were encouraged to represent source and receiver positions, spatial aspects, and temporal development of sound fields, frequency characteristics, and comparisons between halls, using animations and interactive graphics. The primary learning objectives were for the students to translate their skills across disciplines and gain a working understanding of high-dimensional data visualization techniques. Accompanying files present examples of student-generated, animated and interactive visualizations.
I. Introduction and background
Courses in acoustics are often taught as part of electrical or mechanical engineering or similar programs (Kessissoglou, 2012). Programs focused on room acoustics are generally housed within architecture departments, e.g.,Virginia Tech, Rensselaer Polytechnic Institute, and The University of Nebraska (ASA Committee on Education in Acoustics, 2013). As acoustics is an interdisciplinary field, much other research is done outside these contexts, but integrating the research into the curricula is less straightforward and must contribute to larger departmental teaching goals. The present work presents an example of integrating concert hall acoustical quality research into the curriculum of the the Department of Media Technology within the Aalto University School of Science in Espoo, Finland. The structure and results of a recently taught seminar course are detailed.
Within the department, research group specialties are broad ranging, covering computer graphics, video games, human-computer interaction, image quality, semantic computing, and acoustics. The Department of Media Technology offers majors of Media Technology and WWW-Technologies as part of the degree program in Computer Science and Engineering and the major of Media as part of the degree program in Information Networks. Courses are offered during two teaching periods in each of the spring and fall terms, which run from the end of August to the beginning of December and the beginning of January to the end of May, respectively. Teaching periods are approximately 6 wk with 1 wk evaluation periods before and after. Each term, one research group in the department takes its turn to organize a seminar course of four to eight credits (ECTS) with a topic of general interest to students in the department but also open to all students in the school.
In the spring term of 2013, the course was delegated to the Virtual Acoustics Team.1 Research on concert hall acoustic quality is the main focus of the group, but this topic was deemed too specific and misaligned with other studies in the department to be of general interest. The solution was to offer a course in data visualization and analysis, which complimented the strengths of the department, while taking advantage of the research group's extensive database of concert hall measurements.
II. Course content
The course contact time consisted of six sessions, each 2 h long. The first session was utilized to introduce the data set and some background on its meaning and measurement. The introduction to the data sets was followed by a short precedent study of spatial acoustic impulse response visualization. This survey included examples by Gover et al. (2004), Abdou and Guy (1996), Bassuet (2010), and Pätynen et al. (2013). The second consisted of a short survey of high dimensional data representation techniques and some possible visualization tools. Geometric (Ankerst et al., 2011), icon-based (Turner and Woodhouse, 2012), and pixel-oriented (Lammarsch et al., 2009) methods were covered, even though many of these methods are not generally applied to acoustic data directly. The intent was to incorporate various methods to encourage novel applications. The following three sessions were executed as design studio style critique sessions in which students presented concepts for their visualizations and works-in-progress, and the instructors provided comments and suggestions. In the final session, students gave presentations of their final visualization designs.
The course had two primary objectives at the outset. The first was to learn to apply computer graphics methods to design novel constructs to visually explore high dimensional abstract data. The second was to learn to apply knowledge across disciplines; in this case, the application was from computer and information science to room acoustics. Assessment was based on a presentation of the final result, but as importantly, active participation in lectures was strongly weighted. This was an important feature of the assessment scheme because the success of the course depended heavily upon the interactive design review sessions. After the first such session, students were asked to produce a written plan for their visualization project, which was then used to guide future sessions.
While students were expected to have some background in computer science or computer graphics, not all participants in the course were familiar with acoustical concepts or room acoustic measurement techniques. This was covered in the first lecture. The data set was a collection of spatial room impulse responses that the Virtual Acoustics Team measured during a tour of 10 of Europe's great concert halls in November of 2012. These measurements are similar to those described by Lokki et al. (2012). Measurements were conducted with 34 loudspeakers controlled by 25 signal channels on the stage of each hall and for multiple listening positions in the audience areas. The receiver was an array of 6 omni-directional microphones, which allowed post-processing to estimate the direction of arrival of the sound at each instant in the impulse response. The Spatial Decomposition Method developed by Tervo et al. (2013) was applied to the captured pressure signals to estimate the spatial data in four vectors: The monaural pressure impulse response and the X, Y, and Z components of the sound arrival direction vector for each sample in one monaural impulse response. These vectors, for each hall, and source-receiver pair, are the data set that the students received. Thus at least eight dimensions could be represented. Students were encouraged to explore visualizing source positions, receiver positions, spatial aspects, and temporal development of the sound field, frequency characteristics, and comparisons between halls.
In technical disciplines, science, and engineering, the importance of creativity can be underestimated but is receiving increasing attention in pedagogical studies (Costantino et al., 2010). Student engagement is an important component of creative education, and cooperative problem solving has been suggested as a strategy to foster engagement (Smith et al., 2005). The design studio model that is well established in the arts and architecture is seeing successful new applications in technical disciplines (Thompson et al., 2011) and was utilized in this course to develop the student visualizations. Three sessions were conducted in which all students gathered with the instructors and discussed ideas for and progress on visualizations. These sessions facilitated exchange of ideas, brainstorming, and interaction about the conceptual and practical challenges of representing the high dimensional acoustic data.
III. Student project results
A. Lateral plane energy distribution in octave bands (Aki Haapaniemi)
This visualization demonstrates the lateral energy, integrated over an interactively selectable time window, and divided into octave bands; see Fig. 1(a). Octave bands are illustrated in concentric rings with the lowest frequencies at the center. The data disk is located within the hall at the receiver position, and the energy from each direction is highlighted in green. The lateral energy is isolated from the vertical components via a weighting based on the cosine of the elevation of the incident direction. This visualization is novel because while time-frequency plots and spatial energy plots have been utilized in the past, spatial frequency plots are much less common in room acoustics. Interactive sliders allow the viewer to examine various time windows. The visualization was produced using matlab software, incorporating a three-dimensional (3d) model produced using sketchup.
In the accompanying video, Mm. 1, a comparison of two halls is presented. The left hall is the Berlin Konzerthaus, and the right is the Beethoven Hall in Stuttgart. The animation begins by showing the integrated energy of the first 10 ms, with multiple sources on stage, only the direct sound is visible in the response. As the animation progresses, the time window lengthens, and more of the impulse response energy is incorporated, including lateral reflections and eventually diffuse reverberation. There are significant differences between the halls. In Hercules Hall, strong broadband lateral reflections are evident early in the response accompanied by one from the rear. No such reflections are present in Beethoven Hall before the arrival of the diffuse reverberation. Also in Hercules Hall the reverberation is much stronger, and more uniformly distributed than in Beethoven hall. These spatial differences affect perception of the halls, and may be difficult to detect with traditional measures of concert hall quality.
B. Nested tessellated spheres (Petri Leskinen)
Similar to the previous one, this visualization places a representation of the sound energy within the concert hall model at the receiver location; see Fig. 1(b). Here, the data are represented broadband rather than in octave bands. The visualization consists of a sphere that has been tessellated and distorted into a balloon to represent the room impulse response. The balloon is distorted to represent the energy arriving from each direction during the selected time window of the impulse response. The time window begins at 0 ms, and the end time can be adjusted. The energy for each direction is integrated over the time window to calculate the radius of the sphere in each direction. The spheres' transparency can be adjusted such that several time windows can be nested to examine the early and late sound responses simultaneously. Because the integrated energy of each sphere always begins at 0 ms, the balloons are progressively larger for later time windows. Hence, if the inner balloon is opaque and the outer balloon is transparent, it is easy to see spatial relations between early and late sound. The graphic is interactive to allow the viewer to fly around and zoom the response, which yields additional insight into spatial relationships.
The model can be viewed in a browser. It uses the WebGL-technology with Three.js library. Library Three.js provides further support to needed matrix algebra, handling camera movements, and creating geometries. Using the controls for starting and ending times, one can observe how the architecture of the halls reflect the incoming sounds.
In the accompanying video, Mm. 2, exploration of the visualization is demonstrated. Data from the Berlin Konzerthaus are superimposed on a 3d model. Initially, only the first 80 ms of the response is displayed. Within this time window, the direct sound and a reflection from the rear are prominently visible along with some smaller reflections from above and the sides. Subsequently, a second tessellated sphere is added to display the energy until 250 ms. This reveals additional lateral energy, as well as diffuse reverberation, which roughly defines a sphere around the listener position.
C. Dynamic sound arrivals (Morley Zan-Bi and Antti Kuusinen)
This visualization is based on the open-source programming package Processing. The concept is to visualize the time and direction of arriving sound with parallelepipeds that converge on the listening position. In Fig. 1(c), the sound source is to the right, and the listener position is at the center. The direct sound is indicated in red, the early reflections in green, and the late reverberation in blue. The size of each particle and its final distance from the receiver position represent the strength of the reflection at that particular time instant. The visualization is animated to allow the viewer to watch the development of the room response over time and interactive to allow the user to pan and zoom around the response for a complete view.
In the accompanying video, Mm. 3, a measurement from all sources of the loudspeaker orchestra [see Lokki et al. (2012), Fig. 1] in the Vienna Musikverein at a receiver position 1 m stage right from the conductor position and 15 m into the audience is displayed. Initially, the sources are to the right, the animation then proceeds to rotate to reveal the rest of the response. The direct sound can be seen arriving as red particles from the right, then the early reflections build up in green. This early energy is concentrated in the front, from the sides, and is very diffuse. This is in agreement with observations by Beranek (2004) and others. Toward the end of the animation, the late reverberation is displayed in blue, a very uniform spherical cloud of points.
The design studio approach to the data visualization course allowed the students to use their own creativity and various software tools to illustrate a large database of high dimensional data. These outcomes demonstrate the achievements that are possible when the students' existing skills are leveraged toward the learning objectives. Rather than teaching unfamiliar methods, software, and techniques, students were able to explore possibilities using methods familiar from their diverse backgrounds. The measured concert hall data provided an interesting and challenging context for this exploration. The resulting novel visualization techniques may have further utility for room acoustics research and education and are a tangible and useful course outcome. Room acoustic research may be too narrow a topic for broad application across a diverse departmental curriculum. However, this course demonstrated successful utilization of specialized research data to convey more general principles while, at the same time, furthering research goals.