In terms of post-production development, 2017 belonged to the VR industry. A major breakthrough occurred during the course of last year when audio was finally (despite being totally obvious) recognised as a critical element in VR experiences.
The premise of VR is to create an alternate reality, but without the right audio cues to match the visuals, the brain doesn’t buy into the illusion. Sound isn’t just a cue used to enhance the auditory experience anymore — it’s something that can totally change the VR adventure or even affect the entire storyline. The cinema industry has transitioned to immersive sound, primarily employing the Dolby Atmos, DTS and Barco Auro 3D systems, so is it time to adopt or adapt these systems for television? As we roll into 2018, we might just see a change in the way we mix audio forever…
In today’s audio environment, systems and methods for reproducing audio can broadly be categorised as either channel-based or object-based. In channel-based audio, a soundtrack is created by recording a separate audio track (channel) for each speaker. Common speaker arrangements for channel-based surround sound systems are 5.1 and 7.1, which utilise five and seven surround channels respectively, and one low-frequency channel. A major drawback of channel-based audio is that each soundtrack must be created for a specific speaker configuration.
Object-based audio addresses this drawback by representing a sound scene as multiple separate audio objects, each of which comprises one or more audio signals and each audio object is associated with metadata that defines a location and trajectory of that object in the scene. Object-based renderers provide greater flexibility than channel-based audio systems, in that they can cope with different numbers of speakers creating a greater spatial experience. Spatial audio is key in delivering lifelike VR experiences. Spatialised sound sources have three-dimensional positional data to make listeners feel like sounds are actually coming from those visual locations. In real acoustic spaces like movie theatres for example, the delivery of spherical audio comes from technologies like Dolby Atmos, DTS Neo, and Barco Auro. However, these immersive real-space technologies require loads of speakers and amplifiers, something that he average consumer will never be able to afford, let alone have the space to install them. Developers are aware of this and realise that the primary thrust of 3D audio innovation will occur over headphones. Headphones have the added advantage in that the listener is free to move around a room or environment. Back in the 70’s the British National Research Development Corporation developed Ambisonics, a full-sphere surround sound technique. In audio, sound is normally recorded in the horizontal plane. Ambisonic recording techniques also cover sound sources above and below the listener creating a 3D effect.
Unlike other multi-channel surround formats, its transmission channels do not carry ‘speaker’ signals. Instead, they contain a speaker-independent representation of a sound field called B-format, which is then decoded to the listener’s headphone setup. Immersive sound is enhanced when it is “tracked” with natural head movements. When you turn your head, the virtual sound must react like they would in real sensory space and this is why Ambisonics offers so much for headphone users. Audio specialists Sennheiser, recently brought out a suite of 3D audio technologies called Ambeo, which included a VR microphone that captures ambisonics and an upmix algorithm that converts stereo tracks into a high-quality 9.1 sound experience. Ambisonics B is the accepted audio format for spatial sound on web browsers and therefore the preferred choice for the production of VR audio. Both VR goggles and web browsers include a Binaural Renderer, enabling binaural playback. If Head Tracking is applied, Ambisonics B-format can be spun and dynamic binaural playback is achieved! This means that listeners can freely position themselves within the sound field, with their audio perception changing accordingly.
While plenty of headphones offer surround sound, they don’t really emulate 3D space around you. For example, as soon as you move your head, the sound moves with you, instead of staying relative as it would in an actual space. Ossic X claim to be the world’s first true 3D headphones, capable of recreating sound the same way you hear it in the real world. If the reaction to their Kickstarter campaign is any indication, plenty of people are very excited about what this could mean for the future of headset audio.
The Ossic X automatically calibrate themselves to your head and torso as soon as you put them on, enabling more accurate sound placement. The real difference comes when you move your head. Instead of the sound moving with you, as with most headphones, the sound stays put — turn your head to hear something behind you, and suddenly it’s coming from in front of your face, at least as far as your ears are concerned. The Ossic team has some noteworthy names to its roster, including Dr. Kaushik Sunder, previously of NASA, and industry thought leader Sally Kellaway, who now serves as Creative Director.
Dolby Laboratories have brought binaural and object based audio editing into television post-production suites by introducing Dolby Atmos workflows into Avid’s Pro Tools HD. Until now, mixing in Dolby Atmos required additional software plug-ins and complex routing workflows to accommodate the popular format. Pro Tools HD now features an inclusive suite of Dolby Atmos workflows, enabling audio post professionals to work more fluidly in the multi-channel and object-based audio format especially in the VR environment.
BBC Research and Development are experimenting with Binaural Sound – their version of immersive spatial audio for television. They have been working on 3D sound technology that can bring a theatre type experience to listeners’ headphones. This technology can be applied to traditional programme content but also to new interactive content experiences including virtual and augmented reality. They have already produced content for several of the Planet Earth II 360˚ videos for example and plan to have an open technology standard enabling binaural television broadcasting in the not too distant future.
The next step in VR audio will be exploring exciting new techniques during post-production in order to leverage the full spectrum of possibility in television broadcast and in the VR world. Advances in VR audio are going to be key in perfecting the experience for mainstream consumers and for giving pioneers the tools they need to drive innovative experiences. These developments will go a long way toward achieving both goals and will help shape the industry in the years to come.
