The final hurdle

So i tested my attempt at setting up ambisonic audio for my application today,
This consisted of

• taking the 3d objects coordinates
• converting them from Cartesian to spherical coordinates
• sending them to the max ’sound track’ the object is associated with via my max Connections class
• routing the numbers into the ambisonic encoder for that track in max.

Unfortunately after setting up the 8 speakers in the room i was using i did not get the desired results, so i decided to do some experiments as ambisonic speaker setup has to be very precise to work and the listener needs to be in the ’sweet spot’ for it to work at all.

at first it appeared that the speakers were in the wrong positions and so i changed the routing from the sound card to the speakers so that they would be in the right positions for how i had encoded and decoded the sound which was for a 3d cube, so each speaker would be at one corner of the cube like the diagram below ( each speaker in the diagram would have two in that position one on top of the other).

 However this did not solve the problem, and after experimenting some more with the set-up and what coordinates i was sending to the ambisonic encoder, i decided that it was not really viable to use in my project as either the rooms acoustics i was using were not good enough or i did not know enough about how to set it up.

So i have decided that i will use 5.1 surround sound to produce the audio for my project which seems to be the next best method for spatialising audio in the time frame i have. Although this will only be 2 dimensional and will not take into account the height of the sound it is better than using flash for processing the sound.
To do this i will extend the example provided with max/msp that encodes audio for 5.1 speaker setups to fit within my framework and enable it to have the X+Z positions specified to move the sound around.

There are several options for the creation of 3d sound in my project

1. Use the ICST ambisonic externals for max/msp to provide a very accurate 3d sound representation
2. use flash for the sound and adjust the volume and panning of each sound based on the objects position relative to the cameras view.
3. use some form of sound spatialization such as quad-phonic panning or use of a surround sound system such as 5.1

I’ve looked at some of the pros and cons of each method and will update this page when i have done some experiments with each method as well.

Ambisonics

This would work by having each 3d object in the world associated with a sound ‘track’ in max/msp which would feed the audio stream through the ambisonics engine,  then using an XML connection each tracks position would be updated fairly regularly, probably around 5-10 times a second.
In order for this to work the objects Cartesian coordinates would need to be converted into spherical coordinates before being passed to the ambisonics engine.

pros:

• very accurate representation of 3d sound
• separates the sound side of the program from the visuals spreading out the CPU load

cons:

• difficult to setup requiring very specific speaker placement and room acoustics
• to allow the listening point to move around would involve not actually moving the listener but rotating the world around them.

Sound spatialization

5.1 - by manipulating which speakers the sound is coming out of.

pros:

• easy setup - 5.1 surround sound setups are fairly common

cons:

• not really 3d sound only in 2 dimensions
• same problem as with ambisonics when wanting the listener to move around

Flash 3d sound

pros:

• listener is not limited to a fixed position.
• whole program can be done in flash meaning a wider audience could use the program

cons:

• not really 3d sound only in 2 dimensions
• all processing is done by flash so need to be very careful with resources

This is just a small update on some of the key research i have been doing for my final project, for which i have been looking at interactive and immersive audio/visual environments.

Visual space music

Visual Space Music Introduction (HD) from Ian McKinlay on Vimeo.
One of the first projects i saw which encouraged me to continue looking into this area was Visual space music by Ian McKinlay from the City University of Hong Kong. In his own words visual space music is

“The VSM installation is a musical instrument, a unique space-based arrangement tool, a surround sound spatial mixer, a real-time 3D visualizer. More importantly however, it is the combination of these things; a surreal experience in an electronic world.”

I really liked the idea of visual space music and it started to make me think about other ways environments like this could be used, perhaps as a creative tool for making music and visual entertainment, a bit like VJ’ing but with the visuals and audio actually linked together. Find out more at - http://mememamo.com/vsm/

AudioVisual Environment Suite

Once i started looking into the possibilities of using visual art to create music at the same time i discovered the AudioVisual Environment Suite which is a set of software tools that allow the user to create animations and sound in real time using gestures, it is well described on the website for the project.

“The Audiovisual Environment Suite (AVES) (1998-2000: Golan Levin) is set of seven interactive software systems which allow people to gesturally create and perform abstract animation and synthetic sound in real time. Each environment is an experimental attempt to design an interface which is supple and easy to learn, yet can also yield interesting, infinitely variable and personally expressive performances in both the visual and aural domains”

My Idea

After looking at these projects and some other similar ideas, i had an idea of what i wanted to do, which was to create an immersive environment that would allow the user to create and manipulate musical content in a way that allowed them to create a visual environment that showed what was happening to the music.

Some of the key things i want my project to be able to do are:

• Allow the user to assign sounds samples to objects in a 3d world which could then be manipulated visually.
• Create sounds from visuals using gesture drawing or some other form of drawing input, which can also be assigned to objects in the 3d world.

• Allow them to do this for a performance so that elements of the sound and visuals could be pre-constructed and then also manipulated in real-time.
• Let them define a ‘path’ to flow through the world which defines when sounds are started and stopped or when effects are applied to the rather than traditional time-based sequencing.

The ‘Flow’

Recently after seeing the game Auditorium, i decided on how the sounds in the world would be told to start playing.
Auditorium uses the concept of ‘flow’ which is a stream of particles that are direct by the user into bars of sound to start it playing. i really liked this idea and thought that it would be great way to let the user decide the order in which sounds are started playing by letting them draw the path the flow would take through the 3d world therefore providing a visualisation of the order of the music.

auditorium

Technologies

After a lot of research into the technologies i could use i have settled on two pieces of software that will be combined to create this tool.

• Max/msp for capturing drawing input and turning it into synthesised sound. maybe also using it for the manipulation of samples.
• Flash with the use of a 3d library such as Papervision3d or Away3d for displaying the visual side of the world.
• Flash could also be used for some of the sound manipulation of samples as it now has very good sound capabilities