The final hurdle

So the project was finished a week or two ago and handed in, I think the presentation went quite well although I could have done with more time to show it off but oh well.
Here are some screen shots to give you a taster of what the application looks like, the app will be available to download in the next few days. ( once the projects link in the menu is turned on you can get it)

The Object Selector

Add Custom Materials

Rotating an Object

Scene Example

Scene Example 2

For my project i have had to write an class that will handle all the communication with max/msp, in max the connection is handled by the flashserver external by Olaf Matthes which is available from:

http://www.nullmedium.de/dev/flashserver/

The connection works over an xml socket and the class i have written (basically a conversion of the as2 example to as3) handles the connection and dispatches events depending on the status of the connection. so that’s when its opened,closed or data is received ect..

you can send messages to max as strings or specify a receive object in max to receive the object (if flashserver has remote mode turned on).

i’ve also implemented a function to send file paths to max (as my app will be loading sounds into max but using flash for the gui).
this takes a string and formats it so that the file path remains intact as blackslashes are used as escape characters in both programs so i had to use a regular expression to ensure that the slashes in the path remained the same by the time the path got to max/msp.

i.e.

e:\dir\folder\blah.mp3
you need to send this (in quotes in case there are spaces in the string as max doesn’t like that either!
“e\:\\\\\\dir\\\\\\folder\\\\\\my sound file.mp3″

so a little regex can do this for us,

maxlocation.replace(/\\/g, “\\\\\\”);

I will be releasing this class once I’ve tidied it up a bit and made it suitable for others to use.

also i would like to thank fellow student Alex Riley(http://www.alexryley.net/) for his input into this class as it was built on top of something he created for a similar purpose.

Each of the ‘sound tracks’ in my application will have a surround sound panner module in it thorugh which the audio will be routed after any effects processing has been applied

patch that takes X,Z coordinates of an object to position a sound

each of the panning modules then sends its audio stream to the output once its posistion has been adjusted.

output patch for surround sound

So i tested my attempt at setting up ambisonic audio for my application today,
This consisted of

• taking the 3d objects coordinates
• converting them from Cartesian to spherical coordinates
• sending them to the max ‘sound track’ the object is associated with via my max Connections class
• routing the numbers into the ambisonic encoder for that track in max.

Unfortunately after setting up the 8 speakers in the room i was using i did not get the desired results, so i decided to do some experiments as ambisonic speaker setup has to be very precise to work and the listener needs to be in the ‘sweet spot’ for it to work at all.

at first it appeared that the speakers were in the wrong positions and so i changed the routing from the sound card to the speakers so that they would be in the right positions for how i had encoded and decoded the sound which was for a 3d cube, so each speaker would be at one corner of the cube like the diagram below ( each speaker in the diagram would have two in that position one on top of the other).

 However this did not solve the problem, and after experimenting some more with the set-up and what coordinates i was sending to the ambisonic encoder, i decided that it was not really viable to use in my project as either the rooms acoustics i was using were not good enough or i did not know enough about how to set it up.

So i have decided that i will use 5.1 surround sound to produce the audio for my project which seems to be the next best method for spatialising audio in the time frame i have. Although this will only be 2 dimensional and will not take into account the height of the sound it is better than using flash for processing the sound.
To do this i will extend the example provided with max/msp that encodes audio for 5.1 speaker setups to fit within my framework and enable it to have the X+Z positions specified to move the sound around.

I’ve loosely based my applications design on  the model – view – controller design pattern,

here is a nice little diagram to illustrate that..

model-view-controller relationship in my app

i’ve been researching into current practices in VJing and looking at some of the guys pushing the boundries in this area and this installation is nothing short of amazing!

Etienne De Crecy Live 2007 Transmusicales de Rennes from Clement bournat on Vimeo.

p.s.

turns out that mtv and the killers ripped this off quite badly – see link below

http://blog.antivj.com/2008/exyzt-installation-ripped-off-by-the-killers

one important part of my application will be how you create 3d objects to be added into the world. i have decided that i will create a seperate ‘sandbox’ for the creation and manipulation of  3dobjects before they are added and positioned in the world.

features that i am planning to implement in this are:

• select an object from away3d’s primitives to manipulate and apply textures to.
• adjust the size and shape and rotation of the primitives main parameters e.g. radius for spheres, height,width and depth for cubes and so on.
• ability to apply different material types such as:
  • bitmap materials
  • video(flv) materials
  • basic colour materials
• apply filter effects to the shape – that could also be linked to the sound track the object is assciated with in max/msp to more tightly link the visuals with the audio stream.
• associate a sound track in max/msp with the object.
• ability to add movement animations to the object there are two options for this
  • have a set of predefined animations to select from and apply to the object
  • create some sort of path plotting tool that lets the user plot the objects movement in 3d space.

If the time allows i will also make a stand alone version of this object creator to use to simply test how materials look on objects and exprot the configorations to xml to be imported into other projects using my app, this would allow more reuseability of objects created with it and let the user build up a base collection of objects for use in their projects made using the application.

decided on a structure for the xml format for the storage of projects made using my application which looks a bit like this at the moment

<scene>
<projectProperties name=”Demo” location=”" />
<assets materials=”materials” sounds=”sounds” />
<FlashObjects>
<!——-   new obj set container    ————————————————>
<objectContainer id=”2″ x=”100″ y=”100″ z=”60″ numModels=”1″>
<3dPrimitiveType id=”2.1″ params=”wdith,radius” materialType=”bitmap” materialName=”Sunset.jpg” path=”materialsSunset.jpg” x=”0″ y=”100″ z=”0″>
<animation type=”path” preset=”demo” values=”-700,0,400; -300,0,400; 0,0,400; 0,0,400; 300,0,400; 700,0,400″ />
</3dPrimitiveType >
<sound trackName=”two” type=”sample” name=”sevenNotes” file=”/bass/NT_174_E_PloddingBassLoop.mp3″ />
<filter type=”blurFilter” blurX=”50″ blurY=”50″ />
</objectContainer>
</FlashObjects>
</scene>

so I’ve knocked up a few diagrams using a UML diagram creator extension for eclipse to outline how the program will work and what classes may need to be written etc. these are just ideas for a basic starting point to my application and will no doubt change once i start development.

There are several options for the creation of 3d sound in my project

1. Use the ICST ambisonic externals for max/msp to provide a very accurate 3d sound representation
2. use flash for the sound and adjust the volume and panning of each sound based on the objects position relative to the cameras view.
3. use some form of sound spatialization such as quad-phonic panning or use of a surround sound system such as 5.1

I’ve looked at some of the pros and cons of each method and will update this page when i have done some experiments with each method as well.

Ambisonics

This would work by having each 3d object in the world associated with a sound ‘track’ in max/msp which would feed the audio stream through the ambisonics engine,  then using an XML connection each tracks position would be updated fairly regularly, probably around 5-10 times a second.
In order for this to work the objects Cartesian coordinates would need to be converted into spherical coordinates before being passed to the ambisonics engine.

pros:

• very accurate representation of 3d sound
• separates the sound side of the program from the visuals spreading out the CPU load

cons:

• difficult to setup requiring very specific speaker placement and room acoustics
• to allow the listening point to move around would involve not actually moving the listener but rotating the world around them.

Sound spatialization

5.1 – by manipulating which speakers the sound is coming out of.

pros:

• easy setup – 5.1 surround sound setups are fairly common

cons:

• not really 3d sound only in 2 dimensions
• same problem as with ambisonics when wanting the listener to move around

Flash 3d sound

pros:

• listener is not limited to a fixed position.
• whole program can be done in flash meaning a wider audience could use the program

cons:

• not really 3d sound only in 2 dimensions
• all processing is done by flash so need to be very careful with resources