The final hurdle

For my project i have had to write an class that will handle all the communication with max/msp, in max the connection is handled by the flashserver external by Olaf Matthes which is available from:

http://www.nullmedium.de/dev/flashserver/

The connection works over an xml socket and the class i have written (basically a conversion of the as2 example to as3) handles the connection and dispatches events depending on the status of the connection. so that’s when its opened,closed or data is received ect..

you can send messages to max as strings or specify a receive object in max to receive the object (if flashserver has remote mode turned on).

i’ve also implemented a function to send file paths to max (as my app will be loading sounds into max but using flash for the gui).
this takes a string and formats it so that the file path remains intact as blackslashes are used as escape characters in both programs so i had to use a regular expression to ensure that the slashes in the path remained the same by the time the path got to max/msp.

i.e.

e:\dir\folder\blah.mp3
you need to send this (in quotes in case there are spaces in the string as max doesn’t like that either!
“e\:\\\\\\dir\\\\\\folder\\\\\\my sound file.mp3″

so a little regex can do this for us,

maxlocation.replace(/\\/g, “\\\\\\”);

I will be releasing this class once I’ve tidied it up a bit and made it suitable for others to use.

also i would like to thank fellow student Alex Riley(http://www.alexryley.net/) for his input into this class as it was built on top of something he created for a similar purpose.

I’ve loosely based my applications design on  the model - view - controller design pattern,

here is a nice little diagram to illustrate that..

model-view-controller relationship in my app

one important part of my application will be how you create 3d objects to be added into the world. i have decided that i will create a seperate ’sandbox’ for the creation and manipulation of  3dobjects before they are added and positioned in the world.

features that i am planning to implement in this are:

• select an object from away3d’s primitives to manipulate and apply textures to.
• adjust the size and shape and rotation of the primitives main parameters e.g. radius for spheres, height,width and depth for cubes and so on.
• ability to apply different material types such as:
  • bitmap materials
  • video(flv) materials
  • basic colour materials
• apply filter effects to the shape - that could also be linked to the sound track the object is assciated with in max/msp to more tightly link the visuals with the audio stream.
• associate a sound track in max/msp with the object.
• ability to add movement animations to the object there are two options for this
  • have a set of predefined animations to select from and apply to the object
  • create some sort of path plotting tool that lets the user plot the objects movement in 3d space.

If the time allows i will also make a stand alone version of this object creator to use to simply test how materials look on objects and exprot the configorations to xml to be imported into other projects using my app, this would allow more reuseability of objects created with it and let the user build up a base collection of objects for use in their projects made using the application.

decided on a structure for the xml format for the storage of projects made using my application which looks a bit like this at the moment

<scene>
<projectProperties name=”Demo” location=”" />
<assets materials=”materials” sounds=”sounds” />
<FlashObjects>
<!——-   new obj set container    ————————————————>
<objectContainer id=”2″ x=”100″ y=”100″ z=”60″ numModels=”1″>
<3dPrimitiveType id=”2.1″ params=”wdith,radius” materialType=”bitmap” materialName=”Sunset.jpg” path=”materialsSunset.jpg” x=”0″ y=”100″ z=”0″>
<animation type=”path” preset=”demo” values=”-700,0,400; -300,0,400; 0,0,400; 0,0,400; 300,0,400; 700,0,400″ />
</3dPrimitiveType >
<sound trackName=”two” type=”sample” name=”sevenNotes” file=”/bass/NT_174_E_PloddingBassLoop.mp3″ />
<filter type=”blurFilter” blurX=”50″ blurY=”50″ />
</objectContainer>
</FlashObjects>
</scene>

so I’ve knocked up a few diagrams using a UML diagram creator extension for eclipse to outline how the program will work and what classes may need to be written etc. these are just ideas for a basic starting point to my application and will no doubt change once i start development.

This is just a small update on some of the key research i have been doing for my final project, for which i have been looking at interactive and immersive audio/visual environments.

Visual space music

Visual Space Music Introduction (HD) from Ian McKinlay on Vimeo.
One of the first projects i saw which encouraged me to continue looking into this area was Visual space music by Ian McKinlay from the City University of Hong Kong. In his own words visual space music is

“The VSM installation is a musical instrument, a unique space-based arrangement tool, a surround sound spatial mixer, a real-time 3D visualizer. More importantly however, it is the combination of these things; a surreal experience in an electronic world.”

I really liked the idea of visual space music and it started to make me think about other ways environments like this could be used, perhaps as a creative tool for making music and visual entertainment, a bit like VJ’ing but with the visuals and audio actually linked together. Find out more at - http://mememamo.com/vsm/

AudioVisual Environment Suite

Once i started looking into the possibilities of using visual art to create music at the same time i discovered the AudioVisual Environment Suite which is a set of software tools that allow the user to create animations and sound in real time using gestures, it is well described on the website for the project.

“The Audiovisual Environment Suite (AVES) (1998-2000: Golan Levin) is set of seven interactive software systems which allow people to gesturally create and perform abstract animation and synthetic sound in real time. Each environment is an experimental attempt to design an interface which is supple and easy to learn, yet can also yield interesting, infinitely variable and personally expressive performances in both the visual and aural domains”

My Idea

After looking at these projects and some other similar ideas, i had an idea of what i wanted to do, which was to create an immersive environment that would allow the user to create and manipulate musical content in a way that allowed them to create a visual environment that showed what was happening to the music.

Some of the key things i want my project to be able to do are:

• Allow the user to assign sounds samples to objects in a 3d world which could then be manipulated visually.
• Create sounds from visuals using gesture drawing or some other form of drawing input, which can also be assigned to objects in the 3d world.

• Allow them to do this for a performance so that elements of the sound and visuals could be pre-constructed and then also manipulated in real-time.
• Let them define a ‘path’ to flow through the world which defines when sounds are started and stopped or when effects are applied to the rather than traditional time-based sequencing.

The ‘Flow’

Recently after seeing the game Auditorium, i decided on how the sounds in the world would be told to start playing.
Auditorium uses the concept of ‘flow’ which is a stream of particles that are direct by the user into bars of sound to start it playing. i really liked this idea and thought that it would be great way to let the user decide the order in which sounds are started playing by letting them draw the path the flow would take through the 3d world therefore providing a visualisation of the order of the music.

auditorium

Technologies

After a lot of research into the technologies i could use i have settled on two pieces of software that will be combined to create this tool.

• Max/msp for capturing drawing input and turning it into synthesised sound. maybe also using it for the manipulation of samples.
• Flash with the use of a 3d library such as Papervision3d or Away3d for displaying the visual side of the world.
• Flash could also be used for some of the sound manipulation of samples as it now has very good sound capabilities