The final hurdle

Development - at the moment this is just a brief outline of the development a more detailed overview of the patch will be up soon.

The development of the software was split up into several stages, the first of which was a simple prototype to make sure that the core features I wanted could be achieved. The first objective was to capture input from the graphics tablet and map it onto a drawing pad which was achieved with jitters jit.lcd object, I used this object rather than the usual LCD object as it allows the drawing to be drawn over the top of an incoming image or movie if needed which I thought would add to the visual experience.­ This was achieved by using a technique called Chromakeying (also known as green screening).

Once this was achieved I moved on to experimenting with how to synthesize the sound from the drawing, I tried several different methods using MSP objects but decided to use a VST synthesizer (Native Instruments FM7), which allowed a more simple technique to map the input from the drawing to sound as they receive MIDI messages to control the sound, and as MIDI messages take 3 parameters I could map the X, Y and pressure values taken from the graphics tablet directly to the synthesizer.
Now that I had the core functions ready I moved onto building the infrastructure of the program. As I was going to have quite a few audio tracks I needed to create a good system to mix the audio together so I created patch that could take up to 10 audio inputs and using a matrix would mix the streams together into one stereo mix to be sent to the output.

So the next step was to put together the image mixing functionality which first required the creation of the 6 image channels that I thought would be used and then creating a switching mechanism to route the two ’sketch pads’ currently selected for mixing to the output via a sub-patch which would fade them together with a mathematical function applied to the colors of the two images to create some interesting effects.

The next step was to create the audio channels to be associated with each sketch pad, as I wanted the user to be able to apply effects to the audio that they created. So I added 5 VST effects to each channel that could be switched between using the buttons on the graphics tablets pen.

Now that the software had all of it functionality I moved onto creating the interface to switch between the different modes of operation. These were: mixing of image streams turned on and off, weather the drawing and synthesis was turned on or off, and also if you were recording into the currently selected audio channel.

I also decided to give the user the possibility to record the output of their audio and video stream which they can then save to disk for playback at a later date.