Luke McGowan ::: Negotiated Project B: Development Blog :::
Live Musical Performances Visualisation System
Here's a diagram of the current system I'm working with at the moment.
The key
components are:
A: PIEZO SENSORS - which are connected to drumkit items to pick up drum hits (on the snare drum, kick pedal, toms, hats etc) B: ARDUINO - which processes the signals from the Piezo sensors and conveys them to the laptop via a USB lead C: LAPTOP - which
is used to process data from the Piezo sensors and to process the visuals via Macromedia Flash software D: WEBCAMS - which are connected via USB to the laptop and can be used to capture events on stage or within the audience E: 3G CARD - enabling me to retrieve and display data from websites, specifically SMS data stored in a MySQL database on my website F: S-VIDEO & COMPOSITE LEADS - used to transmit the video data to LCD screens (alternatively VGA can be used to output to a projector) G: BLUETOOTH ADAPTOR - enabling the sending of MP3 audio tracks to audience members during the performance H: AUDIO LEADS - audio signals are sent to the laptop from the amplifyer head or mixing desk so that the visualisation system can react to live audio
Here's an explanation of how each of the above sections has been developed and works within the system.
A: PIEZO SENSORS
The Piezo sensors are able to pick up vibrations and convert this information into an analogue signal, which is then sent to the Arduino circuit. The Piezos are connected to two wires which are in turn soldered to a 3.5mm jack. This is connected to the Arduino system via a circuit box which has been made to house the Arduino. 3.5mm jacks are used because the resistors required in the circuit are built into the box in which the Arduino is housed. This makes the production of extra sensors quicker and means I can produce backup sensors using less resistor components.
The sensors can be attached to different drumkit pieces using gaffer tape.
The Piezo sensors themselves are taped to pieces of cardboard in order to cushion the impact of the drum vibrations. This is important since the Piezo solder can crack fairly easily, thus breaking the circuit mid-gig and rendering the sensor useless (until it has been resoldered). This sort of setup is also recommended in a website I found at http://www.soundonsound.com/sos/1995_articles/aug95/diydrumpads.html.
B. ARDUINO
A quick introducition is probably required for Arduino:
'Arduino is an open-source physical computing platform based on a simple i/o board, and a development environment for writing Arduino software. The Arduino programming language is an implementation of Wiring, itself built on Processing.
Arduino can be used to develop interactive objects, taking inputs from a variety of switches or sensors, and controlling a variety of lights, motors, and other outputs. Arduino projects can be stand-alone, or they can communicate with software running on your computer (e.g. Flash, Processing, MaxMSP.)'
Arduino is used as part of the system I've built in order to transmit drumbeat signals from the analogue kit into the digital visualisation system I've constructed in Flash.
It took some time to get the system working properly, I've opted for a fairly short delay rate in the Arduino code that is stored in the circuit (checks are made for inputs from the Piezos once every 150 milliseconds).
I attended some Arduino workshops in the summertime last year (2006), during which I had worked along with another attendee to get Arduino and Flash to speak to one another. We managed to get it working at the very end of the workshop using the tutorials files at http://www.progetto2501.com/b/tools/Arduino/. I also attended another Arduino workshop in November '06. During the workshop I worked with one of the other students to get Flash and Arduino working together (the code ended up being used in the Hackable Curator project I believe) but I still haven't completed a proper personal project with Arduino - so this will be a first for me.
Previously I've used a similar device called a Miditron which I still have lying around in my house - but Arduino seems much more versatile. The Miditron relies on midi (surprise!) to convey data to computer systems but Flash has no inbuilt support for midi which obviously was an issue! Also Arduino is cheap and can be powered via USB meaning that whilst it's plugged into my laptop, power is not a problem. So the miditron was rejected in favour for Arduino.
The diagram on the right highlights the main aspects of the Arduino circuit. As mentioned previously, a resistor circuit board is used in conjunction with 3.5mm jacks so that the resistors do not need to be attached to the Piezos themselves. The USB port is used to power the Arduino and to send information to and from the laptop. Not only can drum beats be communicated to the Flash movie for use in the visualisations, but also if need be the Flash system could send back instructions for the Arduino to trigger LEDs / lighting, motors or a similar physical response.
As a point of reference, some examples of other work with Flash and Arduino that I came across whilst trying to find the best solution to my technical challenges include:
* Cyberdolls (which uses Arduino to enable remote web users to control two 'real world' plastic dolls)
* Wireless toys workshops (in which people make interactive childrens toys which incorporate Arduino)
The laptop I'm using enables me to feed audio into the sound card, send video out via VGA and S-VIDEO, and it has a 3G card and 3 USB2 slots. This has given me lots of scope for linking up my system to different hardware items.
Flash 8 (which is as of this month being replaced by Flash CS3) has been used to develop the software which generates and controls video output during the live performances.
D. WEBCAMS
Flash 8 (as of this month being replaced by Flash CS3) now allows you to use webcam feeds. And I was keen to incorporate live video data into the visualisation system.
I purchased two cheap but effective Creative Vista Plus webcams, having realised that driver clashes can cause probems if you use two different webcam models.
E. 3G CARD RETRIEVING DATA FROM SMS SYSTEM
After investigations into several different SMS gateway providers, I finally struck gold with TXTlocal.
They provide a free SMS gateway account which enables users to receive texts and post the messages to a URL via HTTP. This means that an SMS message can be sent to a script (such as PHP) and then stored into a MySQL database. The SMS message can then be retrieved upon demand by connecting to the website and retrieving the MySQL data entries. The documentation relating to this can be seen at
http://www.txtlocal.com/sms-api.php.
I developed a Flash movie which communicates with the main Flash movie which I use to control the visuals. It was important to use a seperate movie since I needed to be able to read the SMS messgaes properly, but didn't want to have to open up menus and use buttons which appeared on the screen seen by the audience. Therefore I opted for opening a connection between the movies with the Flash local connection object. This enables me to browse through messages in the Flash movie diplayed on my primary screen (the laptop screen), and then send the desired messages to the Flash movie used for the visuals system which is displayed on the secondary screen (the flatscreen or projector viewed by the audience). The two Flash movies below demonstrate how the system works - any messages you text to 07786 200350which begin with the letters 'ELL' will appear in the left hand system after a few minutes. You can browse using the arrow buttons, and select which messsages appear in the movie on the right by pressing enter.
F. S-VIDEO AND COMPOSITE LEADS
G. BLUETOOTH ADAPTOR
H. AUDIO LEADS INPUT
The S/PDIF and MIC jack inputs are both available for feeding signals in from either the mixing desk or from the amplifier head of the bass amp (fed from the FX-send output).
7th May 2007
Used the live visualisation system at the gig this Saturday (May 5th) and all went very well. I asked my girlfriend and a friend to film the gig, with the aim of capturing the visuals system working in the context of the live venue. They did a good job for me, even in light of the fact that my camcorder battery is obviously getting old and only lasted for 45 mins of the 2-hour long gig! So unfortunately some of the system features and animations couldn't be captured but I think what I have put together for the video displayed on the left is a good indication of how the system works under live conditions.
The main aspects the footage captures (as flagged by the subtitles at the bottom) are:
1. Input-generated visuals (animations and graphical instances generated by signals from the bass and drums - fed to the system via Arduino) 2. Peak-triggered slides (stills and mini-animations which are triggered by sudden contrasts / peaks in the audio input from the bass guitar) 3.
Live video mixing (chunks of video content which are randomly overlaid and blended together on the fly in Flash) 4. Video / animation playback (straightforward playback of pre-rendered animations and video footage) 5. Webcam input (fed from one of two different webcams into the system and incorporated into various animations)
Aspects of my system which unfortunately weren't captured in the footage are the SMS text display system, plus some of the more interesting webcam-based visuals which I'd developed in order to develop the sense of audience participation.
Some minor issues were experienced with system lag. This seems to have been due to my having to use Firefox browser (a Microsoft update seems to have meant that Explorer won't go into fullscreen mode without keeping some navigation buttons visible at the top of the screen). Although considered by many to be the superior browser, Firefox doesn't seem to handle Flash animation as well as Explorer. I was also experiencing some highly disturbing hard drive issues during setup (my laptop kept freezing and wouldn't reboot for about 15 mins!) which could possibly be attributed to the fact that I dropped my laptop on the kitchen floor during tests on Thursday evening...
I'd like to improve upon the system by building a foot-pedal which would enable me to trigger visual processing changes mid-song, without having to stop playing my instrument. I would also experiment with browser settings to get better performance during the gig on the second screen. I've come to the conclusion that setting the laptop's secondary screen (currently used for outputting to the audience) as the primary screen could improve system performance, since I would then be able to maximise a Flash projector in fullsreen mode. Flash projectors always expand in full-screen mode within the primary monitor. This means the projector would provide much better performance over an SWF being run as a browser-plugin. Annoyingly, defragging my hard drive since the gig also seems to have improved performance somewhat - even though a system check prior to the gig had indicated that my laptop drive didn't require defragging.
Video of tests using the Piezo sensors and drum triggers can be viewed on the left (press play to start the video). As you'll see in the video, drummer Zac is triggering the simple visuals using a midi trigger (red unit attached to the bass drum) and a Piezo sensor attached to the snare drum. Both these inputs are fed into the Arduino unit which then communicates (via a Server Proxy program) with the fullscreen Flash movie which you see playing on the laptop. The onscreen flashes are entirely triggered by the drum rhythms, and the system takes into account the signaldynamics - you'll see the flashes onscreen are more prominent when Zac is playing harder or using more complex rhythms. Note the bar graph readouts which flash up very quickly for both snare and drum inputs.
Towards the end of the video clips you'll also see the main drawbacks of using the Piezo sensors as opposed to the midi triggers - they have a tendency to break (or fall off if you don't tape them well enough to the drums!) and also they act like a microphone, picking up background noise such as conversations or of course other instruments. This could be counteracted by setting minimum signal thresholds which must be exceeded in order to trigger any form of reaction by the Flash movie. However my main concern is that the Piezos are rather fragile and would probably not withstand a full two-hour rock set once the band are really getting into things... Zac plays the drums pretty hard which won't bode well for the Piezos' life expectencies either!
Therefore I'm interested in acquiring more midi triggers for the drums, which allows me to concentrate more on the programming logic and animation rather than waste more time trying to make the Piezos more robust - I've already demonstrated the principal works, so I'd rather improve the visual impact of the system rather than waste lots of time upon the homemade hardware. The downside is that a full set of midi triggers for the kit (snare, bass drum, toms) is not cheap, coming in at around £100 minimum (see here).
You can see previous development notes towards this system below or in my general work blog.
14th April 2007
This weekend I have come up with a system using the Arduino system which is used to pick up piezo sensor signals and convey the signal data to a Flash movie. I intend to connect the piezo system to to my drummer's kit. This will allow me to register his drum rhythms during a live perfmance, and transmit that event data into my live visuals system - for example allowing me to use his drum beats to trigger different graphical animations.
9th March 2007
The gigging visuals system for Neg B has had a bit of a setback, as our next live gig at the Junction (next Saturday, 17th March) has been cancelled. This was due to a double booking by the venue manager. This means that we only have one more Junction gig before my hand-in, so I won't get any more trial runs as part of the development process before the system has to be completed. So, our gig on May 5th will be 'it'. This is a bit of a setback, but I'm trying to stay positive - will have to sort out another 'live conditions' trial run during a band practice or something.
Live tests so far (excluding some tests during band practices) have been:
11/10/2006 at The Junction (Mutley Plain Plymouth) - tested the feasibility of the system, assessed that it would be possible to link up with the flatscreens via the laptop
12/1/2007 at The Junction - got the first prototype system running during the gig, system worked well and visuals got the thumbs up from audience - only had a small selection of animations to use, the sensitivity of the reactive system was highlighted as needing some optimization.
16/02/2007 at The Junction - Got the system working really well, had a lot more imagery prepared and it really contributed to the success of the gig and the impact of the band. System responded really will to my signal inputs this time and the interactive visual elements got a very good response from people I spoke to. Also the system's ability to be configured at short notice by changing the XML files paid off, I incorporated some images of a group of friends from the audiences just before we went on, by taking images off a digital camera and listing them in the XML song files list.
I had a meeting with Eduardo on Monday 5th March and he's happy with the project's progress. He agreed that it's probably most important to develop the way in which the system reacts to the band's dynamics and musical output, and or the user interactivity elements. It may well be worth me getting the Arduino system working within the project so that I can monitor multiple inputs as opposed to just volume peaks from my bass guitar. I now how to give Eduardo a written report just summarising how the project's developed so far and where the projects going. Luckily the notes I've kept to date (see my last post) cover a lot of this for me.
I've also managed to find a good SMS gateway service which allows me to receive texts for free, heavily reducing the cost of developing this part of the system. I've borrowed a friend's 3G mobile card for use with the laptop at live venues - it costs £1 per megabyte of info downloaded using the card, but since I'll just be downloading text info this should be alright. All this means that I can now integrate audience SMS messages into the live visuals.
How SMS integration will work:
People can text a message to a number during the gig which is then relayed via HTTP to a URL of my choosing by the service provider, Txtlocal.com. I put together a prototye which is running at http://www.elevenlivesleft.co.uk/sms/show_texts.php - it can be tested by sending an SMS message to 07786 200350 (the message must beging with the keyword 'ELL' in order to be received by my account). Messages received are logged into a MySQL database (via a PHP script I put together) and I can then retrieve the messages at my leisure by checking a web page on my website using the 3G card during the gig. I will probably make it so that the messages are retrieved as XML so that I can then suck the messages into the Flash visuals system for use in whichever way I want. I'm toying with the idea of some sort of live audience vote taken over the course of the gig, perhaps relating to a choice of song - or maybe superimposing text messages over the visuals to increase the sense of audience participation during the gig.
If I go for the latter option I will obviously need to filter the texts somehow, either using a censorship script which checks for unwanted key words, or by selecting which messages get into the system during the gaps between songs in our set.
Reasons for developing the system:
· Need was identified for my band to develop a system which enables us to create dynamic, exciting visuals to accompany live performances, and which specifically relate to the lyrical and musical content of individual songs
· Wanted to avoid being reliant on a third-party, such as a VJ, who was not a member of the band, would charge a fee and was not likely to be able to make it to all the gigs
· Ties in with previous work as final year undergraduate and my ideas for final year project at BSc level
· Wanted to develop new interface systems (such as motion tracking and alternative input devices) as part of research towards my final project idea
· Wanted to work more with video production (including generative material) which would also tie in with my Liquid Media work (for viral marketing videos)
Research and development progress so far:
Optimum setup found over the course of the last 3 gigs at The Junction:
Flash software quickly selected for the project (enables me to interface easily with video, webpages and services, web cam devices and physical computing devices such Arduino, plus can dynamically load external files and is easy to develop animations and generative content)
Bass guitar found to be best choice of instrument for relaying musical structures to the system – bass lines reflect melody and rhythm of tracks
Signal fed from the FX send of bass-amp to S/PDIF jack on the laptop, to monitor musical activity
Laptop keys used to adjust configuration and to toggle through songs (allowing user to react to on-the-spot set-list changes)
Visuals output via S-video lead (linking to S-VHS/Composite Video Convertor, plus additional composite video leads) linked up to venue’s video system AND/OR a VGA output to an LCD projector
Main SWF developed (800x600px screen size) which acts as central control system:
Content used for each different song is loaded in dynamically (such as SWF mini-movies, JPG images, FLV videos etc)
2-part peak detection system used to monitor the activity levels of the music –upper threshold set by the user to trigger visual events, bottom threshold has to be dropped beneath before the peak triggers another event (to prevent sporadic visuals)
XML set list is used to manage XML library files and behaviours for songs
XML files and content folders used to manage each song’s content files
Setlist-parsing SWF used which runs on laptop screen (makes easier to read)
Visuals relayed to audience on a second screen (running at 800x600), output from the laptop, played in an Explorer browser window running in full-screen mode
Activity fed to sub-movies via global variables and functions
Personal observations from gigs:
Audience reactions and feedback gauged, with more interactive content found to be the most gratifying for onlookers (e.g. web cams featuring audience, or venue specific content)
For visuals to complement band’s performance rather than distract from it, abstract visuals seem to work best – excessive use of recognisable video images means people just stare at the screens
Overtly political imagery really stands out in people’s heads (not always creating positive reactions); notably political lyrics don’t really get picked up nearly as much without images to accompany them
Further research:
Observations made at Transmediale during numerous DJ/VJ sets – personal preference for simple vector graphics over random noise or seemingly irrelevant images for tracks (such as psychedelic noise, fishes, donkey kong graphics etc):
Live visuals used by bands I’ve seen recently such as U2 (Zoo TV), Snow Patrol, Muse plus at shows such as Brits 2007.
Related artist’s works:David Rokeby – Taken (02), Watched and Measured (00), Sorting Daemon(02), Very Nervous System (86-90)
“Untitled 5 is the fifth interactive installation in the External Measures Series, which Utterback has been developing since 2001. The goal of these works is to create an aesthetic system which responds fluidly and intriguingly to physical movement in the exhibit space. The installations respond to their environment via input from an overhead video camera. Custom video tracking and drawing software outputs a changing wall projection in response to the activities in the space. The existence, positions, and behaviours of various parts of the projected image depend
entirely on people's presence and movement in the exhibit area.”
Author’s description of her work Untitled 5 (2004), taken from http://www.camilleutterback.com
Adam Chapman
“Increasingly, we see nature and technology merging in unprecedented ways … I use existing and customized video editing software to deconstruct theatrical elements of cinema, such as time and choreography, to create allegorical new works with a painterly and a poetic sensibility. These pieces often begin with existing popular films and, while retaining clues as to their identity, isolate and emphasize one or
more formal filmic element by using it as an artistic tool. I also use my own footage and create looped video narratives that address such issues as creation, faith, and doubt. “In each form, my work addresses the way people perceive and relate to their surrounding world and the myriad images populating the contemporary landscape. By taming and redefining natural phenomenon through programming, projection and imbuing inanimate objects with human characteristics, my work aims
to create transcendent experiences that make viewers more aware of their mutable surroundings. “
Taken from http://www.adamchapmanart.com/statement.html
Matt Hanson in The End of Celluloid, quoting Anthony Dunne's 'Hertzian Tales':
"Screens are like 'supermatter': once switched on, all attention turns to them, and their material qualities are demoted to the status of package or container as the viewer searchs for the real content, information".
Data visualisation
Music visualisation
Audio visualisation
VJing
Live visuals
AV-performances
Ideas for developing the system further:
Integrating live audience text messages , using a wireless broadband connection and PHP>Flash SMS gateway interface in the visuals… may need some filtering to avoid negative / offensive comments being shown
(Similar to Nooby(?) from Uni LED installation last year, V Festival and
this: http://www.youtube.com/watch?v=tJgReG6MDnA&mode=related&search=, looking at http://www.24x.com/new-integration.asp gateway provider).
Idea for developing music-based narrative VJ content (e.g. part-scripted, part-generative videos which reacts to musical events during a song’s performance)
Incorporate Arduino to capture physical inputs (such as
Create a pedal-system out of a spare keyboard, to enable event-based control over the system during a song whilst using both hands (to play instrument)
Include multiple web cam feeds into the visuals (purchase additional cams)
React to audience reaction volume, band members’ movements on stage or audience’s movements offstage
Project or broadcast video footage out of the venue onto street or walls, using a wireless video transmitter
Broadcast gig onto the Internet via wireless internet
Capture additional instrument activity, e.g. bass drum and snare using https://www.tucsonwebservices.com/drumdial/product_info.php?products_id=30&osCsid=543fe4d8b5c38e116a98806dc2d5d1a6
Capture note pitch information instead of just activity (difficult in Flash)
Download images over the internet live during gig (risky!?)
System bluetooths songs to audience members
Content developed to date:
(Present notes from Steve re: song lyrics)
Gorilla’s style animations
Berlin visuals (red/angry) with scratchy paint texture and halftone patterns
Video matrices (2x2 and 4x4) with NYC video footage and celebrity faces
Environmental images
Generative vector animations (planes, bombs, cubes)
Interactive video for ‘Preaching…’ with truck and Steve’s face in video
Abstract photos
Money in the wind
Political images (e.g. political icons, Tiananmen Square etc)
Psychedelic videoscapes
Webcam feeds of stage and / or audience, with graphical overlays
Google maps zoom in on current venue to bring the event into the visuals
Machinima 3D video for Confident Man
ELL logo holder (with film noise and mini-pics)
Ideas for additional video content
CGI and wireframe vector 3d (dynamically generated in Flash?)
Use my Flash consumer production line animation from BSc FYP
This weekend I have come up with a system using the Arduino system which is used to pick up piezo sensor signals and convey the signal data to a Flash movie. I intend to connect the piezo system to to my drummer's kit. This will allow me to register his drum rhythms during a live perfmance, and transmit that event data into my live visuals system - for example allowing me to use his drum beats to trigger different graphical animations. 
The 
A quick introducition is probably required for Arduino: 
Previously I've used a similar device called a 
Flash 8 (as of this month being replaced by 
After investigations into several different SMS gateway providers, I finally struck gold with 
The S/PDIF and MIC jack inputs are both available for feeding signals in from either the mixing desk or from the amplifier head of the bass amp (fed from the FX-send output).
