A Colour Keyboard

A keyboard that allows you to play coloured squares and full screen colour filters. The premise is to have a keyboard that responds to the different frequencies of the keys by changing colours on the stage

The MIDI controller is an Akai LPK25 mini 25 key keyboard. I’ve used colour dot stickers to give a visual for the colours assigned to each key and stickers with a raised profile on the first octave white keys to give a visual and tactile prompt (as the two octaves do different things).

In Isadora, a single colour is assigned to each note of the octave. The first octave alters the colour (fixed), size (fixed) and position (random) of the square. Squares fade in/out (from 0 – 70% opacity) with a 5 sec sustain. The second octave alters just the colour of the square/full-screen filter with a fixed opacity of 20% – the extra C on the right toggles between full-screen and a centred fixed-size square for the second octave keys. A single button on left toggles the second octave visuals on and off (I do need to figure-out a way of covering the other left hand buttons as you could stop things from working by shifting the key up and down).

Playing possibilities

The instrument can be used to play laid over the base layers of video/images or over a plain colour base layer

• Use just the first octave to play the squares
• Use the second octave set as a fixed background square that you have colour control of, to add the first octave squares to
• Use the second octave set as a full-screen filter and play the first octave squares over this
• Use the full screen filter alone to play a colour filter over the layers below

A Drawing Tablet

An instrument to give a pen and palette of colours to allow freehand drawing onto the projected image.

The hardware is a Huion Tablet, with wireless pen and 12 single press buttons. This makes a serial connection with Isadora. The software treats the pen movements as a mouse and the buttons as computer keyboard buttons.

In Isadora, drawing happens when pen touches on the tablet only and the mark opacity is set at 30 % so the marks sit on top of the base layers, the mark width is affected by the speed of your drawing the mark (faster = thicker) The buttons give a ten colour palette plus two reset buttons – one to erase the last mark and one to erase all marks. There’s a little white dot to show where you’re about to draw a mark on the projected output – without this it’s hard to imagine where you’re actually about to draw on the screen. The patch does run quite resource intensive once you have lots of marks, so it works best when you make fewer continuous marks rather than loads of little marks.

I do think this instrument brings another way to add visual elements to the jam and I’m pleased with that, I’m not a big drawer myself so it will be interesting to see how others chose to use it during the jam sessions.

I want to focus on getting each instrument to be as simple as possible in terms of the Isadora set-up and also what the physical interface looks like. In Isadora they need to run using as little computer resources as possible, so everything runs smoothly together. As physical interfaces they need to be easy to understand and access – I feel like it’s important to make them friendly looking, like something anyone can have a go with.

I’ll include images of the Isadora setup, the physical interface and an example of the output, as I talk about the instruments. At this point, I’m looking to see how I can put things together and identify any potential problems. The output of the jams is to video which will be projected in the jamming space. All thoughts and comments (including ‘What on earth do you think you’re doing?’) warmly welcomed. Here’s the first two instruments:

The Video Mixer

Mixing layers of video is core to my own practice, so first up I wanted to make an instrument to mix videos. This is an instrument to mix 8 videos. The videos are short loops – so the imagery is focused on symbolism, colour, form and texture, rather than narrative, although short spurts of narrative action would be possible. In terms of participatory projects, the idea is to be able to load participants video into the instrument – after I’ve re-encoded it to play nicely in Isadora.

The hardware is an Akai LPD8 with 8 pads and 8 swivel knobs (this controller has useful software to alter the functions of it’s pads and knobs). I’ve added number stickers to the pads and knobs, as I found myself counting to ID controllers when I was trying the interface out. I do think this helps and I also plan to make a printed layout card with space for a thumbnail still printout from each video so the player has a real world visual identifier for each video.

In Isadora I’ve set-up the knobs to control the opacity of each video and the pads to switch a Desaturate filter on and off (fully saturated / 50% desaturated). Once I start to explore using the instruments together, I’ll need to consider where the videos sit layer wise with the other still image and generative elements. Building this has also highlighted that running eight videos at once is resources intensive, as I go forward I’ll need to consider options around reducing the resolution of the videos or reducing the number of videos in play at any one time.

The Image Mixer

An instrument to mix 8 still images. Here I wanted to make something that can enable collage creation, I wanted the images to be able to sit on top of each other, but also alongside each other. Like the videos, participants will be invited to get their images into the instrument for jamming. I make the images 1920 x 1080 px at 150 dpi (to allow for zooming)

The hardware is a Korg NanoKontrol2 with 8 sliders, 8 knobs and three press buttons for each slider/knob. This controller also has some buttons on the left hand panel which I may bring into play as I experiment but I haven’t as yet – I may decide to cover these buttons somehow if they’re not going to be used, to simplify the interface. I’ve numbered each track (or image) for clarity.

In Isadora I’ve set-up the sliders to alter the opacity and the knobs to alter the size of each image (from 30% to 200%). To discourage too much intense focus on exact positioning, I’ve decided to make the position selector a button (‘R’) that selects a random position. The ‘S’ button switches blur on and off and the ‘M’ button switches a 50% Desaturation filter on and off. I’ve made image 1 a base layer image that can’t be zoomed out beyond 100%, so that, as long as this instrument is being used, this image will form the canvas.

Our current focus in the studio is on developing a tool to facilitate audio visual jamming. This is something we started to explore with the Lochgelly project last year. That project was our first time using Isadora software to make a space to enable interactive exploration of sound and video. We had some MIDI controllers and webcams as controls and live video/sound inputs into an Isadora setup with the output projected onto a wall. It made for an exciting and chaotic space. We only managed a six week run of open studio afternoon sessions before everything closed down. And not being able to work alongside people since then has meant everything’s been on hold with this project for a year. A year when we’ve been working with care homes to develop resources to support front line care staff to get co-creative stuff happening themselves with our Jam Today project (but that’s another story).

Moving Forward

With spring 2021 upon us and the hope of things opening up once more later this year, now seems like a great time to get back into figuring out how to use Isadora to make a functional and accessible A-V jamming tool to use in sessions. Our initial sessions in Lochgelly highlighted for me the importance of creating some sort of clear order for the jamming to happen inside of. For these sessions, I’d created minimal order, focusing instead on maximum possibilities – all that lovely chaos and fluidity, but it is possible to have too much of a good thing. When you want to make an activity accessible and friendly, order does help. The people who joined us for a jam did join in and have a good time, but I do think that I can improve things – both in terms of the co-creative experience, and the audio visual stuff we all make together – by making something more structured. Also, it was all a bit exciting (stressful) for me never quite knowing how everything with the hardware devices and software would interact together as we delivered the sessions. Time to embrace a little order.

The Back Story

In my practice, I create video and sound work using other people’s software and I’ve long been curious about trying some creative coding myself. I do plan to explore some coding – I’ve learnt HTML and CSS to build this site as a first step and I want to learn about Processing and explore the possibilities of building interactive input devices using Arduino. This is a longer term plan and I’m not prepared to wait until I’ve learnt to code to be able to dig in with creating our jamming tool. So, this is where Isadora comes in. The idea behind the software – created by musician Mark Coniglio – is to offer creative practitioners a tool to put together creative software spaces without the need to code things themselves. Ideal for me, right now.  

The Plan

I’m going to write (at least) weekly here on the blog as I develop the tool, as a way to think things through as I learn and experiment. Isadora works by giving you building blocks (actors) that interact with each other and with inputs from devices (like MIDI controllers and drawing tablets) and sources (like webcams and mics). It’s all about interaction. It offers a massive range of possibilities and I know that a big challenge for me will be to reign-in everything I’d like to build – and so avoid a big pile of controllers and wires and the possibility of creating stuff that looks and sounds just messy. So, the plan is to focus first on building individual A-V jamming instruments that are stable and accessible. What I’m looking to do is create instruments that will impact the projected video output in real time. Some will also have an audio element. The video imagery will consist of layers of recorded video, still images, live video input and generated elements – you see the potential for chaos! Pre-recorded video, still image and audio resource slots will be available to load participants material.  The musical input into the jam will mainly happen live in the space with acoustic instruments.

The Potential Instruments

Here’s the wish list for now (I’ll write here about them as I put them together):

• A Soundscape/Texture Keyboard – using a small 25 key MIDI keyboard to trigger individual (looped and single-shot) sounds that can be played together in real time. The keys will also trigger still images to create a visual texture layer.
• A Coloured Shapes Keyboard – using a small 25 key MIDI keyboard to trigger and manipulate generated coloured squares, altering size, colour, position and opacity.
• A Live Drawing Tablet – using a tablet with pen input and programmable buttons to draw onto the video output.
• A Picture Mixer – using a Korg NanoKontrol2 MIDI device to mix 8 still images – altering size, position and opacity of each, output to a picture layer.
• A Video Mixer – using an Akai LPD8 MIDI controller to select and mix 8 videos.
• A Sharing Stuff Webcam – using a webcam to offer the ability to share things visually in real time, output to the screen

I also want to explore how to use webcams to capture sound and movement in the space and how this may impact on the visual output. And how to use mics to allow specific directional input of sounds in the space and how these may impact on the visuals. All quite vague with these as of now. I do have a Kinnect movement sensor that I’ll be having a play with too – apparently these work well with Isadora.

Early days, but keen to get cracking.

Find out more about Isadora on the Troikatronix website here:  https://troikatronix.com/

Emma