Joystick piano

For this project I wanted to create a synthesizer that would be appealing to both musicians and to people with no prior music background. I also wanted it to feature interesting way of control different from well known and explored keys-and-knobs layout. Joystick piano allows you to play music instantly.

Design

Taking inspiration from my some of my previous projects in which I have already tested this idea successfully I wanted to include a controller giving ability to control parameters on a XY plane. I wanted to use it to control pitch, volume, expression or various parameters of the audio engine. Talking from experience I can say that adding something as simple as a two axis game joystick to any synthesizer creates whole new world of opportunities. I have already made few attempts at integrating motion control similar to how theremin works – using Leap Motion in my audio creations.

I also have thought of using an ribbon controller with position and pressure sensitivity. I thought that this could be a good alternative to a keyboard for controlling the pitch of an instrument. In order to incorporate even more expression control I thought of adding and soft squish-able layer to a joystick handle. This layer would be capable of transmitting continues pressure data. For the last iteration I went even further and I came up with a design incorporating gyroscope sensor inside of a soft squish-able ball. Motivated to avoid using a musical keyboard for pitch control I decided to try going the motion control way.

I tried to avoid spending an excessive amount of time on building a wireless MIDI controller with gyroscope from scratch. As I wanted to utilize position sensing I also wanted it to be sturdy and self-contained. This way it would not come apart as user would be moving it around. I have decided to go with an off-the-shelf device that already fulfils those requirements – a wireless gamepad with build-in gyroscope.

Joystick

DualShock 4 is advertised in some places as a 6 axis motion sensitive controller. In theory: 3 axis accelerometer and 3 axis gyroscope. Even though its motion sensor gives 6 separate values I was not able to find a software that would produce proper gyroscope yaw readouts. Eventually I decided that two parameters – gyroscope pitch and rotation – will be enough for my joystick piano project.

App for that

Most of the smartphones that are currently available on the market feature a range of sensors. Before acquiring a game controller I have it conducted a test using a gyroscope build into my smartphone. It turned out that the results were unsatisfying. The evaluation was made using an app that was capable of transferring position data over local network to a computer. This solution was also in favour of making the project more accessible since almost everyone owns a smartphone nowadays. Unfortunately the response latency of around 100ms consequently made this solution practically unusable.

Development

My goal for the joystick piano project was that it could have a stand-alone (computer-less) form. I wanted to make a use of my Raspberry Pi 3B with Pisound shield (providing audio and MIDI I/O). My another goal was to program it entirely using Pure Data. I also wanted it to be able to produce visuals that amongst being a nice addition to the audio would also provide feedback of the parameters (controller rotation).

Two platforms

The development begin on the RPi mainly because it was a target platform. Soon I moved it to Windows since self contained laptop is much more convenient for me to carry around. Theoretically the project does not utilise any uncommon hardware of software components and it should transfer between platforms almost seamlessly. I also had in mind an objective that people without a RPi could make use of my code. In result of my approach two interesting challenges presented itself.

On the Linux platform the gamepad worked almost straight-away. All I had to do was to connect the controller using Bluetooth (or USB) and install the pd-hid package. Rest of the work was a matter of configuration in the Pure Data patch. DualShock 4 showed up on the list as three separate HID devices: controller (buttons and sticks), touchpad and motion sensor. Using a [hid] object I was able to receive and process data from the gyroscope.

I found out soon that this process was not as easy on Windows. Initial steps were similar: connecting in to the computer and installing hidio – an external Pd library. The problem was that the gyroscope data was not being transmitted. After an extensive research I have found that people interested in using DS4 for PC games also came across this issue and came up with a solution. Members of gaming community created a software that was capable of emulating Xbox controller using DS4 as input. This software also allowed to remap gyroscope data to one of the joystick axes. I have decide to use software DS4Windows for this purpose.

Three synthesizers

Initially I wanted joystick piano to be made of two synthesizers: one playing monophonic melody and second doing chords. Each of the two XY coordinates would control pitch of one of synthesizers. Later on I have decided to add one more playing bassline, linked to melody synth. All notes depends on globally set music key and master clock BPM tempo. Each synthesizer receives different divisions of the master clock. I configured controls in a similar fashion to my wobble bass patch. Coordinates values (X or Y respectively) are sampled and passed to the synthesizers on each received clock tick. Pitch data is also being scaled and quantized beforehand inside each synthesizer. I have designed custom logic system dedicated to control chords with quantizer switching between preset triads.

Audio engine

My focus was more on control and sequencing aspects rather than the audio capabilities. From the beginning I wanted all synthesiser to have the same audio engines. Sound engine was evolving along with the development of other part of the project. I have started from creating a subtractive synthesis based piano sound. At some point I have decided that it does not sound convincing. I then have tried to make a simple physical modelling synthesis solution but came to the same conclusion. My last idea was a frequency-modulation synthesis attempt and it finally satisfied my needs for E. Piano sound. I made it using 4 FM operators. There are no controls for sound engine parameters.

Range of effects helps putting together sound produced by all synthesizer parts. Voices playing chords are spread over stereo field using a series of amplifiers and three off-setted low frequency oscillators. In the next stage the melody synth is being added to the mix and all combined are going through a stereophonic chorus effect (from my ZD synth). After that bassline is added to the mix and all together go to a tremolo and reverb effects.

Visuals

I have designed visuals in a way that each synthesizer is linked to a different part of the created image. Controller rotation is represented by position of a white dot moving up-down and left-right respectively. Dot changes size along with melody part amplitude. Circles on tips of the triangle outline represents amplitude of the chord synthesizer. Shape of the edges of the triangle represents wave shape of each of the three voices. Size of the blue triangle corresponds to bassline amplitude. When creating the visuals I did not take into account RPi hardware limitations. This compact computer does not have enough computational power to handle them without overheating. I have decided to remove them from the Linux version.

Find this patch on my github (and RPi version)

DS4Windows – gyro2 preset