Synchronizing animation to MIDI note data

What is MIDI?

Read about MIDI controller on Wikipedia in case you don't know what MIDI does or can do for you.

The MIDI-protocol has been developed in 1982/1983 by a group of American and Japanese synthesizer manufacturers, to interconnect musical instruments, dedicated processing devices, and computers. It is also used to control theater lights, and to program the punched cardboard organ books for street organs.

You can even add MIDI to an existing traditional Grand Piano by placing sensors at each key on the keyboard and send the sensor data via a MIDI interface to a computer or a dedicated recording device. The advantage is that you keep the mechanical feeling of the piano keyboard while adding an entire digital infrastructure.

Why using MIDI in animation?

For an animator, animating a character who plays the piano is an elaborate process. There are so many different notes that must physically match exactly at the right moment.

The solution is to use MIDI to let the 3D model of the piano follow the performance of the original artist, this guides the animator so he/she will never plays the wrong note or needs to deal with synchronization problems. A video reference can be used to assign the correct fingers to the notes, so from now on the animator can focus on the functional- and emotional parts of the action.

In Pixar's Soul this technique has been used to guide the animators, and help them to animate the right controls on the virtual keyboard on exactly the right moment, and it was my source of inspiration for my Legato MIDI Keyboard plugin:

Of course the use of MIDI is not limited to piano players. How about the animation of a 3D model of a Pianola or a Street Organ? Or theater lights, fireworks, or any other physical behaviour that needs to be triggered in sync?

Every type of motion cycle that needs synchronisation or needs addressing the right item can benefit from MIDI.
In real life or virtual.

How to pour MIDI data into your animation?

Pixar had its own TD to pave the path but now there is Legato MIDI Keyboard, my latest (free!) LScript plugin in the Legato suite. I wrote it myself.

Basically it decodes a text formatted MIDI file, interprets the Note On/Off events, translates the events and their Velocity into a realistic angle change in time, and copies the result to specific parts of the 3D piano keyboard model.

Apart from that, it has some more advanced features: keyboard type presets (defining range, speed, electrical switching point, velocity handling), a Null keyboard generator including a test pattern generator, track selection, transpose options (to fit a performance on smaller keyboards or visually separate left/right hand performance), hide/unhide during Note On cycles (for example to enable color animation), curve type settings for Note On/Off and Dissolve channels, and audio file handling.

It probably sounds more complex than it is. Simply continue reading to get an idea. I also created a video, see Vimeo link at the bottom. If you are using LightWave 3D, you can use the download link, just below the Vimeo link.

Timing is key

Preparing your MIDI data stream

Extracting MIDI note data is not a basic feature of 3D animation software. That's why I wrote my own plugin for LightWave 3D, and it is part of the Legato suite: a set of LScript plugins that should make your life as an animator slightly less miserable.

So, if your 3D character is playing a (MIDI enabled) musical instrument, you should be able to record the MIDI data with your favourite MIDI software, mix that recording session with other MIDI- and audio tracks, and export one or multiple tracks to a MIDI (*.mid) file.

In case you don't have MIDI software yet, read about Reason for some inspiration. It is not free, but it is comprehensive and reliable.

To get that MIDI data stream into your LightWave scene you must transcode this file into an Allegro (*.gro) file with the help of Audacity (version 2.4.2 or later will do just fine). It is free open software.

The Allegro format is readable by humans in a text editor, and this is the main reason I started my plugin development project with this format. It's a small additional step and for now I'm not sure it is worth the trouble to redo the development process for binary MIDI files.

To make the keyboard key movements audible, don't forget to link the MIDI data to a virtual instrument, combine and mix it, and export an audio file.

Preparing your 3D scene

The Legato MIDI keyboard plugin works in 6 steps after loading your keyboard object, with mesh data of each individual keyboard keys placed in object layers, or keyboard keys in separate objects.

Let's take a look at layer "key_60" in the white box. This is a layer of object "Hammond_XK-3c", and it is referenced with "Hammond_XK-3c:key_60".

In case of layers you should use ident "Hammond_XK-3c:key_", the name of the object and the layer separated by a colon (:) symbol, so the plugin is able to find the numbered key layers. It also needs a common parent to process all child items.
In case you stored the individual keys in separate objects you can simply use "key_" or similar, but remember to manually create a common parent or the plugin will fail to find the mesh for each key.

Preparing your 3D scene

The Legato MIDI keyboard plugin works in 6 steps after loading your keyboard object, with mesh data of each individual keyboard keys placed in object layers, or keyboard keys in separate objects.

This is the "Legato MIDI keyboard" plugin interface. It's a Master Plugin, so after installation of the plugin you can press Ctrl+Q to add it to your scene. Double click it to get started. Settings are stored for reuse, and saved in the LWS scene file. You may leave the dialog open. Fractional Frames is always enabled for the highest accuracy. To reduce random phase errors at rendertime, consider Motion Blur.
  • step 1 - Keyboard
    Set the Keyboard type. This defines the range of target channels that should be processed, the angle of an active keyboard key (a piano has usually longer keys than an organ/synth), and the default speed (in seconds) at which a key is pressed. The duration of each transition is set to a default time value, but can be modulated by the Velocity signal that is sent by the MIDI keyboard. Most keyboards support Velocity at note-on, not at note-off. Therefore Velocity information is copied to the note-off event, but you are able to override this method and use the default Velocity for the note-off event.
    The Electrical Swithing Point defines the moment when a key is recognized being pressed. A Hammond B3 type of keyboard switches very early, while a regular piano usually responds slightly later. These preset values are estimates and may be optimized later.
  • step 2 - Ident
    Set the Ident field: the text string that identifies the keyboard key 3D Item by name. In case you didn't put them in separate object files but in object layers (in one object file) you need to manually add the object name, separated by a colon (:) symbol.
  • step 3 - Initialize MIDI Envelopes
    Then generate the MIDI envelopes for all available keyboard keys (or other type of item you want to simulate), and create a neutral key in that envelope at frame 0. You need to use this button after changing the Ident or when you reset the MIDI Envelopes.
  • step 4 - MIDI Statistics
    Before MIDI data is imported, you MUST select the "Track ID" and press button "MIDI Statistics" at least once to get additional information. For correction purposes, the plugin requires track length information in advance.
  • step 5 - MIDI Data -> MIDI Envelopes
    Import MIDI data into the MIDI envelopes, and create two keys for every change from note-off to note-on events. If you selected the correct track, you may continue with button "MIDI Data -> MIDI Envelopes". This is an additive process, so you can combine tracks into a single keyboard animation. If you've made a mistake you need to reset the MIDI Envelopes and start again with the first track you want to bounce to the temporary MIDI Envelopes. Extremely short events or overlapping events are corrected to get a realistic result. You can also Transpose the selected track to ensure it fits your virtual keyboard range, and set a "MIDI Start Time" for synchronization. The keys that define Note on/off transitions can be Linear or Bezier (default).
  • step 6 - MIDI Envelopes -> Hierarchy
    Select one of the keyboard keys or their parent. This is necessary to aid the algorithm with which all remaining keyboard keys can be found. Press button "MIDI Envelopes -> Hierarchy" to copy the MIDI Envelopes to the 3D Items you want to move: a change in Position, Rotation or Scale. The target channel is defined by the "Bind Channel". By default it is set to the Rotation.P channel.
    You can even add another envelope "Dissolve" to hide (or unhide) a keyboard key. Use this to change the color of an active keyboard key. The keys that define Note on/off transitions in the Dissolve channel can be Linear (default) or Bezier. Set Dissolve Item to None if dissolve isn't needed.
    Press button Clear Bind Channels In Hierarchy to remove the current animation of your 3D keyboard model.
    You can define an audio file (*.wav) and apply it to the Layout timeline, or Clear it.

Review the result

I should mention that this 3D model of a Hammond XK-3c (a digital component for building a detailed Hammond B3 emulation) is far from complete. I'll update it as soon as there is an improved version ready.

Now render the result and combine it with the audio version of your performance.

The demo below skips frames to reduce the file size. Therefore short note durations look slightly weird.

In case you only wanted to animate a Pianola or a Street Organ, you are ready!

If you need a 3D character to play this instrument, there is a lot more work to be done. A video recording of the performance will help to select the correct fingers of your 3D pianist. Don't forget the functional- and emotional parts of your animation!

Watch the entire procedure on Vimeo:

Download the latest Legato MIDI Keyboard plugin from the NewTek forum, or from my homepage: Legato for LightWave 3D.

Have fun!


© Copyrights 2005- by V.D. Mesman, Amsterdam, The Netherlands, EU. All rights reserved.