{"id":164,"date":"2019-04-10T17:10:39","date_gmt":"2019-04-10T15:10:39","guid":{"rendered":"https:\/\/www.codemotion.com\/magazine\/behind-the-scenes-of-codemotion-2019s-opening-live-coding-performance\/"},"modified":"2020-01-31T11:23:23","modified_gmt":"2020-01-31T10:23:23","slug":"behind-the-scenes-of-codemotion-2019s-opening-live-coding-performance","status":"publish","type":"post","link":"https:\/\/www.codemotion.com\/magazine\/dev-life\/stories\/behind-the-scenes-of-codemotion-2019s-opening-live-coding-performance\/","title":{"rendered":"Behind the scenes of Codemotion 2019\u2019s opening live coding performance"},"content":{"rendered":"

Codemotion Rome 2019\u2019s opening performance combines dance, motion capture, live coded generative visuals and live music performance.
\nJoin us in this article to discover how we used WebSockets<\/span>, Vim, visual applications such as TouchDesigner or Blender and much more to build the infrastructure behind it.<\/p>\n

\"\"<\/p>\n

Codemotion<\/span> 2019\u2019s opening performance was such an entertaining experience<\/span> and just as inspiring was the process that we as Chordata New Arts<\/a> went through to give birth to it.
\nAs the artistic branch of the emerging tech project Chordata<\/a>, we are always confronting the new relationships between technologies, media and artistic languages.
\nThis work is powered by Chordata\u2019s homonymous open-source motion capture framework.<\/p>\n

Discussing with Codemotion<\/span> our intentions for the opening show with co-founder Mara Marzocchi<\/a>, we were both immediately intrigued by the idea of combining dance, live coded generative visuals and live music. This led to a performance<\/span> where the dancer\u2019s movements gave life to a digital body, transforming through time and space.<\/p>\n

Join us in this article to discover how we used WebSockets<\/span>, Vim, visual applications such as TouchDesigner or Blender and much more to build the infrastructure behind the performance<\/span>.<\/p>\n

A microservices-like infrastructure<\/h2>\n

Many of you might have heard of the emerging trend of Microservices. As artists, we just recently came across the concept and noticed how, interestingly enough, our systems and workflows spontaneously evolved to resemble the patterns and methodologies of that architecture, where a group of autonomous services contribute to a part of the process in a modular fashion.
\nHere\u2019s a representation explaining of how the machines involved in our system cooperate:<\/p>\n

\"\"<\/p>\n

Our systems and workflows resemble the patterns and methodologies of a microservices architecture.<\/q><\/p>\n

Each laptop is processing one conceptual and technical part of the whole computational process that powers the performance<\/span>, while exchanging informations with the others.
\nOne laptop is being used for real-time audio synthesis, another for real-time video processing and a third one for handling the motion capture data and for the live-coding input. A microcomputer attached to the performer\u2019s body takes care of managing the sensors.
\nLet\u2019s dig deeper into the details.<\/p>\n

Motion capture<\/h2>\n

The core of the motion capture system is the service running of the Raspberry Pi: the Chordata Notochord<\/a>. Its job<\/span> is to read values coming from the motion sensors, processing this data<\/span> and sending it via OSC<\/a> (a flexible protocol built over UDP, very popular among digital artists) to a laptop running Blender<\/a>.<\/p>\n

The open source motion capture framework Chordata, and 3D manipulation software Blender are used to get the live movements of a performer on stage.<\/q><\/p>\n

The real-time motion data is processed inside Blender, through an operation of forward kinematics. This sums up to a fancy way of saying that they end up representing a humanoid figure. This newly acquired data<\/span> is sent to the core visual processor: TouchDesigner<\/a>, a flow based, graphically oriented development platform.
\nOnce again, the protocol of choice for this communication is our beloved OSC.<\/p>\n

\"\"<\/p>\n

Real-time visuals<\/h2>\n

\"\"<\/p>\n

The backbone of the visual processing is created assembling a network of nodes inside TouchDesigner.
\nOne of the first tasks performed is receiving, interpreting and visualizing motion capture data. The incoming movements are applied to a 3D humanoid model. Further controlled deformations are implemented to give us the ability to move towards more abstract dimensions.<\/p>\n

Several 3D and 2D image manipulation processes are implemented as a network of nodes inside TouchDesigner.<\/q><\/p>\n

The nodes in the middle of the flow are responsible for setting the model into a 3D world with virtual lights and camera. The 3D space is then flattened into a two-dimensional texture (Rendered). This will be fed into further nodes, allowing 2D texture manipulations. You can spot in the screenshot above how all the nodes on the right side of the network are violet. In TouchDesigner, this is the color highlighting nodes working with bidimensional textures.
\nThe source image will be finally composited with the code being written and streamed live.<\/p>\n

Audio reactive visual effects<\/h2>\n

You might have noticed, in the diagram shown at the beginning of the article, how our Digital Audio Workstation communicates with the machine running TouchDesigner through OSC messages. These are numerical representation of the results of a real-time envelope following operated on the audio being produced. This allows us to achieve audio-reactive effects, using the sound intensity as a parameter for our video effects.<\/p>\n

Live coding<\/h2>\n

Although TouchDesigner being a flow based programming environment, one of its wonders is being completely scriptable with Python.<\/p>\n

The pink nodes on the left of the flow form a Websocket<\/span> server that listens for messages containing Python code to be executed. This allows us to inject arbitrary Python code written to control the operators in the TouchDesigner flow, making the live-coding approach possible.<\/p>\n

Websocket communication with TMUX and Vim<\/h2>\n

\"\"<\/p>\n

If you came from a Unix-like operating system<\/span> you might already be familiar with TMUX (Terminal Multiplexer). Apart from being a great productivity tool, TMUX is what allowed us to display the code on the screen. Under the hood it works by storing the content of every pane in an internal buffer. The contents of that buffer can be retrieved with a convenient CLI interface. We leveraged that possibility to create a simple program written in Python<\/a>, that sends the content of a particular TMUX pane on every keystroke.<\/p>\n

Code is captured from a terminal emulator session and streamed to the laptop in charge of the graphical processing.<\/q><\/p>\n

On the captured pane, we can issue shell commands, or use a console text editor like Vim to edit the code to be sent.<\/p>\n

However, when there\u2019s code there might be errors. In particular if that code is being written live to control a dynamic process. In order to retrieve the error messages from the TouchDesigner runtime a full duplex communication was vital. That necessity made clear that the adoption of WebSocket<\/span> for this task (instead of UDP-based OSC) was a no-brainer.<\/p>\n

Is that all there is?<\/h2>\n

\"\"<\/p>\n

We tried our best to give you an insightful, while not overwhelming, overview of our work and we deeply hope you can take something from it and put it in your own creations.
\nObviously there would be much more to say for each of the topic we briefly discussed. That\u2019s why we gathered some resources for you to learn more about these subjects:<\/p>\n