From the lab
The Musical Orb: a VFX shot built on the spot with AI
Six seconds of phone video and a written brief: a VFX shot whose sound draws its picture, built with Claude on the spot, down to an editable Blender file.
On frame 136 the orb hits the lens, and on that exact frame the drop lands. Nobody synced that by hand. The music was written around my throw, and the orb itself is made of the music.
I filmed six seconds on my phone. I sit in a chair, hold my palm up as if there is energy in it, then push it at the camera. I gave Claude the clip and a written brief: what I want to see, what is off limits, and how the sound should behave. Two to three hours of working together later, I had a finished shot with its own soundtrack, plus a Blender project and an After Effects project I can open and change.
Nothing here came from a preset library. It was built on the spot, for this clip.
An idea almost a hundred years old
The brief asked for the sound and the picture to be one object, not two things synced later. That idea is old. In 1932 Oskar Fischinger drew patterns straight onto the film's soundtrack, and in Ornament Sound the same patterns make both the picture and the sound.
In 1971 Norman McLaren did something close in Synchromy. He built the piano soundtrack out of shapes, then laid the same shapes, in colour, on the screen. The result, as the catalogue puts it: you see what you hear. John Whitney's Catalog, from 1961, ends on a Lissajous curve multiplied dozens of times. That is exactly the shape an oscilloscope draws when you play two tones into it.
Today's version is called oscilloscope music, music that makes its own visuals. You play stereo sound into an oscilloscope: the left channel moves the beam sideways, the right one up and down, and the music draws a picture. Jerobeam Fenderson turned it into an art form. That is where we went.
How does the music draw the orb?
The orb is a 3D line sampled at audio rate, 1,600 points per frame. The left channel of its stem is the horizontal axis and the right one is the height. Plug the stem into an oscilloscope and it draws the orb the way the camera sees it.
So every musical event is also an event in the picture. When the chord changes, the number of lobes on the orb changes: five on the first chord, four on the second, six on the third. When the riser climbs, a ring opens into a full spherical knot. In the eighth note of silence before the drop there is nothing to draw, so one hot dot is left.
That is why there is no sync step. The score is one list of events at 120 BPM, and both the sound and the picture read from it. It was written so the hit on the lens lands on a downbeat.
How does one phone become a motion capture stage?
This used to take a suit full of sensors and a studio. Now it is a product category. Rokoko Vision has done it for free from a single camera since 2023, and Autodesk bought Wonder Dynamics in 2024, whose product is now Autodesk Flow Studio.
We did not use a product. The tracking was put together on the spot, from open models. MediaPipe Hand Landmarker finds 21 points on the hand in every frame, in pixels and in metres. Three of them give the direction the palm faces, and the orb sits above the palm along that direction. A MediaPipe segmentation model marks where there is skin. That is how my fingers pass in front of the orb, and how its light falls on my hand.
Depth Anything V2 gives a depth map for every frame. Its depth is relative, so it is calibrated at two points: my face at about 1.6 metres from the camera and the wall at about 2.4. Without that, the light on the wall flickers. Last, a virtual camera with the iPhone's field of view puts every point in the right place in 3D.
The hand tracking came from an earlier fireball experiment of mine. Everything else was written for this shot.
Where did the cinema come from?
The first version worked, but it looked like a demo. That is what I wrote back: it feels like a POC, not a cinema-level effect. From there, every round started with one sentence from me.
After that note the orb became a real light source. It lights the room using the depth, casts shadows, and the lens got what a real lens does to a bright light: glow, depth blur and grain.
Next I asked for better design. A colour story came in: a cold red filament that heats to gold as it charges, turns oscilloscope green on the third chord, burns white in the silence, and cools back down after the drop.
Then I asked to move to Blender. The orb moved to Cycles, Blender's physically based path tracer. Every audio sample became a point on a 3D strand. The room was built as geometry from the depth map, so my fingers hide the orb in real 3D, and there is haze in the air that the light passes through.
My last note was about the sound: it sounded like a toy. It was rebuilt with analog synths, a recorded piano, the reverb of a real hall, taiko and thunder.
I did not touch the code. My job was to say, in plain words, what felt wrong. Claude turned that into light, colour and sound.
Why code next to Blender, and not one program?
Code has long been the glue of VFX. The VFX Reference Platform, the standard studios and software vendors line up on, even sets the Python version every year. Blender has a Python interpreter built in that can change anything in a scene. Projects like BlenderMCP already connect Claude straight to Blender, so it can build scenes from a chat.
Here the code did more than drive Blender. Every step went to the tool that fits it. Blender got what it does best: light and rendering. Code got what no program has a button for: tracking with MediaPipe, depth in metres, and a synth whose sound draws a shape.
Code is also what joins the tools. It carries the tracking into Blender, the score into the compositing, and the same data into After Effects. That opens up combinations that do not exist inside one program. On the next project, the most important part may not happen in a familiar tool at all, but in code and a model like MediaPipe.
What do you get at the end?
One command takes the clip from the phone and writes five files: the finished shot, a breakdown video, the orb's audio stem, a Blender project, and a script that builds the composition in After Effects. On a four-core machine it takes about two hours, most of it in Cycles.
The Blender file is made of plain Blender tools. Geometry Nodes load each frame's audio samples and turn them into strands. Another node tree builds the room from the depth. The light, its colour and the focus are keyed on every frame. You can open it, change the thickness of the strands or the colour of the light, and render again.
Two things are not proven yet. So far the After Effects script has only run in EffectCraft, an open-source reimplementation of After Effects, not in After Effects itself. And depth from a single camera is an estimate, not a measurement.
What is the difference from a preset?
A preset is made before your clip exists. It does not know when you throw or where your palm is. Here the score was written so the hit lands on the downbeat, the orb sits on my palm at a distance measured from it, and its light falls on my wall. The drop is on frame 136 because the music was written for it.