Image Source: The Mill
The Chemical Brothers’ video for the single “Wide Open” has been making the rounds. It depicts a barefoot woman dancing through an empty industrial space as her body parts are slowly replaced by see-through 3D-printed material. Watching it, I was amazed to see special effects that I couldn’t figure out immediately. I kept asking, How’d they do that?
Ask and ye shall receive. The Mill has posted a blog post on their site explaining exactly how they “turned wold renowned dancer Sonoya Mizuno see-through.” And wouldn’t you know it, 3D scanning was a central part of the process.
If you haven’t seen the video yet, I recommend watching it before moving on.
A note: In their admirable attempt to keep the post short, The Mill has produced an explanation that leaves out a few of the most important steps. I’ve dug into a few other resources kicking around the web and incorporated bits and pieces here to help clarify things.
Photogrammetry for Capture
From the “making of” video attached to the blog post, it looks like the first thing they did was scan Mizuno with a 360-degree camera array. Next they processed those shots with photogrammetry software to generate an exact digital replica of her body, complete with very high resolution textures.
Next, Mizuno danced. They filmed her moving through the choreography a number of times, using a steadicam and a number of GoPro cameras. They also had her run through the choreography while wearing a motion-capture suit.
These two things combined gave the 3D animators a reference point for hand-animating the see-through body parts in all 6,000+ frames of the video.
LiDAR Makes Her See-Through
One of the great challenges of the video, the animators explain, was to “paint out whole parts of Sonoya’s body to reveal a clean background.” This is where LiDAR comes in.
As part of the preparation work, the team had hired a contractor to gather high-definition geometry of the room with a LiDAR scan. The plan was originally to re-create the camera’s movement without the dancer present and then project the image frames onto the LiDAR geometry when necessary to get a clean background. But there were problems—no camera operator could re-create the take with the precision necessary, and since they were in England, the weather never stopped changing and so the lighting never remained the same between takes.
Luckily, someone on the team was able to write a tool that automatically erased any given section of Mizuno as long as it had the camera’s 3D position and the set’s geometry. The LiDAR data provided both of these bits of information and saved the day.
“The tool would scan the footage forwards and then backwards, then 3D project the last healthy section of the plate before that area went behind the dancer,” the Mill’s co-3D lead artist explained.
LiDAR for Lighting and More
In helping to track the precise position of the camera in 3D space, the LiDAR scan offered a few other important benefits.
For one, it helped them to be sure that they had Mizuno in exactly the right place in the frame at all times. “Matching the dancer,” The Mill’s blog post explains, “would’ve been a whole lot harder if we couldn’t trust the camera track.”
When the team created one of the clean plates discussed above, the LiDAR scan gave them a skeleton on which they could lay HDR images drawn from any take they wanted. This meant that the clean plates had good, realistic, lighting that the team at The Mill could vary to look perfect.
In Other Words
A lot more went into the making of this video than I can fit in a short blog post. It took a ton of patience, ingenuity, technical expertise, and technique. I’ve done the best I can to summarize how important 3D imaging was to The Mill’s process, but I’ve only scratched the surface of their work.
Be sure to check out the video below for some visualizations of the painstaking process, and read The Mill’s original post for a deeper dive on the technical and artistic intricacies of this amazing video. For an even deeper dive, I recommend checking out this explainer at fxguide, which was invaluable in helping me understand this process.
