When Season 12 Attacks: CGI 3D Face Tracking
Well, they say everything has a good reason, and indeed, there is a good reason for a creepy, eyeless Peter Sam face to be on the front of Gauge 1 James:
When I was developing the idea of using the Bandai models for a new series, I knew that a 3D printed face would simply not hold up on screen in 10-bit 4:2:2 4K, nor could my sanding or painting skills make the grade either.
But I did remember Season 12.
Thus, if Nitrogen Studios could successfully track a 3D face onto a video in 2D space, why couldn’t I give it a go? Armed with a reasonable understanding of the concept of 3D tracking in After Effects and 3D modeling (from my Train Simulator days @ www.klwtrainsim.com), I figured on giving it a shot.
Fellow Thomas enthusiast “Festive Nick” @NorrambyBranch was kind enough to lend his Peter Sam face to me to get the project jump started, and I downloaded Blender to work with the 3D model.
And this is where the mismatched face funkiness starts: Seeing as I was treading new ground for myself, I chose to use Gauge 1 James as the guinea pig for the test. The size had already proven workable by Nitrogen, while the tiny Bandai models would be something else entirely. So James was given a face of motion tracking markers, and some very wobbly pan-and-dolly footage was shot to test the limits of the 3D tracker’s ability and accuracy.
And so the footage was in the can. Initial tracking attempts were done in Blender, and every single track failed miserably. After a few days of relentless frustration, I finally tried After Effects’ 3D Camera Tracker as an alternative, with a bit of JavaScript to import the track into Blender. And while the tracker worked perfectly, Blender’s camera sensor and lens focal length settings would not accurately match the track points – and I suspect that the optics of the periscope/snorkel lens caused these issues.
Just the same, I knew I was working with a simple, single plane to which the 3D face had to follow and “attach” to – at most, five motion targets could give the 3D object a rough spatial reference to its necessary position. It didn’t have to be as complicated as Blender wanted to make it. So I started seeking other 3D programs that could import the motion data from After Effects.
I knew Autodesk Maya had been used for the all-CGI seasons – and I seriously considered using it – but before doing so, I was lucky enough to happen across a few YouTube tutorials demonstrating perfect motion tracking with Maxon’s Cinema 4D – and a “Lite” version of the program is included with After Effects.
Thus, after a few failed attempts, I was able to get a perfect track of the 2D video into Cinema 4D’s 3D space (that’s a mouthful, isn’t it?). Shortly thereafter, I was able to export Nick’s Peter Sam face out of Blender and import it into Cinema 4D, where I was able to line it up with the track points. For the sake of simplicity and troubleshooting, I didn’t add any shaders, reflective properties, or depth of field/focus adjustments. A grey texture to the face was the only thing added.
Though the export of the finished composition from After Effects was excruciatingly slow (2 hours for nine seconds – I did not perform the export on my edit bay and performance suffered as expected), the result was a perfect, 10-bit, 4K track of the 3D Peter Sam face onto Gauge 1 James.
As soon as I have time to learn and perfect the shading and material properties in Cinema 4D (and animation in Blender), I’ll assemble a tutorial for the community.