When Season 12 Attacks: CGI 3D Face Tracking
Well, they say everything has a good reason, and indeed, there is a good reason for a creepy, eyeless Peter Sam face to be on the front of Gauge 1 James:
When I was developing the idea of using the Bandai models for a new series, I knew that a 3D printed face would simply not hold up on screen in 10-bit 4:2:2 4K, nor could my sanding or painting skills make the grade either.
But I did remember Season 12.
Thus, if Nitrogen Studios could successfully track a 3D face onto a video in 2D space, why couldn’t I give it a go? Armed with a reasonable understanding of the concept of 3D tracking in After Effects and 3D modeling (from my Train Simulator days @ www.klwtrainsim.com), I figured on giving it a shot.
And this is where the mismatched face funkiness starts: Seeing as I was treading new ground for myself, I chose to use Gauge 1 James as the guinea pig for the test. The size had already proven workable by Nitrogen, while the tiny Bandai models would be something else entirely. So James was given a face of motion tracking markers, and some very wobbly pan-and-dolly footage was shot to test the limits of the 3D tracker’s ability and accuracy.
Just the same, I knew I was working with a simple, single plane to which the 3D face had to follow and “attach” to – at most, five motion targets could give the 3D object a rough spatial reference to its necessary position. It didn’t have to be as complicated as Blender wanted to make it. So I started seeking other 3D programs that could import the motion data from After Effects.
I knew Autodesk Maya had been used for the all-CGI seasons – and I seriously considered using it – but before doing so, I was lucky enough to happen across a few YouTube tutorials demonstrating perfect motion tracking with Maxon’s Cinema 4D – and a “Lite” version of the program is included with After Effects.
Thus, after a few failed attempts, I was able to get a perfect track of the 2D video into Cinema 4D’s 3D space (that’s a mouthful, isn’t it?). Shortly thereafter, I was able to export Nick’s Peter Sam face out of Blender and import it into Cinema 4D, where I was able to line it up with the track points. For the sake of simplicity and troubleshooting, I didn’t add any shaders, reflective properties, or depth of field/focus adjustments. A grey texture to the face was the only thing added.
Though the export of the finished composition from After Effects was excruciatingly slow (2 hours for nine seconds – I did not perform the export on my edit bay and performance suffered as expected), the result was a perfect, 10-bit, 4K track of the 3D Peter Sam face onto Gauge 1 James.
As soon as I have time to learn and perfect the shading and material properties in Cinema 4D (and animation in Blender), I’ll assemble a tutorial for the community.