4D Facial Animation
-
Markerless 4D Facial Animation
This example outlines how Dimensional Imaging's 4D (3D video) systems can be used to create incrediblly realistic, extremely dense, Facial Animation, without the use of any markers or special makeup.
A number of different tools and technologies are demonstrated in this example, including: automatic tracking of landmarks through a sequence, an automatic 'batch' version of our Shape Transfer Tool and the Material Transfer tool.
The output is usually a Fully textured Mesh file, plus a PointCache2 file that contains the animation of the mesh's vertices. This can be easily loaded into most popular 3D modelling and animation packages like 3D Studio Max, Maya, SoftImage, Blender, and so on. We also support the C3D format which allows the output to be used as traditional Motion Capture data in VICON's software applications and MotionBuilder.
Click here to view a movie of sample output from the markerless 4D Facial Animation process.
Click here to view a movie of sample output from the markerless 4D Facial Animation process and the one of the original inout video streams, side by side.
Click here to view a full movie of the process of creating markerless 4D Facial Animation. -
Step 1: Capture Stereo Video Streams
Using one of our 4D systems to capture the actor's performance, the input to the Facial Animation process is a stereo pair of synchronised grayscale digital video streams of the scene. Optionally, a third synchronised color camera can be added to capture the colour surface texture of the scene.
Notice that there is no use of structured light, markers on the face or any kind of makeup used when capturing the video streams.
Click here to view a movie showing a sample of synchronised video streams. -
Step 2: Reconstruct 3D Surface And Track Pixels Through Sequence
Using our proprietary passive stereophotgrammetry, the video streams are automatically processed for two things:
1 - 3D Surface: Each frame of the sequence is treated as a separate stereo pair of images and is processed to produce a separate dense range map that allows us to recover the 3D surface of the subject in that specific frame.
2 - Pixel Tracking: Each frame of the sequence is processed to determine how each pixel in one frame relates to it's neighbouring frames, this allows us to track any pixel location from one frame to the next throughout the entire video sequence.
-
Step 3: Review Raw 4D Sequence
Once the processing stage has completed, a dense range map for each frame of the 4D sequence is created. Each range map can the be polygonised so that it can be displayed in 3D. We call this a 'Raw 4D Sequence'.
We use the term 'raw' because each frame has a discrete 3D shape and mesh topology, and is usually fairly random and heavy weight, so at this stage it is suitbale for review and analysis purposes, but not for direct use in facial animation.
Click here to view a movie showing a sample raw 4D sequence. -
Step 4: Track Landmarks through The Sequence Automatically
In order to use raw 4D sequences to produce facial animation, discrete points must be followed, or 'tracked' throughout the 3D sequence. This means that the XYZ location of each point must be found in every frame of the sequence.
The main work is done automatically during the Pixel Tracking stage of the processing. DI4Dtracker allows you to then choose, select, edit and modify the pixels you are interested in tracking and export them in a number of different formats as 3D points.
The tracked points can be used directly as traditional motion capture data by exporting as a C3D file. Or they can be exported in our own format for use in the next stage of our process, batch Shape Transfer.
Click here to view a movie showing sample usage of the DI4Dtracker application. -
Step 5: Transfer 4D Shape to Uniform Mesh Topology And Export Animation
In order to drive a single, dense animatable mesh with a raw 4D sequence, the mesh with desired topology must be shrink-wrapped or 'conformed' to the shape of each frame in the raw 4D sequence. We do this by using a 'batch' version of the Shape Transfer tool.
The batch Shape Transfer tool lets you use the points generated by DI4Dtracker as input and rehapes the input template mesh to each frame of the raw 4D sequence. The tool also allows you to review your animated sequence in 3D and 'fix' parts of the animation tha need touched up. On each frame, vertices can be selected and dragged along the surface of the corresponding raw 4D frame for quick, easy and accurate adjustments. The completed animation can then be exported as a PointCache file.
Click here to view a movie showing sample usage of the Batch Shape Transfer tool. -
Step 6: Map High Resolution Static Textures To Uniform Mesh Topology
This step is optional.
Using one of our high resolution 3D capture systems and our Material Transfer tool, you will are able to map an extremely high-resolution texture and normal map onto the same 3D mesh as you are using for the facial animation for added detail and realism.
A full description of the Material Transfer tool can be found here. Click here to view a movie showing sample usage of the Material Transfer tool. -
Step 7: Import Animated Mesh And Texture Into Animation Package
Once the animation has been completed and texture maps and normal maps created, you can load the 3D model, images and the PointCache2 animation file into the animation package of your choice to add more effects and edits prior to rendering or exporting to in-game animation.
The PointCache file can be easily loaded into most popular 3D modelling and animation packages like 3D Studio Max, Maya, SoftImage, Blender, and so on. We also support the C3D format which allows the use of VICON software and MotionBuilder.
Click here to view a movie showing the animated mesh in a 3D modelling and animation package. -
Final Rendered Results
In the end, you will end up with a highly realistic, accurate, animated face which has a mesh topology of your choice. The flexibility of this method of facial animation allows you to then render or export the animation using the package you are most familiar with.
Click here to view a movie of sample output from the markerless 4D Facial Animation process.
