Realistic 3D models for fashion, film and healthcare

The creation of realistic and movable 3D models of the human body has previously required the use of complex methods and techniques. Motion capture (mocap) is one of these methods. Here, several cameras record the movements of 30 to 40 reflective markers that are attached to a person. The recorded marker data are usually converted to a 3D skeleton with days of manual work of multiple engineers. Finally, this skeleton is connected to a 3D model, which also takes many days to create. The SMPL model together with many of the automated body estimation technologies, licensed and further developed by Meshcapade, allow the calculation of an accurate 3D model and motion of the person within minutes from images, videos, depth cameras and even motion capture.

Since subtle gestures and facial expressions are often important for human interaction and communication, an update to the SMPL Model, called SMPL-X, makes it possible to create a complex, expressive 3D model of the body. In addition to realistic body poses, this model also includes detailed hand poses and facial expressions. The basis for this is a comprehensive database of high-resolution 3D scans of thousands of people of various body shapes, and in various body poses. SMPL-X is a statistical model trained to learn the difference between body shapes of real people and to create realistic pose and facial expressions for different body types.

The recently licensed technology called SMPLify-X now makes it possible for the first time to create these realistic 3D models using a single two-dimensional image. SMPLify-X estimates expressive 3D models by recognizing key features (such as joints-locations, facial features) in an image and uses the SMPL-X model’s statistical distribution to create realistic body shape, pose and expression that best fits to these detected key-points. This development represents a significant simplification for the acquisition and creation of realistic body models and opens up completely new possibilities for the use of 3D avatars. The models can be used to create convincing animated characters in films and computer games. Online clothing retailers can offer their customers a virtual try-on after simply uploading a picture. In addition, many other fields of application for the avatars are conceivable in our increasingly digitized world.

Another technology that was developed jointly with the MPI and the Fraunhofer Society describes a virtual infant body model (SMIL, Skinned Multi-infant Linear Model). On the basis of video sequences recorded from an infant using a depth camera or standard video, SMIL, together with SMPLify-X, can create an infant-specific 3D avatar with realistic movement parameters. This three-dimensional measurement of the spontaneous movement of infants can allow automated detection of pathological spontaneous motor skills on the basis of machine learning algorithms. Automated, inexpensive and easy-to-use screening with the aid of SMIL opens up completely new opportunities for pediatric screening.

Customers can license SMPL, SMPL-X, SMIL and SMPLify-X from Meshcapade and use them in their applications.

In addition to these technologies, Meshcapade has licensed additional technologies from the MPI in the past twelve months that facilitate the creation of 3D models and provide great benefits in many areas of application.