Realistic 3D models for fashion, film and healthcare

Meshcapade licenses technologies for creating avatars

© Meshcapade

The start-up company Meshcapade, a spin-off from the Max Planck Institute (MPI) for Intelligent Systems, automates creating realistic digital 3D bodies of people, also known as 3D avatars. Parametric 3D models like SMPL (Skinned Multi-Person Linear Model) are used to create 3D avatars of humans with accurate body shape and motion, using statistics and machine learning. Newer versions of the SMPL Model include SMPL-X (SMPL-eXpressive), which extends SMPL to provide hand motion and facial expressions, and SMIL (Skinned Multi-Infant Linear Model), which enables creation of realistic 3D avatars for infants. Another technology called SMPLify-X even allows generating avatars of people from a single image and produces detailed facial expressions, subtle hand gestures and realistic body movement and has great application potential in the fashion, gaming and film industry. This method can also be used together with SMIL Model to enable 3D movement detection of babies, and thus help with the early detection of developmental disorders. These and other innovations are based on proprietary research work of the MPI that has now been licensed by Meshcapade from Max Planck Innovation.

The creation of realistic and movable 3D models of the human body has previously required the use of complex methods and techniques. Motion capture (mocap) is one of these methods. Here, several cameras record the movements of 30 to 40 reflective markers that are attached to a person. The recorded marker data are usually converted to a 3D skeleton with days of manual work of multiple engineers. Finally, this skeleton is connected to a 3D model, which also takes many days to create. The SMPL model together with many of the automated body estimation technologies, licensed and further developed by Meshcapade, allow the calculation of an accurate 3D model and motion of the person within minutes from images, videos, depth cameras and even motion capture.

Since subtle gestures and facial expressions are often important for human interaction and communication, an update to the SMPL Model, called SMPL-X, makes it possible to create a complex, expressive 3D model of the body. In addition to realistic body poses, this model also includes detailed hand poses and facial expressions. The basis for this is a comprehensive database of high-resolution 3D scans of thousands of people of various body shapes, and in various body poses. SMPL-X is a statistical model trained to learn the difference between body shapes of real people and to create realistic pose and facial expressions for different body types.

The recently licensed technology called SMPLify-X now makes it possible for the first time to create these realistic 3D models using a single two-dimensional image. SMPLify-X estimates expressive 3D models by recognizing key features (such as joints-locations, facial features) in an image and uses the SMPL-X model’s statistical distribution to create realistic body shape, pose and expression that best fits to these detected key-points. This development represents a significant simplification for the acquisition and creation of realistic body models and opens up completely new possibilities for the use of 3D avatars. The models can be used to create convincing animated characters in films and computer games. Online clothing retailers can offer their customers a virtual try-on after simply uploading a picture. In addition, many other fields of application for the avatars are conceivable in our increasingly digitized world.

Another technology that was developed jointly with the MPI and the Fraunhofer Society describes a virtual infant body model (SMIL, Skinned Multi-infant Linear Model). On the basis of video sequences recorded from an infant using a depth camera or standard video, SMIL, together with SMPLify-X, can create an infant-specific 3D avatar with realistic movement parameters. This three-dimensional measurement of the spontaneous movement of infants can allow automated detection of pathological spontaneous motor skills on the basis of machine learning algorithms. Automated, inexpensive and easy-to-use screening with the aid of SMIL opens up completely new opportunities for pediatric screening.

Customers can license SMPL, SMPL-X, SMIL and SMPLify-X from Meshcapade and use them in their applications.

With our statistical models we want to improve the world of 3D modelling, sizing and animation of the human body. Our models are extremely versatile and can be used in many areas from fashion and entertainment to healthcare and fitness. The basis of our innovations is in particular the knowledge of the MPI for intelligent systems about the principles of perception, learning and action in machine-learning based 3D systems.

Naureen Mahmood
Managing Director of Meshcapade GmbH

In addition to these technologies, Meshcapade has licensed additional technologies from the MPI in the past twelve months that facilitate the creation of 3D models and provide great benefits in many areas of application.

With Meshcapade we got a very successful commercial partner understanding both customer needs and licensed technologies and constantly advancing them for the best results.

Dr. Bernd Ctortecka
Senior Patent & License Manager at Max Planck Innovation

About Meshcapade

Meshcapade is creating realistic human avatars to recover detailed human attributes such as bodies, faces, hands and soft-tissue movements from 3D & 4D scans, RGB-D sequences, motion capture and image data. Using machine learning and advanced graphics, we model the nuances of how humans look and move. Meshcapade avatars are already being used in a variety of fields, including research, apparel, biomechanics, virtual reality, film and as training data for newer AI-based methods. Using state of the art methods from world leading research institutes and we are pushing to bring realistic human models to life in everyday environments.

About Max Planck Institute for Intelligent Systems

Intelligent systems can optimise their structure and properties in order to successfully function within a complex, partially changing environment. Three sub-areas – perception, learning and action – can be differentiated here. The scientists at the Max Planck Institute for Intelligent Systems are carrying out basic research and development of intelligent systems in all three sub-areas. Research expertise in the areas of computer science, material science and biology is brought together in one Institute, at two different sites. Machine learning, image recognition, robotics and biological systems will be investigated in Tübingen, while so-called learning material systems, micro- and nanorobitics, as well as self-organisation will be explored in Stuttgart. Although the focus is on basic research, the Institute has a high potential for practical applications in, among other areas, robotics, medical technology, and innovative technologies based on new materials.

About Max Planck Innovation

As a technology transfer organization of the Max Planck Society, Max Planck Innovation is the link between industry and basic research. With our interdisciplinary team, we advise and support the scientists of the Max Planck Institutes in the evaluation of inventions, the filing of patents and the establishment of companies. We offer industry central access to the innovations of the Max Planck Institutes. In doing so, we fulfil an important task: the transfer of the results of basic research into economically and socially useful products.

Press contact

Marketing & Public Relations

Markus Berninger

Diplom-Kaufmann, certified graphic designer

Phone: +49 89 / 29 09 19-30