"Understanding and Reproducing Dance Movements with Physics" KAIST Develops 3D Generative AI

An artificial intelligence (AI) technology capable of understanding and realistically depicting the deformation and wrinkles of clothing as it moves during dance has been developed.

Rather than simply creating plausible visuals, this method learns the laws of physics to capture and understand movements that closely resemble reality before describing them. This is expected to enhance immersion and realism in films, the metaverse, and game avatars, and could be applied to the immersive media industry in the future.

KAIST announced on October 22 that the research team led by Professor Kim Taekyun of the School of Computing has developed a spatial and physics-based generative AI model called "MPMAvatar."

(From left) Ji-Hyun Lee, PhD candidate, Taekyun Kim, Professor, Changmin Lee, Master's candidate. Provided by KAIST

To address the limitations of 2D technology, the research team developed the MPMAvatar model by reconstructing multi-view videos into three-dimensional space using Gaussian Splatting, and then combining this with a physics simulation technique known as the Material Point Method (MPM).

The core of this approach is to reconstruct videos shot from multiple angles into a three-dimensional form, enabling AI to understand how objects move and interact as if they were real. In this process, the MPMAvatar model calculates the movements of objects based on their material, shape, and external forces, compares the results to actual footage, and learns to reproduce these movements by applying the laws of physics.

The team also succeeded in representing three-dimensional space as a collection of points, applying both Gaussian and MPM to each point to simultaneously achieve physically natural movements and realistic video rendering. By dividing the 3D space into countless small points and enabling each point to move and deform like a real object, they achieved natural visuals that closely match reality.

In particular, this research precisely expresses the interactions of thin and complex objects such as clothing by simultaneously calculating both the surface (mesh) and particle-level structure (points) of the object, and by applying the Material Point Method (MPM) to compute movements and deformations in three-dimensional space according to physical laws.

Additionally, to realistically reproduce scenes where clothing or objects move and collide, the team developed and incorporated a new collision handling technology. This enabled the MPMAvatar model to realistically depict the movements and interactions of people wearing loose clothing and to generate "zero-shot" inferences, allowing the AI to process and infer previously unseen data or experiences on its own.

The proposed technique can represent a wide range of physical properties, including rigid bodies, deformable objects, and fluids. Therefore, it is expected to be useful not only for avatars but also for generating general complex scenes.

Professor Kim stated, "The technology developed by our research team is significant in that it demonstrates the potential of 'Physical AI,' which goes beyond simply drawing realistic images with AI to understanding and predicting the laws of physics. This technology could be practically applied across the immersive content industry, including virtual production, film, and short-form media."

This content was produced with the assistance of AI translation services.